PII Redaction
Design pipelines that detect, minimize, redact, and govern personally identifiable information.
Start with minimization
The safest personal data is data the workflow never receives. Define the task’s required fields, remove unrelated content, and collect only what is necessary for the stated purpose. Map where data enters, is transformed, logged, cached, sent to providers, and deleted. Minimization reduces exposure and makes later redaction more reliable, but it does not eliminate the need for access controls.
Types of personal information
PII can include names, emails, phone numbers, addresses, identity numbers, account IDs, faces, voices, locations, and combinations that identify someone indirectly. Sensitive data may require stronger controls depending on context and jurisdiction. Create a project-specific inventory rather than relying on a generic list. Include data in prompts, attachments, tool arguments, errors, traces, analytics, and backups.
import re
text = "Contact anita@example.com or call +91 98765 43210."
text = re.sub(r"[\w.+-]+@[\w.-]+\.\w+", "[EMAIL]", text)
text = re.sub(r"\+?\d[\d ()-]{8,}", "[PHONE]", text)
print(text)Detection methods
Use deterministic patterns for well-formed numbers and addresses, named-entity detectors for contextual text, and specialized tools for documents, faces, or audio. No detector is perfect: formats vary, OCR introduces errors, and names overlap with ordinary words. Combine methods, preserve confidence and location, and send uncertain cases to review instead of treating one regex as complete protection.
Redaction and pseudonyms
Replace values with stable placeholders when the model needs to understand relationships, such as matching the same customer across messages. Keep the mapping in a separate protected store with a short retention period. Irreversible redaction is safer when the value is not needed later. Ensure placeholders cannot accidentally resemble real credentials or become instructions in a prompt.
Logging boundaries
A redacted user interface does not help if the original prompt is copied into logs, traces, crash reports, or analytics. Apply redaction before observability code receives the data, and review third-party SDK behavior. Mask secrets and identifiers in exceptions and metrics labels. Restrict investigation access and document when an authorized operator may view original content.
Verification and deletion
Test detection and redaction on realistic formats, languages, OCR output, nested JSON, screenshots, audio transcripts, and adversarial spacing. Verify that original values cannot be recovered from caches, backups, embeddings, indexes, or generated exports beyond the approved retention period. Deletion should cover every copy and provide evidence of completion where governance requires it.
Human review and governance
Some workflows need a reviewer to resolve ambiguous entities or confirm that a document is safe to process. Give reviewers the minimum context and permissions needed, and record decisions without creating another uncontrolled copy. Define ownership, retention, provider terms, incident response, and user rights before launch. Redaction is a process spanning product, engineering, security, and operations.
Worked example: preparing support logs for debugging
A support trace may contain an email, phone number, account ID, address, and free-form personal detail. Redact before writing to a shared log, but preserve a stable pseudonymous token when correlating events is necessary. Keep the mapping separate and access-controlled, or avoid retaining it entirely. Redaction should happen before analytics, error reporting, prompt traces, and third-party transport—not only in the final dashboard.
Code walkthrough
The regex example handles common email and phone shapes and replaces them with labels. It is a useful demonstration, not a complete detector: Unicode names, international numbers, account IDs, images, and context-dependent identifiers need additional methods. Run redaction before formatting logs, test it against adversarial spacing and punctuation, and never print the original text while debugging the redactor. Measure both missed and over-redacted values.
Trade-offs to measure
Aggressive redaction lowers exposure but can remove information needed for diagnosis or model quality. Pseudonymization preserves relationships while creating re-identification risk if the mapping leaks. Named-entity models catch more context than regexes but add cost and their own data-processing path. Define the minimum detail needed by each consumer and use different retention and access rules for raw, redacted, and aggregate data.
Practical exercise
Build a fixture set containing emails, phone formats, addresses, IDs, names, dates, and deliberate near-misses. Run the redactor and manually label misses and false positives. Add a test that asserts no raw email or phone remains in the emitted log. Repeat on OCR text and model output, then document where the pipeline still requires human review or a stronger detector.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.