Data Processing for AI
Prepare documents, records, and datasets for training, retrieval, and evaluation in Python.
Separate pipeline stages
Organize loading, decoding, cleaning, normalization, splitting, enrichment, validation, and export as explicit stages. Small functions make it easier to inspect intermediate results and rerun only the failed work. Pass records with stable identifiers and metadata rather than anonymous strings. Clear stages also help compare two preprocessing versions without confusing data errors with model behavior.
Normalize carefully
Normalization may include encoding repair, whitespace cleanup, date formats, language detection, duplicate removal, and document structure preservation. Do not erase information merely because it looks unusual; punctuation, headings, coordinates, and table relationships may matter for retrieval or evaluation. Keep the original source and transformation version so a questionable output can be traced back.
records = [{"id": "doc-1", "text": "Quarterly results", "source": "report.pdf"}]
for record in records:
if not record["text"].strip():
continue
chunk = {**record, "pipeline_version": "2026-07-18", "chunk_index": 0}
print(chunk)Preserve provenance
Carry document IDs, source locations, timestamps, labels, permissions, and processing versions through every transformation. Provenance supports citations, debugging, access control, deletion requests, and dataset audits. Store relationships between chunks and their parent document. An embedding without its source and permission metadata may be impossible to use safely in a multi-user application.
Quality and validation
Check required fields, encoding, size, language, duplicate rates, label balance, malformed records, and unexpected distributions before generating embeddings or training artifacts. Sample outputs for human inspection, especially after OCR or splitting changes. Fail loudly for systemic corruption and quarantine individual bad records with a reason. Quality checks should run again when data is updated.
Scale and resource limits
Stream large files instead of loading everything into memory, use bounded concurrency, checkpoint completed work, and write outputs atomically. Respect provider rate limits and local disk capacity. Make stages idempotent so a restart does not duplicate chunks, embeddings, or labels. Measure throughput, memory, retries, and failure counts to identify whether the bottleneck is I/O, CPU, network, or model inference.
Privacy and retention
Remove unnecessary personal data before processing and apply access filters before creating searchable artifacts. Review whether embeddings, summaries, and derived labels can still reveal sensitive information. Define retention and deletion across raw data, temporary files, indexes, caches, and exports. Protect local work directories and logs because preprocessing often handles more source detail than the final application displays.
Reproducible exports
Write a manifest containing source versions, filters, transformations, chunking settings, model versions, counts, checksums, and errors. Keep training, retrieval, and evaluation datasets distinct to reduce leakage. Compare distributions before and after a change, then run a fixed quality sample. A data pipeline is production-ready when it can be paused, audited, rerun, and deleted safely.
Worked example: preparing a searchable document set
A document pipeline can ingest PDFs, preserve headings and page numbers, split content into retrieval units, and attach access metadata before generating embeddings. Keep tables and captions identifiable instead of merging every element into plain text. Quarantine files that fail parsing and continue safe records. The search index should be rebuildable from a manifest; otherwise a small preprocessing change can create an unexplained production difference.
Code walkthrough
The existing snippet skips blank text and adds a pipeline version and chunk index to a record. Extend the same pattern with a stable document ID, source checksum, page, permissions, and transformation name. Write each output atomically and make reruns idempotent. Before embedding, validate that the record has a source and allowed access label; an anonymous chunk is not safe to publish.
Trade-offs to measure
Small chunks improve retrieval precision but may lose context; large chunks preserve context but consume more tokens and dilute similarity. OCR recovers scanned documents while introducing character and layout errors. Deduplication saves index space but can remove legitimate repeated policies with different versions. Compare recall, citation quality, index size, processing time, and deletion behavior before selecting a pipeline.
Practical exercise
Process ten documents containing headings, a table, a duplicate version, an empty page, and one unreadable file. Produce a manifest with counts, checksums, errors, and chunk metadata. Ask five questions that require page-level citations and inspect the top results. Change the chunk size once, compare evidence recall, and verify that deleting one source removes its derived chunks and embeddings.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.