Toxicity and Hallucination Filtering
Use layered checks to reduce harmful, fabricated, or unsupported answers.
Separate the problems
Toxicity concerns harmful, abusive, or demeaning content, while hallucination concerns claims that are fabricated, unsupported, or presented with unjustified confidence. They need different signals and mitigations. A response can be polite but false, or factual but unsafe in context. Define the failure categories separately so evaluation and product controls do not collapse them into one vague “bad output” label.
Input and output moderation
Classifiers, policy rules, blocklists, and human escalation can identify requests or responses that need refusal, transformation, or review. Consider context, reclaimed language, quotation, and educational use to reduce simplistic false positives. Moderation scores are signals, not final authorization. Apply checks to streamed and final output, and decide what happens when a classifier is unavailable.
def answer_or_review(answer, citations, moderation):
if moderation != "pass" or not citations:
return {"status": "needs_review", "answer": None}
return {"status": "supported", "answer": answer, "citations": citations}
print(answer_or_review("Claim", ["source-1"], "pass"))Grounding answers
Reduce hallucination by supplying relevant evidence, requiring citations, validating extracted fields, and instructing the model to abstain when sources do not support a claim. Retrieval quality matters: irrelevant or stale context can make an answer look grounded while still being wrong. Compare each important claim with the source where feasible, especially for legal, financial, medical, or operational decisions.
Structured and deterministic checks
Schemas can catch missing fields, invalid values, and unsupported categories, while business rules can check arithmetic, dates, permissions, and allowed actions. These checks are stronger than asking the model to “be accurate.” They should fail visibly and route to correction or review. Do not silently coerce an unsafe value into a valid-looking one.
Uncertainty and abstention
A reliable assistant knows when evidence is weak or the request is outside scope. Design response states such as supported answer, clarification needed, insufficient evidence, and human review. Avoid forcing every request into a confident paragraph. Measure whether uncertainty language correlates with actual errors, and improve the underlying workflow when the model is uncertain for predictable reasons.
Measuring filter performance
Track true and false positives, unsafe completions, unsupported claims, refusal quality, user reports, and reviewer overrides. Test adversarial phrasing, multilingual content, sarcasm, quotations, long context, and attempts to split a harmful request across turns. Review examples regularly because input distributions and model behavior change. An average safety score can hide a rare but severe failure.
Layered response design
Use moderation, grounding, validation, permissions, human review, and incident response together. Give users a safe correction path and a way to report harmful or incorrect content. Preserve enough redacted evidence to investigate. Filters reduce risk, but they do not make a model authoritative; the product must still set boundaries and keep people accountable for consequential outcomes.
Worked example: answer a policy question with evidence
A policy assistant should answer only when retrieved passages support the claim and moderation permits the content. If the user asks for a policy that is absent, return a useful “not found” response and link to the search or contact path. For harmful or abusive input, distinguish refusal from a technical error. Store citations and review decisions so a user can challenge an incorrect filter instead of receiving an unexplained blank answer.
Code walkthrough
The function checks moderation and citation presence before returning a supported answer. Extend it to verify that citations actually entail the claim, that the answer contains no unsupported numerical changes, and that the moderation result is fresh. Return reason codes rather than one status string. Keep filtering separate from authorization and business rules so a toxicity classifier cannot accidentally approve a financial action.
Trade-offs to measure
Strict toxicity thresholds can block reclaimed language, quotations, or legitimate discussions of abuse; loose thresholds can expose users to harm. Citation presence is weaker than citation correctness, and a fluent hallucination may include irrelevant links. Combine classifiers, retrieval checks, human sampling, and product-specific policies. Measure false positives, harmful misses, unsupported claims, correction rate, and appeal outcomes.
Practical exercise
Create fixtures for supported answers, missing evidence, contradictory sources, quoted harmful language, direct abuse, and benign discussion of safety. Implement separate statuses for supported, unsupported, blocked, and review. Inspect each result with a human rubric and add one regression test for every discovered failure. Verify that the user receives an actionable next step in every non-success state.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.