Quality assurance for AI-assisted content is frequently reduced to a single question — is it accurate? — which is both the most important criterion and the least tractable one. Accuracy cannot be checked without knowing what the truth is, and a machine that knew that would not have needed to draft anything.
The productive move is to stop treating quality as one judgement and break it into criteria that can be defined, some of which a system can genuinely evaluate. What remains is a much smaller amount of irreducibly human work, aimed at the things that actually require a person.
Quality is several different properties
Content can fail in ways that have nothing to do with each other, and a process that checks only for the failure it last suffered will keep being surprised. The criteria worth naming separately:
- Accuracy. The statements are true, and true now.
- Grounding. Every substantive claim traces to an approved source. Distinct from accuracy — text can be true and still ungrounded, which means nobody can verify it or notice when the underlying fact changes.
- Completeness. Nothing a reader needs in order to act has been omitted. The characteristic AI failure is a fluent answer that leaves out the exception.
- Consistency. It does not contradict other published content, or itself.
- Clarity. A reader in the intended audience can follow it at the reading level they actually have.
- Voice. It sounds like the organization rather than like generic prose.
- Accessibility. Heading structure, link text, alternative text, and table markup are usable by assistive technology.
- Findability. Titles, descriptions, and structure allow both search engines and internal retrieval to place it correctly.
Writing these down is more than an exercise. "Improve quality" cannot be assigned, measured, or automated; "no published procedure may contain an ungrounded claim" can be all three.
What a machine can check, and what it cannot
The honest division matters, because assuming a check is covered when it is not is how errors reach publication with a green tick beside them.
Reliably automatable: reading level, sentence and paragraph length, prohibited or deprecated terminology, heading-order violations, missing or placeholder alternative text, broken links, missing required fields, duplicate or near-duplicate passages elsewhere in the estate, and content past its review date.
Automatable as a flag, not a verdict: tone and voice alignment, whether a passage contradicts another document, whether a claim appears in the cited source, and whether a summary faithfully represents the thing it summarises. A machine can raise these usefully. It cannot close them, and a system that reports them as pass or fail will be wrong in both directions.
Not automatable: whether a statement is true in the world, whether it is appropriate for this audience in this situation, whether an omission matters, and whether the organization is prepared to stand behind it. These are the checks a person is for, and protecting the time to do them is the actual purpose of automating everything above.
Grounding is the check that pays for itself
Of everything on the list, verifiable grounding does the most work, because it converts an unanswerable question into an answerable one. "Is this true?" requires expertise. "Which approved source says this, and does it still say it?" requires a lookup.
In practice that means substantive claims carry a traceable link to their source, a reviewer can move from a sentence to the passage it came from without a search, and — the part most often skipped — a change to a source can surface the content that drew on it. Without that last link, correcting a source leaves every derived page confidently repeating the old figure.
An assistant that answers users has the same obligation in a more visible form: an answer with a citation can be checked by the reader, and one without it asks for trust it has not earned.
The failure modes worth checking for specifically
Generated content fails in characteristic ways, and knowing them makes review far faster than reading with general suspicion.
- Confident invention. A plausible specific — a figure, a date, a named standard, a citation — that is simply wrong. Numbers and proper nouns deserve disproportionate attention because they are both the most likely to be invented and the most likely to be relied upon.
- Averaged-out voice. Text that is competent and sounds like nobody. Corrosive slowly rather than dangerous immediately, and easy to stop noticing.
- Omission of the exception. The general case stated cleanly with the qualification dropped. Particularly costly in procedures and eligibility rules, where the exception is often the reason someone is reading.
- Silent staleness. Fluent, well-formed, and describing how things worked two years ago. Nothing in the text signals it.
- Near-duplication. Cheap generation makes it trivial to produce four similar pages instead of maintaining one. Each is individually fine; together they split search relevance and guarantee that three of them will drift out of date.
Where generated metadata rather than prose is the concern — tags, summaries, descriptions — the evaluation criteria differ enough to be treated separately, in metadata generation.
Checking at scale means sampling, not reading
Once volume rises past what anyone can read, quality assurance stops being an activity performed on every item and becomes a measurement of the process producing them. That is a genuine change in method, and organizations tend to arrive at it late.
Three practices do most of the work:
- Sample what was approved, not only what was rejected. The rejected items tell you the checks are catching something. Only the approved sample tells you what they are missing.
- Score against a rubric, by more than one person. A short scoring sheet against the criteria above, applied by two reviewers, converts opinion into a number you can track. Disagreement between reviewers is information — usually that a criterion is underspecified.
- Keep a fixed evaluation set. A stable collection of real questions with known-good answers, re-run after any change to prompts, models, or content. Without it you cannot distinguish a system that improved from one you happen to feel better about.
That last practice is what makes quality observable over time rather than anecdotal, and it is the one most often missing.
Working out what to check and how? Schedule an AI CMS consultation — defining the criteria is usually the shortest part and the one that changes the most.
Quality drifts, and nothing announces it
An AI content process can be working well and be materially worse six months later with no change on your side. A provider updates a model and the tone shifts. A prompt is edited to fix one page. Source content degrades and the output degrades with it, faithfully.
None of this produces an error. It produces slightly worse content, continuously, below the threshold anyone would report — which is why quality needs a standing measurement rather than an inspection at launch. The evaluation set above is what detects it; a periodic sample is what confirms it.
Reader signals are a quality measure
The cheapest quality data most organizations already hold and rarely read: zero-result searches, queries repeatedly refined, pages people leave immediately, and the questions support keeps answering that the content was supposed to answer. Each of those is a reader telling you a specific piece of content failed, without being asked and without a survey.
Making this usable is mostly a matter of routing — a simple report-an-error mechanism on published content, and a path from a support agent's observation back to the owner of the content set. Neither is technically demanding. Both are frequently absent.
How LABUSA approaches quality assurance
We start by writing down the criteria that matter for a given content set, because it is not the same list for a marketing page and a safety procedure, and then automate the checks that can be automated so reviewer attention goes where it is genuinely needed. Grounding and traceability get disproportionate weight in an AI-powered CMS design for the reason above: they make verification cheap.
Where content is already structured, much of this is configuration rather than construction — required fields, validation, review dates, and reporting are things a mature platform such as Drupal already supports, which is one reason our Drupal development and managed hosting packages are often the foundation rather than a separate track.
Frequently asked questions
Can AI check its own output?
Usefully for form — structure, readability, terminology, internal contradiction. Not for truth. A model evaluating its own factual accuracy shares the assumptions that produced the error.
How much should we sample?
Enough to see a pattern, reviewed consistently, rather than a percentage chosen for its appearance. Sample more heavily after any change to prompts, models, or source content, and after any escaped error.
Is AI-assisted content held to a higher standard?
The standard should be the same; the checks differ because the failure modes differ. A human writer rarely invents a plausible citation, and rarely produces forty near-identical pages in an afternoon.
What single check would you start with?
Duplicate and near-duplicate detection across what you already have. It is cheap, fully automatable, and tends to reveal that the quality problem is partly a volume problem.
Who does this work?
The automated checks belong in the platform. The judgement belongs to whoever owns the content — which is a governance question rather than a QA one.
Related reading
- Human-in-the-Loop AI for Content Management — the approval that these checks inform.
- AI Content Governance — who sets the standard and who is accountable for meeting it.
- AI Metadata Generation — evaluating generated field values rather than prose.
- AI Content Lifecycle Management — keeping content within its review period.
- AI-Assisted Editorial Workflows — where these checks sit in the production process.