Classification is the act of deciding which of your existing categories a piece of content belongs to. It is a different operation from writing a description: the system is not producing new text, it is making a choice from a list you defined in advance. That single difference changes almost everything about how it behaves, how it fails, and how you control it.
If what you need is descriptive text rather than a category — meta descriptions, alt text, summaries — that is AI metadata generation, and it carries a different risk profile.
What gets classified
- Taxonomy terms. Which subjects a document covers, from an agreed subject list.
- Content and document types. Is this a policy, a procedure, a form, a case study, a contract?
- Topics and themes. Broader groupings used for navigation and recommendation.
- Entities. The organizations, products, locations, and people a document refers to — extracted and matched to records you already hold, rather than invented.
- Audience. Who this content is for, which frequently drives what a search should surface.
- Sensitivity. Public, internal, confidential, restricted. Different from the rest, for reasons below.
Nothing works without a vocabulary worth using
The most common cause of disappointing classification is not the model. It is that the organization never agreed what the categories mean.
If two experienced staff would assign a document differently, a system will too — and it will do so at scale and consistently, which makes the underlying ambiguity look like a technology failure. Before classifying anything, the vocabulary needs three properties:
- Definitions, not just labels. A written statement of what belongs under each term and what does not. "Compliance" means nothing on its own.
- Reasonable size. Large taxonomies are abandoned. If a term has never been applied to anything, it is not a category, it is an aspiration.
- Distinguishable terms. If two terms overlap heavily, classification will split content between them arbitrarily and search will suffer either way.
Building that vocabulary is content strategy work, not a machine-learning task — see enterprise AI content strategy.
Confidence scoring, and what to do with it
Unlike generated text, a classification usually arrives with a number attached: how confident the system is in the assignment. That number is the main control you have, and using it well is most of the design.
A workable three-band pattern:
- High confidence — apply automatically, and sample a proportion for quality.
- Middle confidence — suggest to an editor, who confirms or corrects in one action.
- Low confidence — leave unclassified and route for human attention.
Two things matter more than the exact thresholds. First, "unclassified" must be an allowed outcome. A system forced to always choose will produce a confident-looking wrong answer instead of a useful "I don't know", and the wrong answer is harder to find later. Second, set the thresholds from measurement, not intuition. Run a sample, compare against human judgement, and place the boundaries where the accuracy actually justifies them.
Silent drift is the failure mode
Misclassification rarely announces itself. A category applied slightly too broadly, a new kind of content the vocabulary never anticipated, a model update that shifts behaviour — each is individually invisible, and the aggregate shows up months later as "search has got worse".
What catches it:
- A gold set. A fixed sample of content with human-agreed correct classifications, re-run whenever the model, the prompt, the vocabulary, or the content mix changes. This is the single most useful artefact in the whole exercise and it costs a morning to build.
- Distribution monitoring. Watch how content is spread across terms over time. A term that suddenly absorbs a third of everything is a signal, not a success.
- Unclassified-rate monitoring. A rising proportion of low-confidence results usually means new content types the vocabulary does not cover.
- Sampling. A small, regular human review of automatic assignments. Habit, not audit.
Sensitivity classification is a security control, not a tidiness feature
This is the part that deserves separate treatment, because the consequence of an error is different in kind.
Every other classification error makes content harder to find. A sensitivity error can make content visible to people who should not see it — particularly in a platform where retrieval is permission-aware and the label determines who can retrieve a document into an answer. The controls that depend on getting this right are set out in security and privacy for AI-powered CMS platforms.
Three rules follow:
- Fail closed. An uncertain document is treated as more restricted, not less, until a person confirms.
- Never auto-downgrade. Moving a document from confidential to public should require a human decision, always, regardless of confidence.
- Log every assignment and change. If a document was exposed, you need to know what label it carried, when it changed, and on what basis.
This is where classification stops being a content-operations question and becomes part of the security posture, alongside the wider controls in our managed IT and cybersecurity practice.
Working out whether your taxonomy is ready for this? Schedule an AI CMS consultation — the vocabulary review usually tells you more than a model evaluation would.
Where it earns its place
- Backfilling a legacy library. Thousands of documents nobody will ever tag by hand. Even imperfect classification transforms findability from nothing to something.
- Keeping a large library consistent. Human tagging drifts between people and over time; a system is at least consistently wrong, which is easier to correct in bulk.
- Routing. Sending content to the right reviewer, team, or workflow based on what it is about.
- Search facets. Filters are only as good as the classification behind them, and empty or misleading facets actively harm trust in search.
- Retention. Knowing which documents are of a type with a defined retention period is a precondition for managing them at all.
How LABUSA approaches it
We start with the vocabulary rather than the model: is it defined, is it used, do people agree on it? Frequently the first deliverable is a smaller, better-defined taxonomy, which improves search on its own before any automatic assignment happens.
From there we build a gold set, measure accuracy per term rather than overall — an average hides the two terms that are failing — and set thresholds from that evidence. Our AI content management work treats classification as a governed, monitored capability rather than a one-off tagging exercise, because the value is in it staying accurate.
Frequently asked questions
How accurate is AI classification?
It depends far more on how well your vocabulary is defined than on the model. A clear, distinguishable taxonomy classifies well; an ambiguous one classifies badly regardless of technology. We would measure it on your content rather than quote a figure from someone else's.
Do we need training data?
Modern approaches often work from term definitions alone, without a labelled training set. You still need a gold set for evaluation — but that is tens of examples, not thousands.
Can it apply several terms to one document?
Yes, and usually it should — most content is genuinely about more than one thing. Set a limit, or documents accumulate weakly relevant terms that dilute search.
What about content in other languages?
Generally handled, though accuracy varies by language and by how much domain vocabulary is involved. Evaluate per language rather than assuming parity.
What happens when we change the taxonomy?
Existing assignments become stale. Plan for re-classification when terms are added, merged, or redefined, and re-run the gold set afterwards — a vocabulary change is a behaviour change.
Related reading
- AI Metadata Generation — producing descriptive text, as opposed to assigning categories.
- AI Content Automation — deciding which tasks run unattended.
- AI Content Lifecycle Management — how document type and retention connect.
- Enterprise AI Content Strategy — designing the vocabulary in the first place.