Resources August 5, 2026 8 min read

How to Choose an AI CMS Solution

How to evaluate AI content platforms once you know what you need: the criteria that actually differentiate, the questions vendors find hard, and how to run a comparison on your own content.

By Martin Laster CEO & Enterprise Architect, LABUSA

A fingerpost with arms pointing in several directions, silhouetted against a pink sunset sky.

This article assumes two earlier decisions are made. Whether to move at all is the subject of AI CMS versus traditional CMS, and what you actually need — goals, content condition, ownership, structure, security posture — belongs to enterprise AI content strategy, which should be settled before a single product is considered.

What follows is the selection itself: how to compare candidates, which criteria separate them, and where the process tends to go wrong. It names no products, because the right answer depends entirely on the requirements you arrived with.

Selection is mostly a test of your own requirements

An uncomfortable observation worth stating first. Most difficult selections are difficult because the requirements are vague, not because the candidates are close. When a shortlist feels impossible to separate, the usual cause is that the criteria being applied are ones every serious product satisfies.

The corollary is practical: if you cannot articulate what would make you reject a candidate, you are not ready to evaluate. Write down the two or three conditions that would disqualify a product outright — a data-residency requirement, a permission model you must honour, a content type nobody can express — and lead with those. They will shorten the list faster than any feature matrix.

The criteria that actually differentiate

Feature lists converge; these do not.

The content model. Can it express your content — the types, the fields, the relationships between them — without forcing you into a shape the product prefers? A product that assumes a content model you did not design will fit some of your content and quietly distort the rest.
Permission-aware retrieval. Not "does it have permissions" but: is filtering applied within the query and evaluated against the requesting user? Retrieve-then-filter is a leak waiting to happen, and it is a genuine architectural difference between products.
Where processing happens. Which regions, which providers, what is retained, and whether your data is used for training. This eliminates candidates outright in regulated environments, so it belongs early rather than in a compliance review at the end.
The editorial experience. The single most under-weighted criterion. A platform editors avoid produces no value regardless of its capabilities, and structured authoring is a real change in practice. Have actual editors try it.
Integration reality. Not "does it have an API" but whether the specific systems you named in strategy can be read, whether their permissions can be carried across, and what happens when one is unavailable.
Separability. Whether the CMS, the retrieval layer, and the model provider can be replaced independently. Providers, prices, and model behaviour all change; a design that keeps model calls behind an internal interface is inexpensive insurance.
Refusal behaviour. What it does when it finds nothing relevant. A product that improvises rather than declining is a liability, and this is testable in an afternoon.

Questions vendors find hard

These separate candidates because they cannot be answered from a marketing page. Ask them in writing, and treat a confident non-answer as an answer.

Is our data used to train your models, or any third party's, by default and under the contract we would sign?
Exactly where is a request processed, and what is retained — including anything held for abuse monitoring outside the main retention commitment?
How is a user's permission evaluated at query time, and what happens on the error path if that check fails?
When the underlying model changes, how are we notified, and what recourse do we have if behaviour changes?
Show us the system declining to answer. Not a slide — the product.
What does it cost at our realistic volume in year two, including growth?
If we leave in three years, what do we take with us and in what format?
Which of your named references is closest to our content volume and condition, and may we speak to them unaccompanied?

Answers to the cost question in particular deserve scrutiny — see AI CMS cost and budget considerations for what tends to be omitted.

Evaluate on your content, not theirs

A demonstration on a vendor's curated corpus tests the corpus. The only evaluation that predicts anything is one run on your own material, and it is worth insisting on even when it slows the process.

A workable structure:

Fix the question set first. Twenty to fifty real questions, drawn from your search logs and your inbox, with known-correct sources. Write them before you see any product, or they will drift towards what the first demonstration did well.
Include content in poor condition. Deliberately. Your corpus contains contradictions and stale documents, and how a candidate behaves there is more informative than how it handles your best material.
Test the same questions across candidates and record the answers verbatim. Impressions do not survive a week of demonstrations; a table does.
Test as more than one role. Confirm that retrieval never surfaces to a role what that role may not see. This is where retrieve-then-filter architectures reveal themselves.
Have editors do editorial work — create a record, revise one, find something from three months ago — rather than watching a scripted flow.

Running a selection? Schedule an AI CMS consultation — an independent view on the criteria is usually more valuable than another vendor demonstration.

Build, buy, or extend

Rarely a clean choice between three options. Most implementations combine a bought CMS, bought model access, and built integration and content modelling — the judgement is about where differentiation actually lies, and strategy is where that is settled.

The option most often missed at selection time is the third one. Where an organization already runs a capable content platform, extending it is frequently cheaper, faster, and less disruptive than replacing it — and it avoids a migration, which is its own project with its own risks. A selection process that only compares replacements will never surface that answer, so put it on the list explicitly even if it is quickly dismissed.

Traps in the process itself

The feature matrix. Long checklists reward products optimised for checklists. Weight the handful of criteria that would actually disqualify a candidate, and let the rest be pass or fail.
Evaluating the demonstration rather than the product. Demonstrations are built to succeed. Insist on your questions, your content, your roles.
Deferring security and residency to the end. These eliminate candidates. Discovering that after a three-month evaluation wastes the evaluation.
No editor in the room. Selections made without the people who will use the product daily produce platforms that go unused.
Scoring the model rather than the system. Retrieval quality, chunking, and permission enforcement affect answer quality more than model choice does, and they are properties of the system around the model.
Treating the reference call as a formality. Ask a reference what they would do differently and what took longer than expected. Those two questions produce more signal than the rest of the call.

Decide, then write down why

Record the decision with the criteria that drove it and the conditions that would change it — a residency rule, a volume assumption, a provider's retention terms. This costs an hour and pays for itself twice: when someone asks in a year why this product was chosen, and when one of those conditions changes and you need to know whether the decision still holds.

It is also the document that makes an exit decision rational rather than political, which is the harder of the two conversations.

How LABUSA approaches selection

We start from requirements rather than a shortlist, and we are frequently engaged to challenge assumptions that have gone unexamined internally — most often the assumption that a replacement is needed at all. Because LABUSA's AI CMS practice sits alongside enterprise architecture, integration, and security work, our evaluations account for what it will take to operate the result rather than only to buy it.

Where the assessment shows that extending your current platform delivers most of the benefit, that is what we will recommend — and where Drupal is the right foundation, our Drupal development and managed hosting packages cover that path directly.

Frequently asked questions

How long should a selection take?

Long enough to test candidates on your own content with your own questions, and no longer. Processes that run for many months usually indicate unresolved requirements rather than close candidates.

Should we run a formal RFP?

If procurement rules require it, the criteria above are what belongs in it. If they do not, a written question set and a hands-on evaluation on your content will tell you more than a document exchange.

How much should the model provider matter?

Less than expected. Retrieval design, content structure, and permission enforcement determine answer quality more than model choice, and providers change. Design so the provider can be swapped.

What if no candidate fits?

That is a useful outcome, and it usually means the requirement is genuinely specific — in which case building the differentiating part over bought components is the answer, not compromising the requirement.

Can we change our mind later?

Only as far as the architecture allows. Keeping the CMS, retrieval layer, and model provider separable is what makes that a decision rather than a rebuild, and it should be a selection criterion for exactly that reason.