Twelve months after an engagement ends, somebody who did not authorize it will ask what it achieved. Often that person is new, or is in finance, or is preparing a budget request and needs to know whether the last one worked.
The honest answer at that point is usually assembled from memory, because nobody wrote down beforehand what would count as an answer. The engagement may have gone perfectly well. Nobody can now show that it did.
This page is about the twenty minutes at the start that make that conversation possible: deciding, before the order is issued, what evidence would show the work was worth doing, and where that evidence will come from.
A different question from what it cost
What an engagement costs is settled in advance under this contract and is set out in what an hour costs and how a budget is built from it. That page answers the question a finance officer asks before signing.
The question here is the one asked afterwards, and it is not answerable by the same means. A cost is a number the contract produces for you. Whether the spending achieved anything is a judgment your organization has to make about its own circumstances, using evidence it has to decide in advance to collect.
What to measure
Four families of evidence are worth agreeing before a purchase. None of them requires a dashboard, and all of them can be captured by one person in an afternoon.
Did the stated problem stop happening? Most engagements exist because something specific is going wrong: a system falls over on results day, a manual reconciliation eats two days a month, a report that should take an hour takes a week. Write down what is happening now, in a sentence a non-technical colleague would recognize, with a rough frequency. Afterwards it is either no longer true, still true, or true less often, and all three are useful answers.
The evidence for that sentence almost always exists already, which is why this is cheaper than it sounds. The helpdesk queue knows how often the thing fails. The finance calendar knows which week is lost to the reconciliation. The person who maintains the workaround knows exactly what it costs them, and has usually never been asked. None of that needs a new system, and it is worth naming the source in the order alongside the statement, because a measure whose evidence has no home tends not to be collected.
What did it cost you internally? Every engagement consumes your own people: interviews, decisions, environments, testing, review. That cost is real, it is rarely in any quote, and it is the figure most often missing when the same kind of purchase is considered again. Nobody needs a timesheet. A note at the end recording roughly how many of your staff were involved and for roughly how long is enough to make the next estimate honest.
Did the capability stay? Some engagements leave your organization able to do something it could not do before; others leave it dependent on the supplier to do that thing again. Both can be legitimate purchases, but only one of them should be described as building capacity. The test is simple and worth applying six months later: can somebody on your staff do it, or explain it, without calling anyone.
Did the next purchase get easier? This is the least obvious and often the most valuable. A completed piece of work usually leaves an organization better informed about its own environment. If the next requirement in the same area takes less time to specify, draws fewer questions from suppliers and comes back with tighter quotes, the earlier engagement is part of the reason.
What good looks like
Write the answer as a plain statement of before and after, in the ordering document, with somebody's name on it and a date to check it.
The discipline is that it must be checkable by a person who was not involved. "Improved reliability" is not checkable. "Enrollment ran in September without the overnight failure that required a manual restart last year, confirmed by the registrar" is checkable, by the registrar, in September. The second sentence took no longer to write than the first.
Three properties make such a statement work, and all three are ordinary.
- It names the person who will confirm it, and that person is not the supplier and not the project's sponsor. Somebody whose ordinary work is affected is the right judge.
- It names when. A statement with no date is checked when somebody remembers, which is usually when something has gone wrong.
- It is written before the work starts. Afterwards, everyone involved has an interested view, and the measure tends to become whatever the engagement happened to produce.
Two or three such statements are plenty. A dozen is a sign that nobody decided what mattered, and a list nobody can check is worse than no list, because it looks like rigor.
How long it takes
Expect different shapes of engagement to show a result on different horizons, and say which you are buying, because a measure checked at the wrong moment reads as failure.
A piece of advisory work produces its result almost immediately: the document exists, the decision is either made or is not, and the test is whether the organization acted on it within its own decision cycle. If a recommendation sits unactioned for a year, that is a finding about the organization rather than about the supplier, and it is worth recording as one.
An implementation shows its result when the thing it replaced would next have failed, which is a seasonal question in a school district and a monthly one in a finance office. Choosing the check point is a matter of asking when the old problem would ordinarily have recurred.
The commoner error is checking too early rather than too late. An implementation reviewed the week after handover measures the handover, which is a fair thing to measure and a different one. Where an engagement has a natural settling period, say so in the order and set the date past it.
Work that builds a capability inside your team is the slowest to read and the easiest to misjudge, because the visible activity ends before the effect starts. The honest check is the second time the situation arises, not the first, and it may fall in the following budget year. That is not a reason to skip it; it is a reason to write the date down while anyone still remembers the engagement. Which of these shapes you are buying is worth settling early, and the engagement models a public agency is likely to be offered sets out the differences.
The measures that mislead
Some numbers are easy to produce, arrive unprompted, and say very little about whether the money worked.
- Hours delivered. This measures that the engagement happened. Under an hourly contract it is also the invoice, which makes it the least independent evidence available.
- Volume of activity. Tickets closed, systems touched, documents produced. Activity rises when work is happening and also when work is inefficient.
- Satisfaction. Worth collecting and easy to misread. Satisfaction usually reflects how an engagement felt to the people in it, which is a real thing and is not the same as whether the problem stopped.
- On time and on budget. Against an estimate your own organization supplied or approved, this measures the estimate. A project can land exactly on a number everybody agreed and still deliver something nobody needed.
- Anything that moved for another reason. Systems get replaced, staff change, seasons turn, and a measure that improved during an engagement did not necessarily improve because of it. The guard is to write down, at the start, what else was expected to change in the same period. It costs a sentence and it is the difference between evidence and coincidence.
Be most careful with a calculated saving. A figure arrived at by multiplying an assumed time saving by an assumed hourly cost will be asked about eventually, and it will not survive being asked about. This is not a small-organization failing: the Government Accountability Office reported that federal agencies did not consistently track their cloud savings, and recommended they improve that tracking, in its review of federal cloud computing usage and savings data. If agencies with dedicated oversight staff find attribution hard, a claimed return on a consulting engagement deserves the same skepticism. For a longer treatment of how public bodies assess their own technology investment process, GAO's framework for IT investment management is the standard reference.
Writing it into the purchase
All of this amounts to a few lines in the same document that carries the scope: the current situation in a sentence, the two or three statements that would show it had changed, who confirms each one, and when. It belongs there rather than in an email, because that is the document both parties keep, and where the engagement is meant to produce a written deliverable it is the same document that names it: specifying the document you will own covers that side.
It also improves the work itself. A supplier who can see the test tends to aim at it, and a supplier who disagrees with the test will say so during scoping, which is a conversation worth having before the order rather than after the invoice. Where in the purchasing sequence that document is produced is covered in how a requirement becomes an order.
The service categories, roles and published ceilings this work would be bought against are set out on the agreement this spend was authorized against. If you can describe the problem you want to stop having, LABUSA can help turn it into a scope with a test attached, and the fastest way to start is to bring us the problem you want to stop having.