
Government Consulting
Your AI pilot might finally tell you how your service performs
I forget why I wanted an Apple Watch for Christmas; I just remember really wanting one.
This was during that body optimization craze a few years ago. Measure everything, optimize everything, become your best, leanest self. My wife and kids obliged, and by Christmas morning I was counting steps (from the sofa to the turkey), tracking my heart rate (surprisingly high that holiday, which I blame on the gravy), and running ECG tests, though I wasn’t quite sure what useful information that contained other than my ECG readings were quite average, thank you very much.
Soon after, a friend directed my gaze towards another shiny object: a sleep tracking app for the watch that helps improve sleep habits.
I mean, it’s not like I was having a bad sleep experience, really. But knowing my chronotype—I’m a night owl forced into an early-bird schedule against my will—I’ve always struggled to get my brain fully rebooted in the mornings.
But no longer! Finally, I would be able to get better rest, waking up sharp, refreshed, and ready to tackle the day. I imagined leaping out of bed every morning, instantly awake, my feet slipping through my socks before they hit the ground as I ran multiplication tables in my head.
What I wasn’t prepared for was this app telling me how poor my sleep actually was. Five hours of sleep! I thought it was at least seven. I woke up five times during the night? I don’t even remember dreaming. No wonder I was shuffling around like a zombie every morning.
I bought the app to sleep better. It didn’t make me sleep better. As a sleep aid, it failed. As a measuring tool, it gave me the first honest look I’d had at my own sleep “performance.”
Better than what?
Earlier this year, I started working with government teams, and talking with public-sector leaders across Canada, to help assess opportunities for AI integration into public digital services.
Should we use AI in this service? Where? Will it make things better?
One thing I’ve learned so far: the most valuable thing an AI pilot produces might have nothing to do with the AI. It might be the first honest measure of how your service performs today, the same thing my watch did for my sleep.
(A quick note on scope: governments are using AI in many ways, from helping staff build services faster to automating or assisting specific steps inside a service. Baselines matter for all of them, but I’m focusing on AI inside the service here.)
Everyone close to a service, especially the staff who run it every day, can probably tell you exactly where it falls short and where it could improve. Many can already picture where AI might help, whether by taking on repetitive work or by improving accuracy.
It’s reasonable to think that the outcome they want—work done faster, cheaper, or better—makes for a simple test. Run the AI proof of concept, pilot, or tool, and the answer to “Is this better?” is yes or no.
But it isn’t that simple. Better than what? If no one has measured the before, the comparison comes down to how much better it feels. It sounds obvious, even a little basic, to say you need a baseline to judge how much something has improved.
And yet, in our experience across many government services, few have clear performance measures at the step where AI would sit. Teams can put a shape around the problem (say, 40,000 subsidy forms, a six-month permit backlog, a phone queue that’s full by 9 a.m.), but they can’t tell you whether a new intervention, in this case an AI tool, would meaningfully improve the measure they are trying to improve.
Because they don’t even have a baseline metric.
So how can a pilot prove whether AI is better or worse? Decisions to scale are then made on excitement, demos, or anecdotes. And teams drop promising AI interventions because there’s no meaningful way to measure them.
It’s like watching a home renovation show that only shows you the after shot. Without the before, there’s no way to judge how well the hosts did. Plenty of government services are measured. The Government of Canada, for example, publishes service standards for more than 1,000 of its services and reports whether it meets them. But a service standard measures the service as a whole. An AI tool usually sits at one step inside it, and that step rarely has a before shot.

Some people push back on this. “If you know the service, the problem is obvious. We don’t need a baseline for that. Just fix it.”
A visible problem is a symptom. It shows that something is wrong, but not whether AI fixes it, or whether AI simply pushes the problem to a different step in the process. Fix one thing, break another, and the net result is no change, or worse. And without a baseline, the AI takes the blame.
But let’s say you can measure a point in the system against a baseline after AI. And let’s say that metric improves, but the overall output gets worse. That tells us that, while AI might work in that one part, there are other downstream impacts. And maybe what we thought AI was fixing was not the actual problem. So measure both: the step where AI sits and the service end to end.
Why services go unmeasured
How did we get here?
The reasons, I’m sure, are many and complex. But allow me to reduce that complexity to three generalized but useful observations.
No one wants to look bad
We’re human, and humans don’t want to look bad. Departments don’t want to look bad. Governments don’t want to look bad.
And because of this, we fear that quantifying our efforts will reduce the nuance and complexity of those efforts to a soulless and soul-crushing data point that, in relation to other soulless and soul-crushing data points, may not compare very well at all.
We do the measurement, set the baseline, and then… “What, now I have targets? But don’t you know that every case is unique and requires special, human care? You want me to reduce a human connection—this opportunity to build trust with citizens and others—to some hyperoptimized number?”
I hear you and I agree with you. The risk even has a name: Goodhart’s law. When a measure becomes a target, it stops being a good measure.
And yet.
I bought that sleep app. It told me I slept badly and that by all reasonable measures my level of consciousness throughout the day should probably be a step above that of a barely functioning zombie.
Was I a “bad person” because I wasn’t sleeping well? No, of course not. The app was measuring my sleep, not me. And once I had that number, I could see the tool wasn’t the thing that was going to fix it.
Government baselines need the same separation. Measure the service, not the staff.
Start with the end-to-end outcomes, for example, the time from a request to a decision. This avoids comparing individuals while also showing if the work actually improved.
Staff fear what the numbers will be used for
What if a pilot shows that AI does the job better than people? People are scared they’ll lose their jobs. This is a legitimate concern, and one I’ve heard from lots of different public-sector organizations. So, naturally, any effort to set a baseline before a pilot can meet resistance.
Unions take that concern seriously, and for good reason. Unions exist to protect workers, to give employees leverage and rights around conditions, pay, and security.
In one conversation, I heard of a union that had negotiated a ban on all AI tools in the workplace. It’s an understandable response when AI feels like a threat, but it also shuts the door on help for staff who are often overworked and under-resourced.
This is where “measure the service, not the staff” matters most: it gives staff and their unions a baseline they can agree to.
Another public-sector organization I heard about recently took the opposite approach. Before bringing in any AI, their unionized call centre measured how the service performed as a whole: how many calls they handled, how long it took to resolve a complaint, and client satisfaction. Not who was fastest, just how the team did together. Then they brought in a tool to help staff answer questions and resolve complaints. Because they had the before numbers, they could show exactly what changed: complaints that used to take days were being resolved in hours. Everyone got more done and felt more satisfied.
No one agrees on what good looks like
Evaluation criteria are a hobbyhorse of mine. We’ve come across pilot projects that stall at the baseline step not because the department isn’t willing to set baselines, but because they can’t agree on what “good” looks like, or which measures actually matter.
Like, my sleep tracker gave me a score. But was “good sleep” the hours, the deep sleep, the wake-ups, or how I felt in the morning? My watch would chirp and congratulate me on a sleep score of 90, but… I woke up feeling like I’d been flattened by a bus. Obviously the app and I had differing definitions of what mattered and why.

Some people think “efficiency” is the most important metric. That makes sense. There’s little point spending time, money, and effort implementing AI if it doesn’t save a lot more time, money, or effort.
Others may say efficiency’s cousin “speed” is the most important metric. After all, if we can get everything done faster, we’re already saving time and effort. And citizens will probably be happier too.
But if speed is your only metric, and your service depends on “quality,” you risk optimizing your service for the wrong thing. For example, frontline staff in a department might care more about error rates than about how many forms were processed, especially if those faster forms were processed incorrectly, which will create more work for them down the line. Not everyone will agree with that.
There are so many possible measures that could be used to evaluate an AI intervention in a service. Accessibility. Staff experience. Citizen experience. Trust. None of those are wrong. But if a department can’t agree on what’s important and why, teams will struggle to develop meaningful evaluation criteria for what a good outcome looks like.
Start with the baseline
If you’re one of those leaders tasked from above with “doing something with AI to make that program better and more efficient,” or one who has been wrestling with a gnarly process problem and suspects AI could help, I recommend you start with the baseline.
- Map your service end to end and measure the key steps within it.
- Agree with staff on what you will measure and what you will not use it for.
- Run the pilot, compare, and keep measuring after, even if AI is not the solution.
The baseline is the deliverable
And this brings me to my gift to you.
When it comes to evaluating whether AI belongs in your service, the most valuable deliverable is not the answer to that question. It’s the baseline itself.
Even if the AI pilot underdelivers, even if the answer is a giant “NOPE,” that pilot will give you the first honest measure of your service. And that measure can guide investments and decisions moving forward.
Almost better than an Apple Watch.