AnswersAI agents
Why most AI pilots never make it to production
Rob CollieFounder and CEO, P3 Adaptive
Most of them never had one real job to do. A pilot that starts from a tool spends its time looking for a use. The ones that reach production start from a specific piece of work that a specific group of people does every week, and they get judged on whether that work got easier. Pick the job first and most of the other decisions get a lot simpler.
I hear some version of this story all the time. The company bought licenses. There was a kickoff. A working group met for a couple of months, somebody built a demo that got real applause, and now it’s the following quarter and nobody can say what the thing is for.
Nobody in that story did anything dumb. Everyone acted sensibly on what they knew. The whole effort just started at the wrong end.
How most pilots actually start
Look at where the effort went. Somebody chose a tool. Then somebody else was asked to find uses for it. That second step is the problem, because it turns a purchase into a scavenger hunt.
You can hear it in how people talk. “We bought Copilot, now what?” “What are our AI use cases?” A committee gets formed to answer that, and committees are good at producing lists. The list has twenty items on it. Each one gets a little attention and none of them gets enough to actually work.
The demo makes it worse, oddly. A good demo runs on clean sample data, driven by someone who knows exactly which questions to ask. It impresses the room. Then it meets a Tuesday in accounts payable and falls over, and the people who clapped quietly stop bringing it up.
The tools themselves are fine. Claude, ChatGPT and Microsoft 365 Copilot are all very capable, and we build agents in all three. Which one you picked matters much less than people think. I’ve watched companies swap one for another after a stalled pilot and then stall again in the same spot, for the same reason.
Start from one job with a name on it
The pilots I’ve seen reach production had something in common from the first week. Somebody could point at a piece of work and say who does it and how often. The weekly branch recap that eats someone’s Friday afternoon. Vendor questions that land in a shared inbox and sit for two days until the one person who knows where the contract lives gets to them.
That’s the job. It’s small on purpose. Small is what lets you tell whether it worked.
Pick one real job
A specific task a specific group does every week, with an owner who will tell you honestly whether it got better. If you can’t name that person, keep looking.
Build the agent for that job
Give it the data and documents that job needs, plus a way to hand the result to wherever it goes next. An agent scoped to one job is a lot easier to get right than one that’s supposed to help with everything.
Prove it on real work
Run it on live data, with the people who actually do the job, for a few weeks of their normal rhythm. Sample data will tell you it works. Real work tells you where it breaks, and that’s what you need to know.
Then widen it
The second job is cheaper than the first. The data is already connected and you know how your people react to working this way. By then you’ll have an opinion about what to take on next, and it came from evidence instead of a brainstorm.
“Agent” covers a lot of ground here. Some answer questions from your data and documents. Some move work between systems: read the email, check the order, update the record, draft the reply. The approach doesn’t change. One job first.
The agent is only as right as what it reads
This is what sinks pilots after they’ve picked a good job. The agent gets built, it runs, and the answers are a little off. Off enough that people double-check everything, which means it saved nobody any time.
When I dig into one of these, the model is almost always doing its job fine. It’s reading data whose meaning was never written down. Three measures that could all be called revenue. A status field where “closed” means one thing in the East and another in the West. A folder of policy documents where this year’s version and the 2021 version sit side by side with nothing to tell them apart.
A person sorts all of that out by instinct. An agent reads exactly what’s there. I wrote a whole page on why your AI can’t answer questions about your own data, so I won’t repeat it. The short version for a pilot: when you pick the job, look hard at what the agent will have to read. Fixing the definitions one job depends on is a manageable piece of work. Trying to fix them for the whole company first is how you end up in a two-year program.
Check who the agent runs as while you’re at it. If it answers under a service account, everyone who uses it sees everything it can see. That’s fine in a demo. It’s a real problem the first time somebody in sales asks it about payroll.
The half nobody sells
Say the agent works. Right answers, correct permissions, quick. You can still end up with a stalled pilot, because nobody changed how they work.
Vendors skip this part, and I understand why. There’s nothing in it to license. People have a way of doing their job that they’ve refined over years. The spreadsheet they trust. The colleague they ask. An agent asks them to give up something familiar for something they haven’t seen work yet, and busy people make that trade slowly, if at all.
I spent a lot of my years at Microsoft, leading BI in Excel and then designing much of Power Pivot, believing that a good enough tool gets used. It mostly doesn’t work that way. People change how they work when someone they trust shows them a better way on their own work.
So what helps is pretty boring. Put the agent where the work already happens, in the chat window or the inbox they already have open. Get it in front of the people who do the job and let them tell you what’s wrong with it. Fix those things fast, where they can see you did. And have the person who owns the work use it in front of their team, on a real week.
Pick the job first. A pilot that starts from a tool spends its whole life looking for one.
How to tell yours is stalling
These show up well before anyone says the word out loud.
- Nobody can name the job in one sentence, or the answer comes back as a list.
- Success is measured by how many people logged in. Nobody is checking whether the work got faster.
- The people running the demos are on the project team. The people who do the job haven’t touched it.
- Every status update is about the next phase.
- People check its answers against the old report every single time, and nobody has asked why.
Any one of these is fixable. Two or three together usually mean the pilot is waiting for somebody senior to end it quietly.
Four things you can check this afternoon
Diagnostic
Is your pilot pointed at a real job?
None of these needs a tool or a meeting. Each takes a few minutes.
Write down the job your pilot does in one sentence, including who does that job today.
If it took more than one sentence, or the “who” is everyone, the pilot is still looking for a job. Fixing that starts with a conversation with whoever owns the work.
Ask three people who actually do that job when they last used the agent, and for what.
Specific answers from last week are a good sign. Vague ones, or “I tried it once,” mean the adoption half hasn’t started, however well the agent works.
Give it five real requests from last week, the messy ones, and compare each result with what your team actually did.
Wrong answers usually trace back to the data and definitions it read. Right answers that took longer than doing it by hand mean it’s scoped too broadly or sitting somewhere people don’t work.
Have two people with different permissions ask it the same question.
Where their access differs, their answers should too. Identical answers mean it’s running as a service account, and that needs fixing before anyone widens the rollout.
Where I’d start
Skip the use-case inventory. Skip the steering committee, at least for now.
Pick the one job where the pain is obvious and the person who owns it will be honest with you. Look at what the agent will have to read, and fix the definitions that job depends on. Build it in whatever your people already have open, whether that’s Claude, ChatGPT or Copilot. Then put it in front of the people who do the work and watch what happens on a real week.
You should have something working on real work within weeks. If you don’t, you learned something useful anyway, and it cost a lot less to find out than a year of committee meetings would have.
Got a job in mind? Let’s look at it.
Bring the one piece of work you’d most like an agent to take on. We’ll tell you whether it’s a good first job and what the agent would need to read to get it right. If it’s a bad fit, we’ll say so.