AnswersAI and your data
Why your AI can’t answer questions about your own data
Rob CollieFounder and CEO, P3 Adaptive
Because your data has the answer. It just isn’t ready to give it. To answer “why did margin drop in the Southeast,” an assistant needs a semantic model – something that says what your tables hold, what each measure calculates, and which of your four revenue numbers is the one your CFO means. Most companies either don’t have that, or they have one built for a human clicking through a report. Those are not the same thing, and the difference is exactly why the answers come back vague, wrong, or refused.
The sentence my whole book turns on
I spent most of 2024 quietly worried. Few lines of work are more exposed to AI than telling computers what to do for a living, and I run a data and BI consulting firm. So I did what I usually do with new technology, which is hang back and wait for someone to explain why it matters. Then it got to the point where hanging back felt like the risky move, and I went and dug in.
What I found surprised me enough that I wrote a book about it, and the whole thing turns on one sentence: AI success isn’t really an AI problem. It’s a data problem.
This page is the concrete version of that sentence. It’s what I mean when I say it, and it’s the single most common reason I see companies conclude that AI “isn’t ready” when the thing that isn’t ready is sitting on their own servers.
You already know how this goes
Licenses get bought. The assistant gets switched on. Somebody asks it a real question about the business and gets one of four things back:
- A confident number that doesn’t match the report anybody actually trusts.
- “I don’t have access to that data” – for data the person asking can plainly see in Power BI.
- A generic explanation of what margin is, rather than what yours did.
- Something correct, exactly once, that nobody can reproduce the following week.
Six weeks later usage is flat and somebody senior says the AI isn’t ready. That conclusion is wrong and it’s expensive, because what usually happens next is the licenses get cancelled while the actual problem – which is fixable, and which you’d want fixed either way – goes untouched for another year.
Four things have to be true, and most places have two
An assistant answering a question about your business is doing four separate jobs. They fail independently, they fail quietly, and nothing in the error message tells you which one broke.
It can reach the data
There’s a connection between the assistant and the system holding your numbers. This is the one everybody checks, and it’s the one that matters least.
It runs as the person asking
The query executes under that individual’s identity, so their row-level security applies. If it’s running under a service account instead, everybody sees everything – and you won’t catch it, because the answers look perfectly fine to whoever is doing the testing.
It knows what the fields mean
Not the column names. The meaning. That Amount is net of returns. That your fiscal year starts in October. That intercompany is excluded here and included over there. This is where most estates fall down.
It knows which answer is the right one
When three measures could all reasonably be called revenue, something has to say which one is authoritative and which one somebody left in place in 2019. Without that, the assistant picks. And it picks confidently.
One and two are engineering. They get solved, because they’re visible and somebody owns them. Three and four aren’t engineering at all. They’re a written description of what your business means by its own words, and no vendor can ship that to you, because there’s no general version of it to ship. Yours is yours.
Is this Copilot’s fault?
No, and I’m about the last person who’s going to pile on Microsoft here. I ran the team that owned BI in Excel and designed most of what became Power Pivot. Power BI’s semantic model is still the best in the business, and it’s why we’ve built this company on it since 2013.
Point ChatGPT or Claude at the same estate and you’ll get the same nonsense for the same reason – all three are reading the same missing description of your business. Copilot over a well-tended Power BI model works. The break is upstream of whichever assistant you happened to pick, which is also why switching assistants doesn’t fix it. I’ve watched more than one company spend a year learning that the expensive way.
Your model was built for a different reader
This one lands badly in the room sometimes, so I’ll just say it plainly. Your semantic model is probably fine. It’s fine for people.
A human opening a report brings context the model never had to carry. They know which slicer to set. They know to ignore the two test measures. They know that when the VP says revenue he means the net one. They repair the model’s gaps every single time they use it and never notice they’re doing it.
An assistant brings none of that. It reads exactly what’s written, and what’s written is usually a decade of shorthand. I’ve had my hands in a lot of these models over the years, and here’s the shape of what I find:
What the model says today
Sales_Amt_2
No description.
Sitting right next to Sales_Amt and Sales_Amount_New, both still live.
Nothing anywhere says which one the business actually uses.
What it needs to say
Net Revenue
“Gross sales less returns and trade discounts. Excludes intercompany. Fiscal calendar, October start.”
Graded authoritative. Owner: Corporate FP&A. Reviewed March 2026.
The other two marked superseded, so nothing reaches for them.
Now multiply that by four hundred measures, and you have your real answer for why the assistant is useless. It isn’t reasoning badly. It’s reasoning correctly over a description of your business that nobody ever wrote down.
A person opening a report repairs the model’s gaps without noticing they’re doing it. An assistant reads exactly what is written – and what is written is usually a decade of shorthand.
“Fine, we’ll point it at the warehouse instead”
Please don’t. This is the common next move and it makes everything worse.
Hand a capable model a pile of raw tables and it will write confident SQL against columns it has misread, join on keys that look about right, and hand you back a number with no indication that anything was guessed at. You’ve just removed the one layer that was catching those mistakes.
Here’s the reversal I’d want you to take away from this whole page. The semantic model used to be a convenience for report builders – useful, unglamorous, the thing you did so the visuals would work. It is now the most valuable asset in your data estate, because it’s the only place your business’s meaning is written down in a form a machine can read. Companies that invested in one are roughly eighteen months ahead and most of them have no idea.
Four things you can check this afternoon
Diagnostic
Which of the four layers is actually broken?
No tooling required. Each takes a few minutes and tells you something different.
Ask it something you already know the answer to. Then ask the same thing again, worded differently.
Two different numbers is layer four – nothing is adjudicating between competing measures. One number that’s wrong is layer three; it found a measure and misunderstood it.
Have two people with deliberately different permissions ask the identical question.
Different answers is correct and healthy. Identical answers means layer two is broken, the query is running as a service account, and you have a disclosure problem that has nothing to do with AI. Go check this one first.
Open your biggest semantic model and count how many measures carry a description.
It’s usually a rounding error away from zero, and that number predicts how useful an assistant will be against that model almost by itself.
Search your measure list for “revenue.” Then “margin.” Then “active.”
Count the hits, then ask whether anything in the model says which one is authoritative. If that answer only exists in somebody’s head, the assistant is going to guess – and it will not mention that it guessed.
What fixing it actually takes
Less than people fear, and it is not an AI project. It’s model work: naming, descriptions that carry the business rule instead of restating the column, measure hygiene, relationships that are correct rather than close enough, and a grading pass that says which calculations are authoritative and who owns them.
For one well-used model that’s typically a couple of weeks, not a quarter. And here’s the part I like: none of the output is AI-specific. A model with its rules written down is easier for your own analysts to use, survives the person who built it taking another job, and stops producing two versions of the same number in two different meetings. Most companies should have done this years ago. AI didn’t create the problem – it just made the omission visible and expensive.
What I’d steer you away from is starting a platform program. The instinct after a failed rollout is to go back to foundations and spend two years on pipes. Take one model your people already argue about, make it right, point an assistant at it, and let what they reach for tell you what to fix next.
And a right answer is only half of it
Most people picturing this stop at the chat window – ask a question, get a number you can trust. That part has to work first, and everything above is about getting it to work. It isn’t the part that changes how your company operates.
What changes things is what happens to an answer worth keeping. It becomes a report. A real one, built by the person who had the question instead of requested from somebody who didn’t. And because those pages get authored as HTML rather than assembled out of a fixed set of visuals, they aren’t a thinner version of what your BI team makes today – they’re routinely better, and a lot more specific to the thing being looked at, than any report canvas has allowed in a decade. Why that is, and why sharing them safely turns out to be the hard part, is a longer story and I’ve written it down separately.
Easier to show you than keep typing.
We built Faucet because it’s how we wanted to work: your existing models connected to the AI client your team already has open, every question running as the person who asked it, and the answers worth keeping published as live reports. The sandbox is open – real data, real questions, no sign-up.