← All posts
· Normal View Team

What Most Agencies Won't Tell You About AI

Real talk on what AI actually is, where it works, and why projects fail. Two scenarios we know first-hand — an advisory firm's client data, and writing software — and the four questions that predict which side yours lands on.

aillmragdata qualityconsulting

Where this comes from

This is an opinion piece, so it’s worth saying what it’s built on. Two things:

  1. An engagement with an SEC-registered RIA where we profiled every client record before building anything. The numbers in section 4 are ours, measured, published anonymised.
  2. Our own use of AI to write software, daily, across client work — including the parts where it cost us time rather than saved it.

Everything else is judgement, and labelled as such. Where we cite an outside number, we name the source. Where we’re guessing, we say we’re guessing.

One disclosure up front, because this post is partly about incentives: we sell the work described here. We are not a neutral party. What we don’t do is resell anyone’s platform or take referral fees, so we have no stake in which tools you pick — only in whether the project is real. Read the rest with that in mind.

1. The pitch you are getting, and why it has that shape

If you’ve taken three AI vendor calls this year, you’ve heard roughly the same structure each time:

  • A demo on clean, prepared data
  • A number about productivity gains, usually from a vendor-funded study
  • A pilot offer, priced low
  • A logo slide

None of that is dishonest, exactly. It’s optimised for a specific outcome: getting to a signature before anyone looks underneath. And the reason it’s optimised that way is structural, not moral. Software vendors sell software that starts working on day one. That’s the product. Nobody has ever built a category-defining business selling the six weeks of unglamorous preparation that has to happen first, because that work doesn’t demo, doesn’t scale, and doesn’t have a logo slide.

So it goes unsold. And then it goes undone. And then the pilot underperforms, and everyone quietly agrees the technology wasn’t ready.

The technology is ready. That’s not usually the problem anymore.

2. What the thing actually is, stated plainly

Strip the marketing and a large language model is a function that takes text and produces the text most likely to follow it, given everything it was trained on. That is the whole mechanism. It’s a remarkable mechanism — it turns out predicting text well requires something that behaves a lot like reasoning — but it’s worth holding the plain version in your head, because four consequences fall straight out of it, and every failed AI project we’ve seen traces back to at least one:

It does not know anything about your business. Not your clients, your fee schedule, your ticket history, your product catalogue. Whatever it appears to know about you came in through the prompt, in that conversation, and it’s gone afterwards unless you built something to put it there.

It cannot tell “correct” from “plausible.” It is optimised for text that reads right. Text that reads right and text that is right overlap heavily, which is exactly what makes the gap dangerous — the failures don’t look like failures. A wrong answer arrives in the same confident register as a right one.

It has no access to anything. It can’t read your CRM or write to your calendar unless someone builds that connection, with credentials, permissions, error handling, and logging. That’s ordinary integration work. It is most of the actual labour in an “AI project,” and it looks nothing like AI.

It is non-deterministic. Same input, different output, run to run. Fine for a draft email. Not fine for anything a downstream system parses, unless you constrain the output shape and validate it.

Every one of these has an engineering answer. Retrieval for the first, verification and human review for the second, integrations and tool use for the third, structured outputs and schema validation for the fourth. The answers are well understood. They are also the entire cost of the project, and they are what’s missing from the pilot pricing.

3. The thing nobody sells you

Here’s the sentence we keep coming back to:

AI doesn’t fix broken data. It scales it.

A human analyst pulling a client summary by hand hits a record where the advisor field is blank, thinks “that’s odd,” checks another system, and either fixes it or flags it. Their confusion is a quality control mechanism. It’s slow, unlogged, and it depends on someone caring — but it’s real, and it has been silently protecting your outputs for years.

Automation removes it. The model doesn’t get confused. It produces a fluent, confident, well-formatted summary based on whatever was in the field, and if the field was empty or wrong, the summary is wrong in the same fluent, confident, well-formatted way. Now you’re producing bad output at machine speed instead of good-enough output at human speed, and the error rate is invisible because nothing throws.

This is why “we’ll clean the data later” is the most expensive sentence in this category. Later means after you’ve built on top of it, after people have started trusting the outputs, and after the cost of unwinding it has gone up by an order of magnitude.

4. Scenario one — an advisory firm

Concrete version. An SEC-registered RIA, good firm, competent operations team, wanted AI-assisted client summaries and meeting prep. Reasonable goal. The tooling to do it exists off the shelf.

Before building anything, we profiled their CRM. Read-only, through the API, nothing changed, no client data stored. All 1,953 contact records.

The structural health was fine. Their record model was sound, no widespread duplication, the schema made sense. Then:

  • An advisor was assigned on 9.5% of client records.
  • The concept “advisor” was recorded across 4 different fields, depending on who had entered the record and when.

Sit with what that means for the original project. “Summarise this client’s situation for the advisor’s Monday meeting” requires knowing which advisor. On nine records in ten, the field the automation would read is empty. On the ones that are populated, the value might be in any of four places, and nothing declares which one wins.

Two ways that fails, and the second is the bad one:

  1. The automation errors out on the empty records. Annoying, visible, fixable. This is the good outcome.
  2. The automation reads the field, gets nothing, and writes a summary anyway — because that is what a language model does with a gap. It produces a fluent paragraph that doesn’t mention the advisor, or infers one from surrounding context. Nobody notices for a quarter.

There was no AI problem here. There was a “which system owns which fact” problem wearing an AI problem’s clothes.

What we actually did: handed over the findings and a worklist ranked by what each fix would unlock, then built the thing the data was for — a scheduled Wealthbox-to-Outlook contact sync running in the firm’s own Azure tenant, with advisor routing, incremental updates, failure handling, and a no-client-data-in-logs guarantee enforced by automated tests. Phased, billed hourly, each piece delivered before the next was scoped.

The unglamorous read of that engagement: the highest-value thing we delivered was a document telling them what was wrong. No vendor sells that document, because it is cheap and it sometimes concludes you shouldn’t buy anything yet.

On the industry backdrop, for what it’s worth: Schwab’s 2026 RIA and AI study put adoption at 63% of firms — more than double 2023 — with around one in ten having genuinely integrated it into how the firm operates. We would read the gap between those two numbers as roughly the size of the problem described above.

5. Scenario two — writing software

The other domain we know first-hand, and a useful counterweight, because this is the one where AI genuinely and unambiguously works. Understanding why it works here tells you almost everything about whether it will work for you.

What it is actually good at, in our daily practice:

  • Code that is mostly determined by its surroundings. A tenth API endpoint that resembles the nine that already exist. The model has everything it needs, sitting right there in the repo.
  • Mechanical transformations across many files. Rename a concept, migrate a call signature, update forty imports.
  • Tests for code that already works and whose behaviour you can state.
  • Explaining unfamiliar code. It is a very good reading assistant.
  • First drafts you intend to rewrite. Getting to something concrete to react to is faster than starting from nothing.

What it is bad at, same practice:

  • Anything where the right answer isn’t inferable from the surrounding code. Architecture with real trade-offs; decisions that depend on business context the repo doesn’t encode.
  • Debugging where the cause lies outside the files you handed it. We spent three days on a Shopify bug where every storefront page returned a 500. The cause was an undocumented platform ceiling on Liquid render complexity — invisible in the source, absent from the docs. No amount of reading our code would have found it. It took a probe script and fifty measured page loads. The full write-up is here, including the four hypotheses that turned out to be wrong.
  • Knowing when it is out of its depth. It will produce a confident answer at the edge of its competence in exactly the register it uses when it’s right.

Now the part that matters, and the reason this section belongs in a post about your business:

Programming is the best-case domain for AI because it comes with a free, instant, automatic verifier. The compiler rejects it. The type checker rejects it. The tests go red. The linter complains. A wrong answer gets caught in seconds, by a machine, at effectively zero cost, before it reaches a human.

Almost nothing else in your business has that. “Summarise this client’s situation” has no compiler. “Draft this response to the customer” has no test suite. When people say AI works better for engineers than for other departments, this is the actual reason — not that engineers are better at prompting. They have a verifier and you don’t. So the engineering question for any AI project outside code is: what plays the role of the compiler? And if the answer is “a person reads it,” has that person’s time been budgeted, because it isn’t free and it doesn’t scale.

Two more honest notes from the software side, because they generalise:

The demo is 80% in an afternoon. The last 20% is where the schedule lives. That ratio isn’t specific to AI — it’s the oldest observation in software — but AI makes the first 80% so fast that it distorts everyone’s estimate of the rest. The plausible-looking prototype is the least informative artifact in the entire project.

Reviewing code you didn’t write is slower than writing it. If you generate more than you can review, you haven’t saved time; you’ve deferred it into a form that is harder to pay back. The teams getting real leverage from AI are, without exception, the ones who could have written the thing themselves and are using it to go faster. It is an amplifier, and amplifiers work on whatever signal you feed them.

And the punchline that ties this section back to the last one: the codebases where AI works well are the ones that were already well-organised. Consistent conventions, real types, tests that mean something, small modules with clear boundaries. Same lesson, different noun. In an advisory firm the constraint is data structure. In a codebase it’s code structure. Either way the model’s ceiling is set by the quality of what sits underneath it, and nobody sells the work of raising that floor.

6. The four questions

Both scenarios reduce to the same diagnostic. Before you spend money, ask these about the specific task — not about “AI” in the abstract, which is not a thing you can evaluate.

1. Can something check the answer automatically? A compiler, a schema, a checksum, a reconciliation against a second source. If yes, you’re in the good case and you should be aggressive. If the only check is a human reading the output, you have a review cost that scales linearly with volume, and it belongs in the budget and the headcount plan. If there is no check at all, you are not automating a process — you are removing the last person who would have noticed.

2. Is the input authoritative? Is there one place that owns each fact, and does everything agree on which place that is? If the same concept lives in four fields, no model resolves that for you. It picks one and sounds confident. This is the question the RIA engagement turned on, and the one most often skipped.

3. Is the task bounded? “Extract these six fields from this document type” is bounded — you can enumerate the ways it fails and test them. “Handle customer emails” is not. Bounded tasks ship. Unbounded ones become a permanent research project with a monthly invoice attached.

4. Does it fail loudly? When it goes wrong, does something turn red, or does a slightly-wrong answer flow downstream and get trusted? Silent failure is the characteristic failure mode of this technology, because the output is fluent regardless. Anything worth building logs what it read, what it wrote, what it skipped, and what it wasn’t sure about. If a vendor’s answer to “how do I know when it’s wrong” is a confidence percentage with no worked example of what you do at each threshold, that is not an answer.

Then one more, a business question rather than a technical one: what does being wrong once cost? Marketing copy, essentially nothing — ship it. A number in a client statement, a compliance-relevant record, anything a regulator might read: the economics are completely different, and so is the amount of verification you should be paying for.

Score your project honestly. Four yeses, go fast. Two or fewer, and it isn’t an AI project — it’s a data project that hasn’t admitted it yet.

7. How to buy this without getting taken

Plainly, and at the risk of describing our own sales process:

Ask for the failure mode before the demo. “Show me what it does when the input is bad” is the most informative question you can ask a vendor, and the reaction to it tells you as much as the answer.

Don’t start with the flagship use case. Start with a bounded, checkable, low-blast-radius task — not because it’s impressive, but because it’s the cheapest way to find out what’s actually wrong with your data. Something is.

Buy the assessment separately from the build. If one fixed-fee proposal covers both, the assessment cannot honestly conclude “don’t build this yet.” Anyone who scopes an entire project before looking at your data is quoting a guess.

Insist on your own tenant, your own credentials. Your data should sit in infrastructure you own, and the integration should keep working if you fire the people who built it. If a vendor’s architecture requires your client data to route through their cloud, you should get a real answer about why.

Be suspicious of any number without a denominator. “40% faster” — at what, measured how, against what baseline, over how many trials? We published a fifty-trial probe script for a single storefront bug. Ask for the equivalent.

8. Where we’d tell you not to bother

Because “no” is more useful than a menu:

  • Don’t build a chatbot over documentation nobody reads. If the docs are wrong or stale, you’ve built a faster way to distribute wrong answers.
  • Don’t automate a process nobody has written down. If you can’t describe the current process in a page, there is nothing to automate — there’s a process design problem, and it’s cheaper to fix on paper.
  • Don’t buy AI to avoid hiring. It reduces the cost of things you already know how to do. It does not supply judgement you don’t have on staff.
  • Don’t start with the messiest, highest-stakes workflow because it has the biggest theoretical payoff. That’s where the data is worst and where being wrong costs the most. It’s the last thing you should do, not the first.

The short version

The technology works. The demos are real. The failures are almost never about the model.

They’re about the input being wrong, the output being unverifiable, or the task being unbounded — and all three are knowable in advance, cheaply, before you commit to anything. What most agencies won’t tell you is that this diagnosis takes a couple of weeks, sometimes concludes you shouldn’t build anything yet, and is worth more than the pilot they’re proposing.