#8 - Why AI Projects Fail
Are you sometimes wondering, why your AI adoption stalls, why scaling pilots fails and efficiency gains do not materialize?
This is I am a Teapot, your weekly newsletter about #Engineering #Digital #Ideas and this week, I got some ideas answering the above that I would to bounce off off you.
Why do AI projects fail?
Let's kick things off with a quick reminder: LLM based AI is probabilistic. This translates to it never being right, only probably right. AI is guessing both your question and the answer. It got really good at guessing, but it guesses. Which implies the answer might be wrong.
Being probabilistic means a second thing: Remember stochastics? The more often you flip a coin, is it more or less likely it shows head every time? The answer is less. Similarly AI's probability to be right degrades with the amount of steps or size of a task.
Despite frontier models training on more data, more often and getting better architectures with better harnesses, these are the fundamental issues with any LLM based AI. From them we can derive two bounding factors for an AIs performance in a certain context.
First factor: Verification
Only probably right - so, if you want to be right for sure - because your LLM is selling cars, offering medical advice, writing code for Nuclear power plants - you need verification: An extra check on top.
Sometimes verification is easy: "One plus one is?" -"Two." , so easy your calculator can check, "The meaning of life?" -"42." But only in the specific context of a Douglas Adams book, in general the space of possible answers is so vast, verification is impossible. Or other cases, such as rocket engines, drug development or marketing dashboard, where verification is possible, but takes very long or is very expensive.
So, this is the first bounding factor of AI performance: The ability and complexity of verifying it being right.
Second factor: Integration
Less right for larger and longer tasks. Easy, you say. No one shot mega-prompts, we will just use the LLM itself to make a nifty plan of many small steps. Great! Now you got a new problem: Putting it all to work and back together: Integration.
Yes, there are cases where this is easy. A small amount of steps and a low degree of interdependence and a small size and low complexity of results might be indicators for that. Extending the example above: "One plus One" plus "One plus One", is "Two" plus "Two", is "Four". A few steps, fully independent, easily integrated. But many AI applications are much more complex than that.
And this is the second bounding factor of AI performance: The ease of compartmentalization and integration.
There you go, two easy to reason about predictors of an AI projects success: Ease of verification and ease of integration. Before we have Auf Wiedersehen in part II let us quickly discuss two modifiers to these factors.
First modifier: Right enough being enough
While there certainly are many cases where ultimate correctness matters, there are many others where we can get away with probably right quite easily.
Either, because we "only" need to be better than existing models: Weather forecasting, identifying skin cancer or predictive analytics come to mind. Or because correctness is not a relevant category for judgment at all: Each and every artistic image, video, audio or text generation, just needs to be "good enough".
Forcasting the weather, identifying cancer or predicting demand with 85% instead of 75% probability to be right, is quite an improvement. And, nobody ever claimed that starship, beautiful beach or Santa Claus was real in the first place.
Second Modifier: A little AI being enough AI
While a lot of the hype around Agentic systems is going all in on AI use, releasing the self improving sorcerers apprentice spirits and calling it a day, many tasks actually do not need to be done by AI only.
You can actually get quite far with classic automation and algorithms for many aspects of a task, only calling AI for those tasks it truly is the best or only option. Similarly you can make use of AI to whip up a script or small tool to do the job. All of a sudden, the above problems only need to be solved once, for the tool being created by AI, that will then happily and non-deterministic ally will do its job.
It's about using the best, not the shiniest tool for the job.
So here you go: Ease of verification and integration are good predictors for AIs performance at a task. "Right enough being enough" and "A little AI being enough" are modifiers you can use to your advantage.
This is it, a rough cut. Would love to get your feedback. Both on the ideas and format. Check back next week for part II - Greenfield's, Brownfields and Everything Else you need to know for a reasonable approach to AI.
On to this week's reading list:
ENGINEERING
Production Engineering When Trading Billions
Agents Need Control Flow
DIGITAL
Super Intelligence - The idea eating smart people
"The AI has all the attributes of God: it's omnipotent, omniscient, and either benevolent (if you did your array bounds-checking right), or it is the Devil and you are at its mercy."
The New AI Superpowers: Focus and Follow through
IDEAS
Taking a Walk May Lead to More Creativity than Sitting, Study Finds
Don't use your phone while you poop
Enjoy your tea.
\\[ ]]