Here is the meeting that's happening right now in a few thousand companies.
A board member forwards an article. ChatGPT and Gemini have each passed a billion users. The email says four words: "Where are we on this?" The CEO asks the CTO for a plan by Friday. The CTO already knows something the board doesn't: last month's release needed a rollback, the staging environment hasn't matched production since March, and the two people who understand the billing service are both on holiday in August.
So the CTO does the rational political thing. They promise a chat assistant in the product by Q4.
That decision will cost the company twice. Once to build the AI feature. Once to rebuild the base underneath it when the feature exposes every weakness the team has been quietly working around for three years.
Mass adoption didn't just raise urgency. It raised the bar.
The billion-user number is usually read as a deadline. It's more useful read as a benchmark.
Before, if your internal tool took six seconds to return a search result, users compared it to your old tool. Now they compare it to something they used on their phone twenty minutes ago. That thing started answering in under a second. It streamed text as it thought. It handled a typo, a half-finished sentence, and a photo of a receipt without complaining. When it didn't know, it said so and offered something adjacent.
A billion people have been trained on what good feels like. They are not going to grade your AI feature on a curve because it's your first one.
This is the part boards miss. The headline creates pressure to ship something. But the same headline is what makes shipping something mediocre worse than shipping nothing. An AI feature that's slow, wrong, and forgetful doesn't read as "early version". It reads as "this company can't do software". You spend budget and lose trust in the same quarter.
The four things a good assistant does that most enterprise stacks cannot
Strip away the model and look at what a modern assistant actually demands from the system behind it. Four requirements, and most legacy platforms fail at least three.
One: a sub-two-second budget, end to end. Not for the model call. For everything.
User presses Enter
├─ 40ms TLS + edge routing
├─ 120ms auth check (JWT verify, cached)
├─ 250ms fetch user context (permissions, tenant, entitlements)
├─ 550ms retrieval: embed query → vector search → rerank
├─ 150ms prompt assembly + token accounting
├─ 700ms time to first token from the model
└─ then stream to completion
The model is the one line you can't optimise much. Everything above it is yours. If your auth check is a live database round trip because nobody added a cache, that's 400ms gone. If "fetch user context" means four sequential internal HTTP calls, that's another second. If your search is a LIKE '%query%' over a table with forty million rows and no covering index, you're finished before the model has been asked anything.
Teams discover this in week three of the build and try to fix it with a bigger instance. That doesn't work. Sequential latency is an architecture problem.
Two: streaming. The perceived speed of every assistant people love comes from tokens appearing immediately. The application code for this is trivial:
export async function POST(req: Request) { const { messages } = await req.json(); const completion = await client.chat.completions.create({ model: "gpt-4o-mini", messages, stream: true, }); return new Response(toEventStream(completion), { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", "Connection": "keep-alive", "X-Accel-Buffering": "no", // nginx buffers your stream otherwise }, }); }
The code is the easy part. The hard part is that your infrastructure was built to buffer complete responses. Common blockers: an nginx or ALB layer that holds the body until it's finished, an API gateway with a hard 29-second timeout, a web application firewall that inspects the full payload before forwarding, a serverless runtime with a response size cap, a CDN that strips text/event-stream. Every one of those is a config change owned by someone who isn't on your product team.
That's why "add a chat box" turns into a six-week negotiation with infrastructure. Nobody put that in the Q4 plan.
Three: it writes back, safely. A chat widget that only reads data is a demo. Useful AI takes action: cancels the order, reschedules the delivery, updates the record. Which means the model's output now reaches your write path.
Model calls fail. Networks drop mid-stream. Users double-tap. So retries are guaranteed, and retries on a non-idempotent write path mean duplicated records and double charges.
// Every AI-triggered write needs a key the model does not control. async function executeAction(action: Action, ctx: RequestContext) { const key = `${ctx.conversationId}:${ctx.turnIndex}:${action.name}`; return db.transaction(async (tx) => { const existing = await tx.actionLog.findUnique({ where: { key } }); if (existing) return existing.result; // replay, don't re-execute assertPermitted(action, ctx.actor); // authorise against the user, not the model const result = await handlers[action.name](action.args, tx); await tx.actionLog.create({ key, actor: ctx.actor.id, action, result }); return result; }); }
If your system can't produce that log, you can't answer the only question that matters after an incident: what did the AI do, to whose data, and on whose authority. We wrote about why governance has to be code rather than a policy document - this is the concrete version of that argument.
Four: it changes behaviour without a deploy. Your app is deterministic. Same input, same output, and a test suite that pins it down. A model is not. The provider ships a new version, your prompt drifts, someone adds a line to the system message, and the output shifts.
So the AI layer needs a different kind of test. Not "does the function return 4", but "does the system still get the right answer on 200 known cases".
// evals/order-intent.test.ts const golden = loadCases("./cases/order-intent.jsonl"); // 200 real, anonymised inputs test("intent extraction holds above threshold", async () => { const results = await Promise.all(golden.map(runPipeline)); const exact = results.filter((r, i) => deepEqual(r.parsed, golden[i].expected)); const accuracy = exact.length / results.length; // Fail the build on regression, not on imperfection. expect(accuracy).toBeGreaterThan(0.93); });
Two things matter here. You need a threshold, not perfection. And you need real inputs, which means you need clean logs of what users actually typed. Teams with no usable production logging can't build this set. That's usually the first sign the foundations aren't ready.
The contrarian bit: chat is the worst first AI feature
Standard advice says start with a chat assistant, because that's what everyone recognises. That advice is backwards.
Chat is the hardest interface in software. It accepts unbounded input, so you can't scope the problem. It implies memory, so you inherit state management. It invites the user to ask about anything, so every gap in your data becomes a visible failure. And a billion people have a reference point for how it should feel, so the quality bar is set by a company with a research lab.
The strongest first AI features are narrow and boring. One button, one job, one measurable outcome. Draft this reply. Categorise this ticket. Extract the fields from this PDF. Match this candidate to these roles. You control the input, you can evaluate the output, and you can put a number on the saving.
Narrow features also let you attack the foundations honestly. Getting one document pipeline fast, observable and idempotent forces you to fix caching, queues and logging. Those fixes pay off across the whole platform. Chat, by contrast, spreads a thin layer of AI over everything and fixes nothing.
There's a second reason to go narrow. Broad AI features need broad context, and enterprise systems are too tangled to hand to a model wholesale - the context window problem is a hard limit, not a temporary one. Narrow scope is how you stay inside it.
The honest test
Before committing to a date, answer these. Not as a wish list - as a status check on what exists today, in production.
Can you name the p95 latency of your three most-used API endpoints without opening a dashboard nobody trusts? Can you ship a change on a Wednesday afternoon without a rollback plan involving a phone tree? Do you have three months of clean, queryable logs of real user input? Does your write path survive a duplicate request? Can a developer stand up a realistic environment locally in under an hour? Does someone own the data model, or has it been shaped by whoever needed a column most recently?
Six honest yeses, and you're ready to build. You should move now, because the window where an AI feature is a differentiator rather than table stakes is closing.
Three or fewer, and the AI project is not an AI project. It's a stabilisation project wearing a nicer suit. Run it as one, on purpose, with the AI feature as the stated goal and the fixes as the visible path to it. That framing is what gets stabilisation work funded - boards approve AI budgets, not refactoring budgets.
What this actually means for next quarter
The billion-user headline is real pressure, and it's not wrong. Your customers, your staff and your board all now assume AI competence the way they assumed a mobile app in 2012. Deferring is a decision, and it's getting more expensive.
But urgency and readiness are separate questions, and conflating them is how companies pay twice. The right response to the headline isn't a chat widget by Q4. It's two weeks of honest diagnosis, then one of two plans: fix the base, or ship the feature.
We do that diagnosis for a fixed fee, in two weeks, and you get a red/amber/green report you can hand to your board. Sometimes the answer is "your foundations are better than you think, go build". Sometimes it's "fix these four things first, and here's the order". Either way you stop guessing, which is worth more than another quarter of promising Friday plans.
Stabilise first. Improve second. Add AI last. The order isn't caution. It's the only sequence where the AI still works six months after launch.
Book a Free Rescue Call
