A new report from MIT Media Lab’s Project NANDA—The GenAI Divide: State of AI in Business 2025—looked at hundreds of real deployments and found a harsh truth: about 95% of generative-AI pilots don’t produce business impact. Only ~5% translate into revenue or real savings. The problem isn’t “AI is bad.” The problem is how we build and run it.

The short version: we’re trying to run AI agents like old, deterministic software. That doesn’t work. AI is probabilistic; outputs vary; providers change models; quality moves if you don’t measure it. You need a different playbook—one built around evaluations, simulation, guardrails, and cost/quality routing. Even OpenAI’s leadership keeps saying the quiet part out loud: evals are the work.

Where AI projects go wrong—and how to fix them

The playbook we use at Tastefully.ai

We build agents like systems, not stunts. Four pillars make the difference.

1) In-house Router (quality-, cost-, and latency-aware)

• Task-aware selection. The router chooses the smallest model that can pass the task’s acceptance criteria; escalates only when needed.

• Version control. We can pin model/provider versions and re-run evals on every change.

• Cost guardrails. Token caps per step, caching, and batch policies keep spend predictable.

• Quality arbitration. For high-stakes steps, the router can cross-check with a second model or a verifier before acting.
  This turns “black-box AI” into managed, measurable behavior instead of hope. (Usage-based pricing is real; routing is how you keep it sane.)

2) In-house Simulator (our “agent gym”)

• We rehearse real tasks end-to-end: multi-page flows, tool calls, dead ends, and recovery paths.

• The simulator runs deterministic replicas and scripted obstacles so we can measure repeatability and catch regressions.

• We keep an evaluation logbook (by scenario and version) so production changes never outpace quality checks.
  Benchmarks like WebArena and REAL show why deterministic web/task replicas matter; we’ve built our own for your workflows.

3) Spec-and-Tickets Generator that isn’t “just a template”

• It listens to real work (docs, meetings, tickets) and proposes diffs to your spec with clear rationales.

• Tickets come with acceptance criteria, dependency notes, and confidence tags; low-confidence items auto-escalate for review.

• Every suggestion is trace-linked to the evidence that justified it (meeting excerpt, doc section, prior decision).
  Result: less “AI slurry,” more auditable decisions your team can trust.

4) Dynamic UI/UX Generator

• Agents don’t leave you with raw JSON. They generate small, adaptive interfaces—fields, tables, and checklists matched to the task.

• UI elements are typed and validated (think: schema + state machine), so clicks and updates are idempotent and traceable.

• This is how we turn probabilistic text into predictable actions your team can review and approve.

“Deterministic” behavior—without pretending AI is deterministic

Truth: you can’t make a large language model fully deterministic in the wild. Nature-level research shows that hallucination risk never vanishes; you manage it.

What you can do is box in the variability so outcomes are repeatable enough for production:

• Constrain the task (tools, scopes, allowed actions).

• Define acceptance thresholds per step; fail closed.

• Use retrieval/grounding for facts; verify or defer on low confidence.

• Route smartly, pin versions, and re-eval on change.

• Wrap actions in typed UI/state machines so the end effect is consistent.

That’s how we get “deterministic-feeling” agents—reliable, auditable, and ready for real work.

Why this matters now

Two things are true at once:

1. Most pilots fail—not for lack of ambition, but for lack of the right operational model.

2. Evals + simulation + routing are the difference between a cool demo and a durable system. This is fast becoming industry consensus

If you’re done with pilot theater and want production-grade agents—with privacy, quality, and cost under control—we should talk.