How to Build an AI Agent That Works

LangChain’s survey of teams puts the number at 57% for those running AI agents in production over the 2024 to 2026 period. Yet MIT has put forward figures from the same timeframe indicating 95% of generative AI pilots have yet to make a dent on the P&L, and Gartner is forecasting that by 2027 more than 40% of agentic AI projects will be shelved. The two can coexist. Teams are shipping agent projects, it is just that most of them are failing quietly.

What you will find is that the divide has little to do with the model. It is the engineering around it: how an agent fails safely, what job it is given, which systems it interfaces with, and whether one can be sure it is putting in useful work. This article is concerned with those details. For any team scoping an agent project and looking to sidestep an expensive demo, the practical choices made here are what will put you in the 5% that see a return rather than the 95% that do not.

We have been at this for 12 years at Refact, taking 200 or so projects from vague concept to something of use. Our AI development service operates on a simple tenet: clarity before code. With agents in particular, that is the whole game.

Why AI Agent Development Is Not a Model Problem

There is a temptation to begin by pitting models against one another, but that is misdirected. LangChain’s 2025 practitioner survey makes it plain: while 89% of teams have observability in place, only 52% run structured evals, and for 32% quality remains the chief obstacle in production. You will hear engineers on X, GitHub and Reddit tell you the same; they do not blame the model. They point to memory, cost, error recovery and integration.

An agent is no ordinary chatbot. It is a small system in production with an LLM for reasoning, connected to business rules, tools and memory. It takes an input, formulates a plan, calls on its tools and renders an outcome. Should any link in that chain be weak, it may look good in a demonstration but will not hold up in the wild. For a plain-language guide to the terminology when making these calls, we have put together an AI terminology cheat sheet.

The bottom line is you do not require a research staff to put together a sound agent. What is needed is product judgment and the software discipline to know where the investment is warranted.

Start With the Workflow, Not the Agent

The projects that give back all have the same origin. A single workflow of high volume and measurable value, one the agent owns from start to finish.

Take Klarna’s customer service operation, often held up as an example. Their agent handles two-thirds of the 2.3 million chats, cutting response times from 11 minutes to under two and moving $60M in annual profit. Then there is Morgan Stanley’s advisor tool, which saw 98% adoption by doing one job – synthesising the right internal document – better and faster than a person could. IBM’s AskHR disposes of 94% of routine HR matters. These are not “AI employees” so much as narrow agents solving a problem that was once a pain point.

Before going further, ask some hard questions:

• What is a time sink that comes around every week? Find a task your team is already performing in the hundreds.
• Is context from several systems required? If an API call is enough, write a script. An agent is overkill.
• Does the result inform a decision? Be it a triage, a draft or a summary, the business has to be able to act on it.

If the answers are not clear, then discovery is in order, not building. Our product discovery process is designed to close that gap in a matter of weeks, and we stand behind it with a money-back guarantee because we know most of these projects are dead in the water before a line of code is put down.

Agents vs. simpler options

The real question is if the task demands judgment. That is when an agent justifies its expense, in the interpretation and reasoning across sources. For anything else, a deterministic approach is superior. A rule engine or a RAG-powered AI chatbot from Zapier will outperform an agent on reliability and cost for a great many of the problems people set out to solve. Have the evidence to rule out the simpler option first.

The Architecture That Survives Production

By 2026 a pattern has emerged in production. Whether enterprise or open-source, the architecture is now a matter of a straightforward orchestrator calling on specialist agents via typed interfaces and standard protocols such as Anthropic’s MCP for auth and tool discovery. The numbers bear it out: the MCP ecosystem has swelled from 4,900 registered tools in early 2025 to over 177,000 in a year.

You will not find emergent multi-agent swarms conversing in natural language in a working system, despite what the marketing material from Salesforce might suggest. Their data indicates the average firm has about 12 agents and half of them are uncoordinated. The same goes for autonomy. According to Anthropic’s 2026 report on coding trends, developers may be using AI for 60% of their work but will only cede 0 to 20% of tasks entirely. In production, a Level 2 or 3 setup with defined escalation is preferred to full autonomy, save for the occasional coding scenario where version control provides a natural boundary. Here are the rules of thumb that should be observed:

• One orchestrator and specialist agents. Do not put together a swarm.
• Typed interfaces for your components (Protobuf, OpenAPI, JSON schema). Leave natural language out of it; between agents it is nothing but a fragility multiplier.
• Design for semi-autonomy. Human-in-the-loop points belong in the architecture from the start, not as an afterthought once you have launched.
• Let the task risk dictate the architecture. A high-stakes move warrants multi-step verification. For low-risk actions there is a fast path.

Memory is an architectural decision, not a setting

There is a separation of concerns now with Letta and others, who view memory as its own layer on par with models and tools. You will find three types: short-term working memory for what is at hand, episodic for what has come before, and structured knowledge for facts that stick. The research from Lakera AI on memory injection makes plain why this is more than an academic exercise. If third-party content seeps into long-term memory, an agent can form persistent false beliefs. The fix is mandatory if unglamorous: immutable audit trails, ways to correct or forget, and the default position that any retrieved content is untrusted.

Scoping Your First Agent MVP

An MVP spec that stands up to reality will have clear answers to these seven questions in plain English. If the team cannot provide them, the build will drift.

• Trigger: the event that sets the workflow in motion
• Inputs: the documents, records or messages the agent is permitted to read
• Actions: precisely what it can do in each system
• Guardrails: what must be reviewed before being sent, edited or deleted
• Output format: how “done” is defined for the recipient
• Escalation rules: the process for when a human steps in
• Logging: what is kept for debugging and audit

See the difference in the two columns below. On the left is where founders tend to begin. On the right is where projects make it.

Wrong first version	Right first version
An AI employee for customer success	An agent that drafts renewal prep summaries from CRM history
An AI research assistant for the whole company	An agent that produces a first-pass report summary for the analyst team
An agent that manages our inbox	An agent that classifies inbound requests and drafts replies for human review

We saw the pattern first hand building Workform with a project management consultant. We started with an AI assistant concept to help PMs with “everything.” Our blueprint process whittled that down to a sensible MVP: an assistant that ingests from Slack, Asana, email and meetings to keep a PM focused on one project. That narrowing was the hard part; the build was easy. For a broader view on scoping, our MVP development process guide lays out the discipline required to keep first versions honest.

Match models to jobs, not prestige

You do not need a premium model for every step. A tiered approach is more economical and as a rule more reliable. Put small, quick models to work on routine extraction and classification. Save the stronger ones for ambiguous reasoning or where the stakes are higher. With workloads like those handled by Phi-4, you can get comparable results from a small language model at a tenth of the cost. How you route the work matters far more to unit economics than which top-tier model you pick.

Keep this in mind:

• Repetitive, high-volume work: speed and cost are paramount
• Writing for customers: tone and format stability
• Workflows heavy on tools: function calling must be reliable
• Decisions with risk: require better reasoning and a human eye

Testing Before You Trust It

Most teams will either make or break themselves on evaluation. Put the eval harness in place before the agent is fully wired, not in the wake of it. When evals only come after launch, judgment is reduced to vibes and whoever puts on the best demo prevails.

Assembling a benchmark dataset is no small matter. It has to mirror the inputs the agent will encounter, including the messy and adversarial cases, not just the happy path. Cost and latency under realistic loads should be checked too. An empirical study from Delft and JetBrains on agent systems from GitHub and Stack Overflow found that the reported pain comes from tool integration, memory and debugging. A proper eval suite will expose those failure modes early.

Then there is the bias in LLM-as-judge scoring. It is quick, but for anything involving compliance, financials or customer comms you need a human review sample. Automate where possible, but review what you have to.

We have found this pre-launch checklist to be sound across our AI work:

What to test	What to look for
Functional accuracy	Does it complete the job correctly on realistic inputs?
Edge case behavior	Does it fail safely on messy or hostile inputs?
Latency	Does response time fit the workflow’s expectations?
Cost per successful run	Do the unit economics survive scale?
Safety and permissions	Does it stay inside approved capability boundaries?
User experience	Is the output easy for a reviewer to accept, edit, or reject?

Running an Agent in Production

Come Monday the agent is live. By Tuesday a timeout has caused some duplication, a missing field has led to a poor recommendation and a customer has escalated an issue that staging missed. This is production. The value is in the operating layer, not the model.

Practitioners will tell you the same stories: silent failure where an agent wanders off without crashing, cascading errors from a slightly off tool output, or cost explosions from oversized models on routine jobs. Take Writer’s client relationship history summary agent as an example of a well-scoped agent. It pulls from the CRM, ranks the important bits and drafts a brief. But the engineering of note is not the summary itself. It is the retrieval, the fallback for a missing record and the audit trail.

To hold up with real users an agent must have explicit operational answers:

• Timeout? Halt and alert, fall back or retry.
• Model is unsure? Escalate, ask for clarification or give a safe default.
• Partial failure? Mark for review, reverse the action or resume from a checkpoint.
• Approval needed? For any purchasing, updating or sending over a certain threshold.

Cost engineering starts on day one

When margins are unattractive, it is tempting for founders to put the onus on model pricing. In most cases, however, the problem lies with the workflow design. You will see runaway loops and oversized models being applied to trivial steps, or long chains of tool calls without any caching in place. An agent that is properly run will take the most economical reliable route at every turn and have the sense to know when to call it quits.

There are practical ways to exert control: rate limits, hard budgets for time and steps, context summarization and model routing. One should be measuring cost per successful run as a top-line metric, not just the cost of an API call.

Before launch there are four budgets to be put in place:

• Latency: what a user can be expected to wait
• Cost: the price of a task that still makes commercial sense
• Risk: delineating which actions the agent can do of its own accord and which need review
• Failure: how many times the workflow can fall back or escalate before it becomes a liability

Security is agent-specific

Do not think standard application security will suffice. Agents open up their own attack vectors: prompt injection from retrieved content, memory poisoning, or an agent misreading the context and misusing a tool. Then there is the matter of non-human identities, the service accounts and API keys the agents rely on; Huntress’s 2026 report has NHI compromise as one of the fastest growing areas of concern. The way to mitigate these is with some unglamorous but effective measures: RBAC on tools, least-privilege access, immutable audit trails, schema validation on structured outputs and capability boundaries at the protocol level. For anything irreversible or of high value, keep a human in the loop.

How to Tell If It Is Actually Working

A measured rollout to a small group is advisable. Put your eye on the completion and human intervention rates as well as the cost of a run. Should you find humans are constantly having to put right the same type of error, your guardrails are not strong enough or the scope is too wide. Correct that before you scale. And if the cost per success is on the rise despite the work getting done, check your model routing and context growth before you start looking at pricing.

You will notice a pattern among the ones that have succeeded. Klarna, Duolingo, Morgan Stanley and IBM AskHR all followed the same course: they took a single workflow and let the agent be responsible for the outcome from start to finish, with observability in place from day one. They did not expand until they could see the return. For the median deployment we have surveyed, payback comes in at five months; the best of them are seeing it in two. The failed pilots are a different story altogether, marked by no operational owner, poor data readiness and no link to ROI.

The Path From Idea to Working Agent

In the end, developing a good AI agent is not a matter of chasing the latest and greatest model, but of making better product calls. Take a painful workflow and define the job in straightforward terms. Make the first version so narrow it is almost dull and test it against reality. Then focus on the operating layer where you will find your cost controls and safety.

For those with an agent idea in mind, resist the urge to begin with a comparison of models. Instead, put pen to paper on the workflow that needs to be owned and where a human must remain involved. If that kind of clarity is in short supply, our buyer’s guide to AI development services is a resource for scoping the work with honesty, and our discovery phase is designed to get those decisions settled before any development is underway.