Getting Out of the Loop: What I Learned Building an Autonomous Agent
A few months ago I wrote about the AI workflow I was building for my podcast. Since then I rebuilt a large part of it, and the biggest change is that I’m no longer the bottleneck.
In that earlier version, I was the one gluing the whole research process together, even though I had a workflow of various agents doing various jobs. After a few rounds of searching and extracting, I would review what came back, pull in ChatGPT to read through the details and summarize them faster than I could, decide how to pivot or what to double down, and send it off for another round. I was the coordinator. It worked, but it was slow and heavy, and the research for a single episode could stretch over many hours and several sittings.
At some point I tried to categorize the decisions I was actually making, and they fell into 3 buckets:
- coordinating the results, and the AI’s summary and recommendation of how a pass had turned out
- judgement calls on whether a story is good enough, or whether a material set is rich enough
- taste: whether something is interesting, whether something is boring, whether something is meaningful for the type of podcast and episode I want to make
The first two seemed totally reasonable to hand to an agent. The third, taste, I knew would be difficult to outsource to AI, but I was hopeful that I could train it over time and gradually take less of the wheel.
What I mean by “an autonomous agent”
It’s worth saying what I mean by an agent, because the word gets used for almost everything right now.
What I had before was agentic, but not really an autonomous agent. It was a workflow: predefined steps, with different AI profiles doing different jobs at each stage. Anthropic’s write-up on building effective agents draws the line in a way I find useful: “Workflows are systems where LLMs and tools are orchestrated through predefined code paths,” while “agents … are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.”
By that definition, my earlier version was a workflow. What I built after is closer to an agent: it makes its own decisions at runtime, handles forks and situations I didn’t script for, and keeps going until it reaches its goal or hits a limit I set.
The setup, quickly
A bit of context so the rest makes sense. The agent is built on OpenAI’s Agents SDK. Its job is narrow on purpose: prepare the raw research material for an episode by finding real sources and pulling out actual passages, not summarizing or synthesizing them. I want real quotes and real detail to work from later, not the model’s paraphrase. It searches and extracts through tools like Tavily, Brave, and Exa, and I route model calls through aisuite so I can test different models and providers behind a common interface.
Here are 5 things I found really helpful while building an autonomous agent.
1. Decision rights framework
The idea is pretty simple. We lay out exactly who owns which decision: code, the LLM, or human. Here’s an example:
| Decision | Owner |
|---|---|
| Which research lanes to dispatch, with what focus / avoid | Lead agent (LLM) |
| Overall readiness | Lead agent (LLM); consults evaluator, bounded by a floor |
| A worker's own queries / search angles within a dispatch | Worker agent (LLM) |
| Which sources a worker extracts; its source-shape scoring | Worker agent (LLM) |
| Whether budgets and limits are exceeded | Code |
| Grounding and validation | Code |
| Escalation (when limits are exceeded) | Human |
| Define the episode goal | Human |
Having these laid out clearly streamlines the implementation of the agent. As I collaborated extensively with AI coding agents on the build, the decision rights framework became a set of principles that many small implementation decisions were based on, which prevented drift.
With a new feature or change, the framework is also a clean way to express how a decision’s ownership is shifting. For example, say the goal is to find a story with a main character. That decision explicitly moved from code to the LLM:
| Decision | Owner | Approach |
|---|---|---|
| Does the material have a real named person? | Code | Check for a person's name in the story |
↓
| Decision | Owner | Approach |
|---|---|---|
| Does the story have a main character? | LLM | Reading comprehension to judge whether the named person is a character the story is built around, or just a name being quoted |
2. Handle AI hallucinations: keep it honest, keep it grounded
AI will hallucinate. A spot check isn’t enough; you need a mechanism built into the agent to keep it honest.
Here is mine. Every claim has to trace back to an extracted passage from a tool call. The model can reason over, organize, and characterize the material however it likes, but those passages are the ground truth, and anything that can’t be traced to one gets dropped.
For now, the system just drops what it can’t trace. A more sophisticated version in the future might fact-check those pieces instead of discarding them.
An interesting trend I noticed: when the agent builds a story from a thin material group, more of its claims get dropped as ungrounded. With less real material to go on, the model makes more up to fill the gap.
3. Decide the core patterns up front
I made these architecture choices upfront:
- Orchestration style: handoffs, or manager-style
- How to keep memory across turns
- Which models and transport to use
- How to trace the agent’s decisions
- The output contract
- Limits and boundaries
The model layer is the one I’m glad I set up with flexibility in mind from the start. Using aisuite, I can swap in and experiment with different LLM models with close to zero effort.
Another I’m glad I planned for early is traceability. To follow how the agent reasons and makes its decisions at each step, you have to leave breadcrumbs, logs or some structure you can go back and examine when you need to debug or evaluate.
But the choice that mattered most was defining the output contract as early as possible. It’s really a way of defining the agent’s goal, the exact shape of the work product it has to produce. It’s a lot like managing people: the clearer you are about what success looks like, the more likely they can achieve it.
4. Start with your strongest model, then trade down
Early on, I read some best practices for building agents: start with the smartest model, and downgrade later if you need to. I wasn’t totally convinced at first, because that material was published by the companies that make money from these API requests. And there were so many times during development where I was so tempted to use the lower-grade model for iterations, because it’s so much cheaper than the latest.
But what I learned is that there’s a good reason behind the advice. If you’re not using the latest model while you’re still developing the agent, it becomes much harder to tell whether your agent can’t reach a certain quality or goal because of the model, or because of something else going wrong (prompt, context, tools, decision-making, etc.). Starting with your strongest model reduces the chance that model capability is the bottleneck, so you can focus first on whether the agent has the right mechanisms and is making the right decisions. After that’s established, it’s not very hard to A/B test different LLM models, evaluate how the results differ, and use that to make the cost-versus-quality decisions you’re willing to accept.
And cost vs. quality isn’t always an exact tradeoff. There are certain jobs where a cheaper, simpler model handles them just as well as a smarter one, and in those cases you get the cost savings without trading off quality. But for a lot of complex tasks, the reduced cost of an earlier or smaller model can mean lower-quality output.
5. Use evals so you’re not playing whack-a-mole
Evals are the single most important lever I’ve found for improving an agent efficiently. And it’s also the part I’m still very much working on.
An eval isn’t just a pass/fail test. It’s a way to score the agent’s behavior objectively*, on the dimensions you care about, so you’re measuring quality instead of eyeballing the outputs. That does a few things:
- Catch regressions
- Identify where to improve
- Validate a change worked
Here’s the testing structure I set up around evals:
That cheap tier once surfaced a regression I’d never have caught by eye, where a change quietly made the agent drop a worker’s honest “this lane is empty” report. The lesson: an eval’s overall score tells you how you’re doing, but it’s the rubric breakdown that tells you exactly what to fix next.
Where this has worked well for me is developing the lead agent’s decision-making, where I used evals to see how well it was doing and where to push. Where it hasn’t worked as well is on taste. I tried building a flywheel: capture labeled data on how each story or material group actually turned out in a finished episode, and use it to calibrate a judge. After collecting data from 10+ episodes, the judge’s evaluation is still not good enough to be trusted to steer the research.
There’s more room for improvement here. But I’m convinced evals are the tool to keep agent development disciplined and efficient. It was the heart of the DeepLearning.AI Agentic AI course I took, and the more I build, the more I understand why.
* "Objectively" is the aim more than a guarantee. Some checks really are objective, like whether a claim traces back to a fetched source, while eval techniques like LLM-as-judge, grading against a rubric, aren't entirely objective.
The interesting thing is that getting myself out of the loop didn’t mean handing the AI every decision. It meant getting much more explicit about which decisions belonged to the model, and which ones I still wasn’t willing to let go. The more autonomous the agent became, the more those boundaries mattered.
Building this has been so much fun! These are the things that have helped me most so far, and I’d love to hear what’s helped you.
References
- Anthropic, Building effective agents
- OpenAI, Agents SDK documentation
- DeepLearning.AI, Agentic AI