Ranking · #6 of 8 · Approach
Build your own agents on model APIs
Your engineers write agents directly against a model provider's API, with your own tools for search, enrichment, CRM and sending.
Only worth it with engineers to build the evaluation and guardrails too; otherwise it is the riskiest path here.
Scored 20/35 · last reviewed
Who it is for
Companies with spare engineering capacity and an unusual GTM motion no product covers.
Who it is not for
Teams without engineers who can build evaluation, monitoring and guardrails, not just the agent.
Strengths
- Complete control of prompts, tools and data
- Model API prices are published
- Nothing to buy beyond the APIs and data
Limits
- The agent is the easy part; the harness (freshness, scheduling, guardrails, evaluation) is most of the work
- Models vary run to run: without repeated-run tests you ship a demo, not a system
- Long-running work is hard: in PipelineBench, some models stopped working before the simulated year ended
What it costs
Model token costs plus engineering time. In PipelineBench, a simulated year of GTM work cost from $9 to $411 in tokens depending on the model; engineering time is the larger cost. PipelineBench leaderboard
How it works
An engineer gives a model tools (web search, an enrichment API, CRM read and write, a send function) and instructions, then runs it on a schedule or trigger.
The parts that make it safe to run unattended for months (re-checking data before use, resuming long work, suppression checks at send time, tests across repeated runs) are what separate a demo from production.
Scores and reasons
Agents use whatever data they are given; PipelineBench found models went with stale numbers about half the time.
1 in 10 model and task pairs in PipelineBench passed one run and failed the next; you have to build the tests.
You can build an approval queue, but it is another thing to build.
No guardrails exist until you write them into the send path.
Needs engineers; out of reach for most small revenue teams.
Everything is your code.
API prices are published; engineering time is the unknown.
Choose it if
- You have engineers who will own evaluation as well as the agent
- No platform supports your motion
Skip it if
- You plan to build it once and leave it running
- You cannot test it across repeated runs before it touches buyers
Evidence cited
- PipelineBench, Frontrunner, 29 October 2026. Runs AI models as the first GTM hire of a simulated company for a year, with the same tools, data and budget, and measures the qualified pipeline each creates. 23 models, 30 year-long runs each, graded against an oracle ceiling and shortcut baselines.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Sierra (Yao, Shinn, Razavi, Narasimhan), June 2024. Introduces the pass^k metric: the chance an agent succeeds on all k repeated trials of the same task, not just one. The standard way to measure reliability across repeated runs.