By 
- AutomationBench tests real cross-app workflows, not isolated skills. It combines discovery, policy-following, and coordination across tools. Earlier benchmarks like WebArena or ToolBench never tested these together.
- The original results were humbling. The best model in April 2026 cleared the bar less than 10% of the time. Top models agreed on solved tasks only around 17% of the time.
- Four failure patterns explained most breakdowns. False confidence, incomplete search, silent partial completion, and instruction drift kept showing up. Every model tested had the same issues.
- Narrow, purpose-fit toolsets boosted scores meaningfully. Businesses that scope automation carefully see far better results. Handing an agent raw, unscoped access rarely works well.
- GPT-6 Astra more than doubled the previous best score. It reached 41.4% on AutomationBench. That put it ahead of Claude Fable 5.1 at 31.4% and Claude Opus 5 at 26.9%.
- Astra’s real breakthrough is knowing when to stop. It paused rather than guessed when instructions were missing. This directly addressed the false-confidence problem the original research flagged.
- The Brihaspati Infotech turns these lessons into working systems. Scoping, policy enforcement, and verification sit at the center of every automation project the team builds.
THE BACKGROUND
What Is AutomationBench, And Why Was It Built
AutomationBench is a AI automation benchmark built by researchers at Zapier and released in April 2026. It tests how well AI models handle real, cross-application business workflows.
That is, tasks that require a CRM, an inbox, a calendar, a spreadsheet, and a messaging tool, all as part of a single task. The tasks were inspired by six business functions and cover more than 600 realistic tasks that require the automation of individual steps across approximately 47 different real-world apps and 500 API endpoints.
Each task is grounded in automation patterns actually requested by Zapier’s users and previously challenging to automate.
It has been created to address a particular set of use cases that were previously unaddressed by any benchmark. Prior benchmarks evaluated individual aspects of the automation workflow but not end-to-end automation, and they did not capture the combination of three distinct challenges that make up a realistic business scenario.
Benchmarks
Like WebArena, Mind2Web, and OSWorld tested long, open-ended web or desktop tasks, but stayed inside a single environment.
Tool-use benchmarks
Like ToolBench and API-Bank tested isolated API calls, one function and one result, with no larger workflow around it.
In these scenarios, an agent must perform three core competencies, including identifying the correct API endpoint, navigating the appropriate organizational policy, and performing deterministic orchestration across multiple disparate applications without implementing redundant steps or incorrectly notifying any given party.
These are precisely the kinds of scenarios in which an automation agent fails in production after succeeding during a sales demo, and AutomationBench seeks to highlight these issues by using deterministic scoring and no leniency to encourage users to adopt only the most reliable automation agents capable of handling production-grade data, software, and policies.
The Brihaspati Infotech believe that this is the first ever attempt at an honest assessment of automation as experienced by the business software user, free from the commercial pressures of vendors seeking to convince customers that their tools are the most compelling choice available.
THE GAP
Why Existing Benchmarks Missed Real Business Work
Prior to AutomationBench, methods mostly evaluate specific aspects of functionality. For example, WebArena and OSWorld benchmark models on their ability to perform extended, multi-step tasks in a single environment, while ToolBench and API-Bank test a models ability to make a single, correct API call one request/response pair. None of these evaluate the sort of multi-system coordination required in a business setting.
| Benchmark Type | What It Tests | What It Misses |
| WebArena, Mind2Web, OSWorld | Long, open-ended web or desktop tasks | Cross-app coordination and layered policy |
| ToolBench, API-Bank | Single, isolated API calls | Real business context and multi-step judgment |
| AutomationBench | Discovery, policy, and coordination together | Built specifically to close this exact gap |
None of these prior methods evaluate an agent’s ability to perform these three specific tasks a real business needs to do. Find the right tool out of hundreds of possible options, without being told what the options are upfront. Apply company policy appropriately, even when multiple rules could apply. And make sure different systems are not doing the same work, or giving updates to the wrong people.
This is part of the reason why many companies find automation to be a disappointment. Often, an automation pilot will look great in a demo, and excite executives and the business analysts who want to watch them run daily. But these same automations often fail to deploy further, because the reality of the data they have to process in the wild is much more complicated than in the demo.
The Brihaspati Infotech frequently sees this pattern enough that it has a name within the industry, when they conduct an audit of a client’s automations, they often find this as the first issue.
THE GRADING SYSTEM
How AutomationBench Actually Grades Success
AutomationBench uses deterministic, programmatic checks for evaluation instead of relying on a subjective opinion of one AI over the output of another. Every task requires a clearly observable, objective result as a form of a pass. That could mean records properly entered into a CRM, a field with an expected value, or a message sent to the right recipient at the right moment.
There are no fractional passes either, as the designers of the evaluation method have purposefully avoided rewarding the close calls. The researchers who designed the system deliberately took into account that a workflow only partially completed still represents a significant cleanup effort for the business.
01
Deterministic grading
outcomes are checked against programmatic assertions, not a model’s subjective read of whether something “looks right.”
02
End-state only
success depends on the final system state, not on how confident or polished the process looked along the way.
03
No AI judge
the researchers deliberately avoided using another model to score results, keeping the process reproducible and free of bias.
04
Cost tracked separately
every task’s total token usage across prompts, reasoning, and tool calls gets converted into a dollar figure.
The tasks themselves are complex enough to require between 12 and 22 steps for completion, with some of them reaching 44 necessary tool calls before reaching an end state. The increased level of automation brings a greater emphasis on the cost per task in dollars that the models charge. Researchers therefore provide a dual assessment of performance that takes into account both the number of completed tasks and the value of resources spent on the endeavor.
Two models may perform similarly in terms of sheer number of completed operations, but the one that produces more results for the same price has clear advantages for the businesses that would utilize them. These advantages scale exponentially with the size of automation deployments that seek to process thousands of tasks monthly as opposed to a dozen or so used in a benchmark evaluation.
THE RESULTS
The Original Scores, And Why They Mattered
Even the best model on the leaderboard cleared the bar less than 10% of the time, this is a sobering prospect for businesses hoping to leave their agents to run unsupervised. Opus and Gemini, the two best performers, could barely agree on which tasks they had managed to complete, their degree of overlap came in at a mere 17%. This has implications for how companies should approach vendor selection.
Automation does not serve as a universal panacea, and experts cannot easily predict or transfer its successes and shortcomings between different models.
01
Testing one AI tool
doesn’t guarantee another will behave the same way, even on the exact same workflow.
02
Cost and accuracy don’t move together
So the cheapest option and the most accurate option are rarely the same model.
03
Low overlap between top models
means a second opinion, or a hybrid approach, can genuinely catch different failure types.
The Brihaspati Infotech interpret these results as a cautionary tale about the importance of conducting one’s own due diligence. Do not judge a model’s capabilities by the marketing spiel of a single vendor, no matter how compelling the demo may be.
THE FAILURES
Why AI Agents Fail At Business Automation
The research discovered four recurring failure patterns that account for the lion’s share of all breakdowns across all the models tested in spite of different sizes and providers, as well as varying degrees of reasoning effort:
01
False confidence
agents frequently reported success even when the task had failed. This happened in over 90% of failed attempts for some models.
02
Incomplete search
models gave up too early. They assumed an answer existed in one tool instead of checking others thoroughly.
03
Silent partial completion
batch tasks got marked “done” even when only some items were actually processed correctly.
04
Instruction drift
models softened or paraphrased precise instructions. They didn’t follow them exactly as written.
None of these patterns show that automation lacks potential for serious business application, instead, each one resembles a general checklist that a competent automation program should be designed to counter from the get-go.
This is exactly what The Brihaspati Infotech incorporates into every client’s implementation of the system as a set of risks to watch out for that could adversely affect the software’s performance, which is why it’s necessary to take each of these four patterns under consideration.
THE APPLICATION
How Companies Can Use AutomationBench Results
Zapier’s researchers have tested the solution in six domains, including Sales, Marketing, Operations, Support, Finance, and HR. These domains are typical for most companies in the world and reflect real-life scenarios associated with daily operations. Therefore, the report is highly relevant and provides answers that help improve decision-making.
| Domain | Example Tasks Tested | Why This Domain Matters for Businesses |
| Sales | Lead routing, meeting conflict resolution, CRM enrichment, follow-up sequencing, duplicate contact merging, pipeline stage updates, quote-to-CRM syncing | Sales teams lose deals when leads sit unrouted or records go stale, so accuracy here directly affects revenue, not just efficiency |
| Marketing | Campaign data syncing, list segmentation, multi-channel status updates, audience deduplication, UTM tracking cleanup, email performance reporting | A single wrong segment or duplicate send can damage sender reputation and customer trust, making precision more important than speed |
| Operations | Cross-tool task creation, compliance checklist generation, incident logging, vendor record updates, SOP-triggered workflows, inventory status syncing | Operations tasks often touch compliance-sensitive data, so a silent error here can create audit risk long before anyone notices |
| Support | Ticket triage, escalation routing, response drafting under policy, SLA deadline tracking, customer sentiment flagging, knowledge-base lookups | Support is customer-facing by nature, so a false-confidence failure here can mean a client receives a wrong or policy-breaking reply |
| Finance | Invoice matching, approval workflows, expense policy checks, purchase order reconciliation, duplicate payment detection, budget threshold alerts | Finance tasks deal directly with money movement, where even a small error compounds quickly across hundreds of transactions |
| HR | Onboarding checklists, policy-gated document routing, scheduling coordination, benefits eligibility checks, offboarding task sequencing, compliance document tracking | HR handles sensitive personal data and legal requirements, so policy-following accuracy matters as much as task completion |
AutomationBench covers all the aforementioned areas, making the research results applicable to the user in the position to make informed decisions about the utilization of AI agents. In other words, the tool can help with due diligence before outsourcing any production data, customer support emails, or financial statements to a specific agent.
Notably, one of the study’s conclusions resonates with the actual application in the context of real-life scenarios. According to the researchers, providing the model with a reduced set of tools relevant to the domain resulted in better scores compared to granting access to all the available instruments and APIs.
Therefore, Zapier’s researchers recommend investing time in the due diligence process and defining the scope of automation rather than relying on the default settings or solution promoted by a higher price threshold.
THE BREAKTHROUGH
What Changed With GPT-6 Astra’s 41.4% Score
OpenAI’s GPT-6 Astra recently achieved a stunning 41.4% on AutomationBench, marking more than double the performance of its predecessor, which had scored 18.1%, and substantially outperforming all other tested models.
For perspective, prior to this update, no model had been able to achieve beyond 40% on the benchmark. Thus, Astra’s result shows significant improvement over past capabilities, achieved in under 7 months since the last release, which had under 10% accuracy in April.
This rapid progress shows that traditional executive planning is outpaced by new developments. Firms that dismissed agentic automation must reconsider their strategies due to rapid advancements in generative AI.
The Brihaspati Infotech sees a vital opportunity for organizations to revisit abandoned automation strategies post-April report.
THE BEHAVIOUR
Why Astra’s “Pause Instead Of Guess” Matters
Two examples from Zapier’s own evaluation that go beyond the headline metric are telling about the model’s thought process and how it tackles uncertain situations.
01
Budget reconciliation task
Astra found finance adjustments buried inside an email thread and got every single number right as a result. Its predecessor’s output looked finished on the surface, but the splits underneath were actually wrong.
02
15 integration inquiries task
Astra couldn’t locate the required reply standard anywhere in the connected systems, so it stopped the task entirely. Zero replies went out as a result. Its predecessor attempted all 15 replies anyway and earned partial credit, despite likely getting several of them wrong.
This directly addresses the issue of “false confidence” identified by the original study as the most common type of error across all models. Addressing it is crucial for training practical agents that can operate reliably in production environments.
By learning to avoid wrong answers, the Astra model demonstrates that it can adapt its behaviors depending on the situation and is not as reckless about committing to an answer as previous models were.
This is especially valuable for enterprises that store sensitive data in their internal systems, from now on, a language model will not blindly assume that its answer is correct when operating in such environments.
THE CONTEXT
What This Means For Businesses Evaluating Automation
These scores are based on the results of testing conducted by OpenAI themselves, which is why it is important to note this fact at the beginning of the summary. Moreover, some articles researching these models also note that researchers carried out testing under customized conditions for a particular model.
Thus, it is necessary to approach these comparisons with some caution so as not to draw erroneous conclusions when choosing particular software. Despite this caveat, the general trend of the research results still seems positive.
01
Purpose-fit toolsets matter
scoping what an agent can access changes outcomes as much as which model you choose.
02
Explicit policy enforcement matters
rules need to be enforced in the workflow itself, not just referenced loosely in documentation.
03
Rigorous verification steps matter most of all
catching silent failures before they reach a customer protects both trust and revenue.
These three factors are critical when it comes to automation and artificial intelligence. Therefore, companies interested in automation should guide themselves by more than one particular article when selecting tools.
However, when forming the basis of a future project, these trends will serve as an indicator of readiness. In other words, companies can implement automation in a production environment with confidence.
THE PARTNER
How The Brihaspati Infotech Can Help You Automate
Reading a benchmark gives you a good idea about where the industry is at any given time. Turning it into an actual working automation system for one particular business though requires real-world experience that comes from multiple deployments.
The Brihaspati Infotech deals with companies exactly on this matter, forming functional automatons out of research data as opposed to simply theoretical numbers on a page.
1.
Narrow, purpose-built workflows – designed around what a business actually needs, not raw, unscoped access to every available tool.
2.
Policy enforcement built into the workflow – rules get enforced automatically at each step, not just written down somewhere and hoped for.
3.
Verification checks at every stage – silent failures get caught before they ever reach a customer or a financial record.
4.
No unsupervised guesswork on critical systems – off-the-shelf agents never get dropped into sensitive workflows without proper guardrails in place.
This approach avoids the major pitfall of using off-the-shelf agents in unique systems without consideration of the specific environment.
Getting started doesn’t require a business to have a comprehensive sense of where to apply AI. Jovian, The Brihaspati Infotech’s AI assistant, quickly identifies integration points and creates a roadmap, transforming “we should automate something” into actionable steps.
Businesses wanting to implement a functioning system don’t need to build one themselves. The Brihaspati Infotech transforms AI automation from a risky pilot into a reliable operational standard. Talk to Jovian to discover where the integration could take your business.
Disclaimer: References to third-party products or companies are informational and do not imply affiliation, association, endorsement or partnership.
