The benchmark covers six domains: navigation, repository management, financial analysis, 3D design, browser automation, and web searching. Within those areas sit 231 tasks, split across 11 live servers, ranging from Google Maps and GitHub to Yahoo Finance, Blender, Playwright, and Google Search. Each domain has its own set of sub-tasks, such as route planning in maps, portfolio analysis in finance, or object creation in 3D modeling, with complexity increasing as models are forced to use multiple steps and maintain information over longer contexts.
Instead of relying on a language model to judge another’s output, which has been common in past benchmarks, MCP-Universe measures success by execution. That means checking if a model formats answers correctly, whether it produces consistent results over time, and if it can work with data that changes. A separate set of evaluators handles each dimension: format evaluators for strict compliance, static evaluators for timeless facts like historical stock prices, and dynamic evaluators that pull real-time ground truth for shifting data such as live market movements or flight fares.
The test results reveal a wide gap between model hype and operational performance. GPT-5 led all systems, but its overall success rate stood at just 43.7 percent. It showed strength in financial analysis, completing two-thirds of those tasks, and performed above 50 percent in 3D design, but it failed more often than not in navigation and browser automation. Grok-4 followed at 33.3 percent, then Claude-4.0 Sonnet at 29.4 percent. The best open-source option, GLM-4.5, reached 24.7 percent, ahead of some proprietary systems but still far behind the leaders.
Looking deeper, the evaluator breakdown shows another layer of fragility. On format checks, most models scored high, with Claude-4.0 near 98 percent compliance, suggesting they can follow rules when tightly defined. But when asked to produce content against static or live-changing data, success dropped to the 40–60 percent range. GPT-5 again led in dynamic cases with 65.9 percent, but that still meant failure in more than a third of scenarios where up-to-date information was required.
Task efficiency also varied. GPT-5 needed on average just over eight steps to succeed, Grok-4 about 7.7, while smaller models like o3 could finish in under five but with less reliability. That trade-off between speed and accuracy highlights how fragile multi-step reasoning remains, especially in domains with long context chains. The context growth was most obvious in maps, browser automation, and finance, where server outputs return large blocks of data. Summarization experiments, meant to shorten context, brought mixed outcomes: slight gains in navigation but losses elsewhere, showing that compression alone does not solve the memory problem.
Another recurring failure came from unfamiliar tools. In some cases, models called functions incorrectly or set parameters in ways that broke execution. One example involved the Yahoo Finance server, where stock price queries require two distinct dates; models often set them the same, leading to errors. Salesforce tested an exploration phase, letting models experiment with tools before running tasks, and saw partial gains — GPT-4.1 improved slightly in browser automation and Claude in finance — but the fix did not carry across all domains.
The benchmark also looked at how frameworks influence outcomes. Comparing agent backbones, the ReAct setup generally outperformed Cursor, despite Cursor being designed as an enterprise agent. ReAct achieved higher overall success with Claude-4.0, while Cursor only excelled in isolated areas like browser automation. With OpenAI’s o3 model, the company’s own Agent SDK produced stronger results than ReAct, particularly in finance and design, suggesting that framework-model pairings can alter performance as much as raw model size.
Adding unrelated MCP servers made tasks even harder. When models had to deal with more tools than necessary, performance dropped sharply. In location navigation, for example, Claude-4.0 fell from 22 percent success to 11 percent once extra servers were included. The decline highlights how easily noise can destabilize tool orchestration, a problem that enterprises will need to address as they scale up.
For all the variety of tests, the conclusion is consistent. Current models, even GPT-5, can handle isolated reasoning or simple calls, but when placed into real environments with shifting data, long contexts, and unfamiliar tool sets, they still fail most of the time. MCP-Universe exposes those gaps more clearly than past benchmarks, offering a way to measure progress as researchers try to close them. For companies deploying AI at scale, the results point to a hard truth: building reliable agents will depend not just on bigger models but also on smarter frameworks, better context handling, and stronger safeguards around tool use.
Notes: This post was edited/created using GenAI tools. Image: DIW-Aigen.
Read next: LLMs Struggle with Reasoning Beyond Training, Study Finds