Industry story
Pydantic AI v2.4.0 Adds Agentic Evaluators and GEval Support
ai-in-adtech engineering measurement
Pydantic AI 2.4.0 is quietly doing important work for anyone trying to run agents in production. The new agentic evaluators — ToolCorrectness, TrajectoryMatch, ArgumentCorrectness, MaxToolCalls, MaxModelRequests — finally give developers something concrete to measure beyond "did it answer correctly": did the agent take the right path to get there. Add GEval and the LLMJudge quality rubrics, and you have a real eval stack, not just vibes. The security split on file handling is the kind of unglamorous fix that bites you hard if it ships without it.
Full analysis
Pydantic AI released version 2.4.0 on July 2, 2026, introducing several agent-focused evaluation tools. New additions include a GEval evaluator and standard quality metric rubrics for LLMJudge (an AI-powered scoring system), plus a suite of agentic span-based evaluators: ToolCorrectness, TrajectoryMatch, ArgumentCorrectness, MaxToolCalls, and MaxModelRequests — giving developers fine-grained ways to assess whether an agent used the right tools in the right sequence.
The release also splits the previous preserve_file_data parameter into separate allow_uploaded_files (for inbound security control) and an AG-UI representation opt-in, tightening security around file handling in agents. Bug fixes include improved xAI XSearchTool behavior, forwarding of user location to xAI web search, and better cleanup of streaming tasks on interruption — all relevant to agents running tool-calling workflows in production.
Comments