Pydantic AI v2.4.0 Adds Agentic Evaluators and GEval Support

Full analysis

Pydantic AI released version 2.4.0 on July 2, 2026, introducing several agent-focused evaluation tools. New additions include a GEval evaluator and standard quality metric rubrics for LLMJudge (an AI-powered scoring system), plus a suite of agentic span-based evaluators: ToolCorrectness, TrajectoryMatch, ArgumentCorrectness, MaxToolCalls, and MaxModelRequests — giving developers fine-grained ways to assess whether an agent used the right tools in the right sequence.

The release also splits the previous preserve_file_data parameter into separate allow_uploaded_files (for inbound security control) and an AG-UI representation opt-in, tightening security around file handling in agents. Bug fixes include improved xAI XSearchTool behavior, forwarding of user location to xAI web search, and better cleanup of streaming tasks on interruption — all relevant to agents running tool-calling workflows in production.