Pi Agent Achieves Highest Success Rate and Lowest Cost in DeepSeek V4 Flash Harness Tests

Composio tests 8 frameworks with DeepSeek V4 Flash. Pi Agent leads with 20/30 success at $0.028/task, while Claude Code is fastest but costs 7x more.

Woofun AI reports that Composio integrated DeepSeek V4 Flash into eight distinct Harness frameworks, evaluating them against a standardized set of 30 agent tasks involving Gmail, GitHub, Slack, Calendar, and Notion. Pi Agent secured the highest success rate with 20 completed tasks and the lowest cost at approximately $0.028 per successful task. In contrast, Claude Code achieved the fastest median execution time of 122.7 seconds but incurred a cost of $0.195, nearly seven times higher than Pi Agent. Oh My Pi recorded the slowest median time at 272.4 seconds.

The performance gap demonstrates that switching frameworks increased successful task completion from 14 to 20.

However, Pi Agent's configuration utilized a 'high' inference level rather than 'max' and processed 24 tasks via DeepSeek's official API instead of OpenRouter, introducing variables beyond the framework itself. Previous tests using Kimi K3 showed Oh My Pi leading with 22/25 successes, whereas Pi Agent ranked second with 18/25.