MARKET-BENCH
Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics
Market Bench Podium
Top performing models ranked by lowest Mean MAE across all three trading strategies
Overall Methodology
We split the strategies up into 3 different strategies: easy, medium, and hard. Each strategy has an associated prompt. In this prompt, we detail out the strategy specifics and task the model to build a backtester which correctly integrates the strategy and outputs the required metrics in CSV format.
This CSV is then compared to our verifiable backtester's outputs to calculate MAE. We also test how reliable the model is using pass@1 and pass@3 metrics.
All three strategies require models to track and reserve the liquidity they remove from the book through simulated trades. We create a synthetic book that nets the raw order book data with consumed liquidity. Strategies 2 and 3 also include a delay between submitting an order and hearing back from the exchange, mirroring the real world.
Take a look at
each strategy
Check it outA simple trading strategy where the model buys and sells MSFT stock at predetermined times using Databento's L10 order book data. Volume is randomized to enforce correct book reservation—if only 8 shares are available but 10 are requested, the model should only take 8. Tracks cash, position, realized/unrealized P&L, equity curve, and maximum drawdown.
A pairs trading mean-reversion strategy on Coke and Pepsi. The model calculates mid prices, creates a spread, and uses rolling z-scores to trigger entries/exits. When z-score exceeds entry threshold, it buys one leg and sells the other; positions flatten at the exit threshold. Includes cooldown mechanisms and shared capital accounting.
A realistic delta hedging strategy using random walk deltas and MSFT order book data. At regular intervals, the model evaluates net delta and trades to get flat. Uses fill-or-kill limit orders with fixed exchange delays. Tracks stock position, options delta, net delta, realized/unrealized P&L, and maximum drawdown.
Overall MARKET-BENCH performance across all three strategies
| Model | pass@3↓ | pass@1↓ | Mean MAE↓ | Avg. attempts↓ |
|---|---|---|---|---|
| gemini-3-pro-preview | 1.00 | 0.93 | 437.51 | 1.07 |
| gpt-5.1-codex-max | 1.00 | 0.93 | 3,102.74 | 1.07 |
| claude-sonnet-4.5 | 1.00 | 0.93 | 5,459.68 | 1.07 |
| qwen3-max | 1.00 | 0.80 | 190,973,360.19 | 1.20 |
| mistral-large-2512 | 0.93 | 0.73 | 46,134.31 | 1.47 |
| deepseek-v3.2 | 0.87 | 0.67 | 4,113.84 | 1.27 |
| grok-4 | 0.87 | 0.47 | 705.16 | 1.67 |
| llama-4-maverick | 0.53 | 0.40 | 9,252.77 | 1.67 |
| claude-opus-4.5 | 0.47 | 0.27 | 2,765.33 | 1.61 |
| llama-3.1-nemotron-ultra | 0.33 | 0.20 | 11,404.92 | 1.83 |
| command-a | 0.33 | 0.07 | 9,817.97 | 2.50 |
| nova-premier-v1 | 0.13 | 0.07 | 688.57 | 1.50 |