The latest articles on DEV Community by Classified (@iamclassified). https://dev.to/iamclassified https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4143571%2F87426864-f60a-4480-b3e5-a0b8c21aab5c.jpg https://dev.to/iamclassified en Classified Sat, 03 Oct 2026 21:45:44 +0000 https://dev.to/iamclassified/liquidity-event-identificationan-llm-reasoning-benchmark-5boi https://dev.to/iamclassified/liquidity-event-identificationan-llm-reasoning-benchmark-5boi <p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <h2> What I Benchmarked </h2> <p>I built a Kaggle benchmark called <strong>Liquidity Event Identification</strong> to test how well large language models can identify liquidity events from controlled price-action scenarios.</p> <p>The benchmark tests whether a model can distinguish between events such as liquidity sweeps, rejections, breakouts, and acceptance based on the evidence provided in each scenario.</p> <p>The benchmark contains seven scenarios:</p> <ul> <li>Equal-high liquidity sweep and rejection</li> <li>Equal-low liquidity sweep and rejection</li> <li>Breakout followed by acceptance</li> <li>Brief break followed by rejection</li> <li>Near-identical scenarios where one detail changes the correct interpretation</li> <li>Conflicting multi-timeframe evidence</li> <li>A deliberately misleading explanation</li> </ul> <p>The benchmark is focused on reasoning about market structure and liquidity events. It is not designed to measure trading profitability or predict real market outcomes.</p> <h2> Models Tested </h2> <p>I ran the benchmark against 20 models available through Kaggle:</p> <ul> <li>GPT-6 Astra</li> <li>GPT-5.6 Sol</li> <li>GPT-5.6 Luna</li> <li>GPT-5.5</li> <li>GPT-5.4</li> <li>GPT-5.4 Mini</li> <li>GPT-5.4 Nano</li> <li>Claude Opus 5</li> <li>Claude Opus 4.6</li> <li>Claude Sonnet 5</li> <li>Gemini 3.7 Flash</li> <li>Gemini 3 Flash Preview</li> <li>Gemini 3.1 Flash-Lite Preview</li> <li>Gemini 3.1 Pro Preview</li> <li>Gemini 3.5 Flash</li> <li>Grok 4.20</li> <li>Qwen3 Next 80B</li> <li>GPT-OSS 120B</li> <li>DeepSeek R1</li> <li>Claude Sonnet 4.6</li> </ul> <p>I used models from multiple providers to compare how different models performed on the same scenarios.</p> <h2> Findings </h2> <p>16 of the 20 model runs completed successfully. All 16 completed runs scored <strong>7/7</strong>, resulting in <strong>112 correct answers out of 112 completed scenario evaluations</strong>.</p> <p>The other four runs — Qwen3 Next 80B, GPT-OSS 120B, DeepSeek R1, and Claude Sonnet 4.6 — encountered execution errors before completing the benchmark. They were excluded from the accuracy calculation.</p> <p>The main result was that the benchmark did not differentiate the completed models. Every completed model achieved a perfect score.</p> <p>This suggests that the seven scenarios are not difficult enough to distinguish model performance.</p> <p>The 100% result also does not indicate that these models can trade profitably or reliably interpret real-world market conditions. The scenarios are controlled and synthetic, and the benchmark measures a specific reasoning task.</p> <p>For the next version, I plan to increase the difficulty with more scenarios involving ambiguity, conflicting evidence, misleading context, stronger multi-timeframe conflicts, and cases where the most obvious interpretation is incorrect.</p> <h2> My Benchmark </h2> <p>My Kaggle benchmark:</p> <p><a href="https://www.kaggle.com/benchmarks/tasks/iamclassified/liquidity-event-identification" rel="noopener noreferrer">Liquidity Event Identification — Kaggle</a></p> <p>The benchmark contains the task definition, scenarios, evaluation logic, and model evaluation results.</p> devchallenge kagglechallenge ai machinelearning