slr_bench_results.csv

Model,LRL,Syntax Score,Logical-Reasoning Acc. Basic,Logical-Reasoning Acc. Easy,Logical-Reasoning Acc. Medium,Logical-Reasoning Acc. Hard,Total Compute Tokens (M),Total Compute Costs ($)
o3,15.5,80,99,93,74,45,4.30,207.24
o4-mini,12.3,86,93,88,52,13,3.98,21.43
o1,11.9,68,92,89,41,15,5.19,364.72
o3-mini,11.6,75,97,90,37,7,4.73,24.71
o1-mini,10.1,95,97,82,20,3,3.65,19.98
R1-Llama-70B,8.8,75,98,67,8,4,11.61,5.33
Gemini-thinking,8.6,83,93,65,13,1,---,---
gpt-4.5-prev,7.3,100,94,47,5,1,0.37,576.40
gpt-4o,6.2,100,93,29,2,0,0.26,20.03
Llama 3.3-70B,5.6,100,90,22,1,0,0.49,0.82
gpt-4-turbo,5.4,100,89,18,2,0,0.41,81.30
Llama 3.1-8B,3.8,80,70,7,0,0,0.20,0.14
Llama 3.2-3B,1.0,25,19,1,0,0,0.10,0.11
Llama 3.2-1B,0.0,93,0,0,0,0,0.47,0.12