LLM SQL Generation Benchmark Results

We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts. The results are shown below and can be compared to a human baseline.

--
human
human
--
--
--
--
--
332.6 ms
31,006,852
759.83 MB
#1
openai
gpt-chat-latest
82.10
99.16
65.05
1.793
1.00
818.72 ms
39,159,059
941.02 MB
#2
anthropic
claude-opus-4.7
81.17
96.86
65.48
4.738
1.00
2,800.88 ms
34,437,823
811.31 MB
#3
google
gemini-3-pro-preview
79.25
98.20
60.30
49.440
1.02
682.04 ms
46,150,625
1,055.25 MB
#4
google
gemini-3.5-flash
78.79
97.53
60.05
9.736
1.00
1,484.88 ms
50,048,646
1,060.75 MB
#5
google
gemini-2.5-pro
78.64
99.31
57.98
19.100
1.06
443.22 ms
42,878,115
826.88 MB
#6
moonshotai
kimi-k2-0905:exacto
78.50
97.95
59.06
4.547
1.02
1,397.327 ms
46,877,411
848.80 MB
#7
openai
gpt-4o-search-preview
78.49
99.76
57.21
3.757
1.40
525.58 ms
41,771,932
394.36 MB
#8
google
gemini-2.5-pro-preview-05-06
78.37
97.81
58.94
21.273
1.00
825.714 ms
41,502,252
947.30 MB
#9
google
gemini-2.5-pro-preview
78.26
98.09
58.43
21.296
1.00
1,031.96 ms
46,575,244
975.60 MB
#10
openrouter
horizon-beta
77.80
98.83
56.78
1.401
1.02
1,202.46 ms
72,041,510
1,141.62 MB
#11
anthropic
claude-3.7-sonnet
77.76
99.44
56.08
3.149
1.10
374.224 ms
40,099,998
824.57 MB
#12
moonshotai
kimi-k2
77.55
99.33
55.77
4.265
1.06
589.22 ms
49,539,148
903.11 MB
#13
openai
o3-mini
77.11
99.39
54.83
9.886
1.14
448.84 ms
49,432,133
844.29 MB
#14
x-ai
grok-4
76.89
97.98
55.81
61.602
1.00
677.06 ms
49,360,869
1,145.95 MB
#15
anthropic
claude-3.7-sonnet:thinking
76.86
96.79
56.93
40.858
1.00
1,425.82 ms
40,046,627
947.37 MB
#16
anthropic
claude-opus-4.5
76.66
99.06
54.26
2.907
1.02
4,677.22 ms
42,411,762
294.93 MB
#17
openai
o3-mini-high
76.59
98.81
54.36
19.254
1.04
625.82 ms
43,602,960
960.85 MB
#18
openai
gpt-5.1-codex-mini
76.32
98.98
53.65
9.049
1.02
639.54 ms
41,901,897
971.73 MB
#19
openai
gpt-5.3-chat
76.23
97.51
54.95
3.188
1.00
1,898.78 ms
57,024,811
1,137.19 MB
#20
mistralai
mistral-medium-3-5
76.07
97.65
54.50
5.209
1.02
1,331.286 ms
52,340,224
974.02 MB
#21
anthropic
claude-3.5-sonnet
76.06
99.70
52.41
3.234
1.02
388.96 ms
37,145,042
684.44 MB
#22
google
gemini-3-flash-preview
75.85
93.36
58.35
2.406
1.00
6,401.3 ms
42,874,619
927.15 MB
#23
anthropic
claude-opus-4.6
75.85
99.44
52.25
3.423
1.02
2,997 ms
37,271,990
261.77 MB
#24
google
gemini-3.1-pro-preview-customtools
75.82
92.15
59.48
20.360
1.02
3,833.3 ms
49,317,675
1,056.97 MB
#25
qwen
qwen3-235b-a22b-07-25
75.77
99.23
52.31
8.620
1.18
397.755 ms
38,751,330
781.80 MB
#26
openai
gpt-5.2
75.73
97.37
54.09
10.389
1.00
1,022.388 ms
53,083,092
1,134.93 MB
#27
openai
gpt-5.2-codex
75.70
98.17
53.23
5.045
1.00
1,374.06 ms
47,296,114
1,000.48 MB
#28
qwen
qwen3-coder
75.66
99.83
51.48
4.571
1.04
457.224 ms
46,666,126
333.42 MB
#29
qwen
qwen3-coder-plus
75.58
99.76
51.39
2.303
1.02
679.163 ms
53,829,416
387.08 MB
#30
qwen
qwen3-32b
75.56
98.17
52.95
37.553
1.06
761.347 ms
44,676,197
795.72 MB
#31
openai
o3-pro
75.42
98.25
52.59
76.620
1.04
746.8 ms
52,804,037
936.55 MB
#32
openai
o4-mini
75.39
99.09
51.69
10.228
1.08
613.66 ms
52,581,751
940.75 MB
#33
openai
gpt-4o
75.33
98.78
51.87
1.534
1.02
1,386.16 ms
60,323,408
988.77 MB
#34
anthropic
claude-opus-4.1
75.22
98.80
51.65
6.342
1.04
580.51 ms
39,294,543
936.76 MB
#35
qwen
qwen3-max
75.12
96.83
53.41
5.172
1.17
550.043 ms
40,419,858
895.38 MB
#36
openrouter
polaris-alpha
75.10
98.70
51.51
2.754
1.00
1,160.78 ms
71,383,218
1,133.85 MB
#37
anthropic
claude-opus-4.6-fast
75.03
99.24
50.82
2.813
1.02
2,053.426 ms
34,047,552
206.91 MB
#38
openrouter
horizon-alpha
75.03
98.65
51.40
1.362
1.02
1,358.24 ms
67,797,316
1,137.85 MB
#39
openai
gpt-4.1
75.01
99.95
50.08
2.074
1.00
421.6 ms
52,027,773
246.69 MB
#40
openai
gpt-4.5-preview
75.00
99.49
50.51
2.955
1.00
442.98 ms
41,636,677
756.28 MB
#41
openai
gpt-5-codex
74.96
98.90
51.01
11.124
1.00
596.52 ms
48,389,329
1,097.99 MB
#42
openai
o3
74.84
99.84
49.84
16.292
1.04
549.54 ms
53,315,039
303.04 MB
#43
openai
gpt-oss-120b:exacto
74.82
98.05
51.60
5.261
1.08
1,351.14 ms
51,060,147
1,021.13 MB
#44
anthropic
claude-opus-4.8
74.82
94.54
55.10
2.776
1.02
852.933 ms
32,995,641
866.60 MB
#45
openai
gpt-4o-2024-08-06
74.82
98.35
51.29
1.375
1.02
1,946.04 ms
49,630,116
923.17 MB
#46
qwen
qwen3-next-80b-a3b-instruct
74.81
99.62
50.00
1.474
1.00
421.14 ms
38,561,447
879.98 MB
#47
openai
gpt-5.5
74.74
94.33
55.16
8.863
1.00
3,651.48 ms
39,451,456
950.34 MB
#48
qwen
qwen3-coder:exacto
74.73
98.20
51.26
1.292
1.02
1,383.265 ms
46,171,856
959.85 MB
#49
x-ai
grok-3-mini-beta
74.73
98.94
50.52
7.127
1.06
651.74 ms
55,296,404
869.75 MB
#50
anthropic
claude-4.5-sonnet
74.59
99.34
49.84
3.702
1.02
585.62 ms
43,365,288
907.62 MB
#51
openai
gpt-oss-20b
74.57
99.13
50.00
2.190
1.06
818.38 ms
54,736,481
995.56 MB
#52
openai
o4-mini-high
74.55
98.83
50.26
21.133
1.04
702.64 ms
68,364,075
1,005.01 MB
#53
anthropic
claude-sonnet-4
74.48
98.35
50.61
3.915
1.02
492.708 ms
41,642,822
913.54 MB
#54
deepseek
deepseek-chat-v3-0324
74.39
99.68
49.10
5.366
1.24
362.62 ms
39,914,537
612.03 MB
#55
nousresearch
hermes-3-llama-3.1-405b:free
74.38
99.32
49.45
7.260
1.12
552.64 ms
52,968,329
862.73 MB
#56
google
gemma-4-26b-a4b-it
74.35
98.07
50.63
5.060
1.06
6,086.8 ms
56,772,354
393.05 MB
#57
qwen
qwen3-235b-a22b
74.31
99.28
49.34
36.262
1.04
439.38 ms
45,468,824
791.67 MB
#58
openai
gpt-oss-120b:free
74.28
99.61
48.95
8.630
1.06
1,214.36 ms
47,908,602
340.50 MB
#59
openai
gpt-4o-2024-11-20
74.20
98.17
50.24
0.876
1.06
1,866.388 ms
55,625,851
880.60 MB
#60
openai
gpt-5-chat
74.15
98.38
49.93
1.158
1.00
1,920.74 ms
61,160,789
1,027.66 MB
#61
x-ai
grok-4-fast
74.12
98.10
50.15
10.611
1.00
762.122 ms
50,156,062
1,088.51 MB
#62
mistralai
mistral-medium-3.1
74.12
99.42
48.81
2.088
1.04
666.02 ms
53,051,447
878.95 MB
#63
qwen
qwen3-next-80b-a3b-thinking
73.96
97.96
49.96
17.344
1.02
720.837 ms
54,897,195
1,106.61 MB
#64
meta-llama
llama-4-maverick
73.96
99.61
48.30
3.095
1.04
410.78 ms
40,161,866
793.26 MB
#65
deepseek
deepseek-chat-v3.1
73.90
99.58
48.22
3.914
1.04
608.681 ms
45,651,463
322.46 MB
#66
google
gemini-3.1-flash-lite
73.89
98.77
49.00
1.281
1.00
1,465.88 ms
53,680,420
964.11 MB
#67
deepcogito
cogito-v2-preview-llama-70b
73.86
99.20
48.51
2.851
1.10
847.14 ms
42,613,427
792.70 MB
#68
qwen
qwen-2.5-coder-32b-instruct
73.82
99.16
48.49
2.456
1.08
732.878 ms
46,841,414
767.00 MB
#69
google
gemini-2.5-pro-preview-06-05
73.70
99.09
48.32
20.782
1.04
579.36 ms
38,815,820
806.77 MB
#70
openai
gpt-oss-20b:free
73.64
97.46
49.82
14.122
1.27
610.104 ms
46,853,009
965.28 MB
#71
openai
gpt-5.2-chat
73.54
97.65
49.42
4.156
1.02
1,685.42 ms
55,308,771
1,106.43 MB
#72
openai
gpt-4o-2024-05-13
73.47
98.83
48.10
1.366
1.04
1,364.88 ms
49,226,881
913.08 MB
#73
mistralai
codestral-2508
73.44
99.66
47.22
0.855
1.00
775.14 ms
42,657,411
620.15 MB
#74
google
gemini-2.0-flash-lite-001
73.17
98.38
47.95
1.286
1.02
1,285.714 ms
50,589,629
969.49 MB
#75
qwen
qwen3-30b-a3b-thinking-2507
73.16
99.03
47.30
19.002
1.02
602.48 ms
45,928,106
890.50 MB
#76
openai
gpt-3.5-turbo-0613
73.03
97.97
48.08
1.101
1.04
1,539.143 ms
47,348,539
989.91 MB
#77
google
gemini-2.5-flash
73.01
99.94
46.09
2.126
1.02
337.4 ms
36,295,667
262.45 MB
#78
qwen
qwen3-coder-flash
73.01
99.46
46.56
2.453
1.02
556.74 ms
42,185,121
868.81 MB
#79
anthropic
claude-3.5-haiku
72.91
99.90
45.93
2.731
1.08
522.38 ms
47,370,988
297.58 MB
#80
openai
o1
72.65
96.79
48.51
10.003
1.02
1,443.469 ms
54,867,793
1,033.70 MB
#81
x-ai
grok-code-fast-1
72.58
98.71
46.46
6.570
1.02
830.98 ms
51,488,460
1,077.86 MB
#82
thedrummer
cydonia-24b-v4.1
72.55
99.42
45.68
1.966
1.10
412.306 ms
36,265,794
823.24 MB
#83
meta-llama
llama-4-scout
72.46
99.97
44.96
2.048
1.04
289.875 ms
39,101,618
134.66 MB
#84
google
gemini-3.1-flash-lite-preview
72.44
93.48
51.39
1.517
1.00
7,438.06 ms
53,824,558
965.34 MB
#85
openai
gpt-5.1-codex-max
72.43
97.86
47.00
4.821
1.04
1,703 ms
59,148,959
1,016.79 MB
#86
openai
gpt-4o-mini
72.38
99.94
44.83
2.145
1.04
690.28 ms
54,131,214
193.58 MB
#87
openai
gpt-oss-120b
72.33
97.98
46.68
3.205
1.09
533.957 ms
41,234,766
980.50 MB
#88
openai
gpt-5.1
72.33
97.30
47.36
9.993
1.02
1,506.08 ms
53,506,834
1,119.95 MB
#89
google
gemini-2.5-flash-image
72.32
99.91
44.74
1.301
1.00
408.34 ms
39,751,249
369.30 MB
#90
google
gemma-3-27b-it
72.25
98.80
45.70
5.021
1.06
694.878 ms
41,212,019
869.17 MB
#91
openai
gpt-5.4-mini
72.20
97.59
46.80
1.300
1.02
1,981.02 ms
36,763,125
1,070.25 MB
#92
x-ai
grok-3-beta
71.95
99.14
44.76
1.701
1.04
633.612 ms
42,572,577
720.40 MB
#93
anthropic
claude-sonnet-4.6
71.83
93.83
49.83
3.549
1.00
5,336.72 ms
46,682,298
961.64 MB
#94
openai
gpt-4.1-mini
71.68
98.26
45.10
1.945
1.00
919.562 ms
44,180,776
913.74 MB
#95
z-ai
glm-4.6:exacto
71.67
96.66
46.68
10.752
1.04
838.771 ms
48,329,302
1,130.98 MB
#96
openai
gpt-5
71.59
98.53
44.64
25.613
1.04
643.3 ms
61,356,069
1,161.59 MB
#97
anthropic
claude-haiku-4.5
71.52
99.46
43.57
1.671
1.12
553.44 ms
49,735,847
918.65 MB
#98
openai
gpt-4o-mini-2024-07-18
71.46
99.52
43.39
2.137
1.04
2,280.84 ms
54,031,752
342.74 MB
#99
openai
gpt-5.3-codex
71.26
98.29
44.24
10.635
1.00
976.7 ms
46,652,690
1,054.90 MB
#100
openai
gpt-4o-mini-search-preview
71.18
99.78
42.58
2.766
1.06
748.437 ms
31,706,247
214.85 MB
#101
google
gemini-2.5-flash-lite-preview-09-2025
71.15
98.73
43.57
1.611
1.08
1,410.02 ms
43,979,391
864.07 MB
#102
deepseek
deepseek-chat-v3-0324:free
71.07
96.13
46.01
5.875
1.11
383.682 ms
38,010,973
813.72 MB
#103
google
gemini-2.0-flash-001
70.99
99.94
42.04
1.426
1.02
350.146 ms
44,547,543
181.54 MB
#104
mistralai
devstral-medium
70.84
99.61
42.08
1.405
1.08
420.714 ms
44,380,748
715.11 MB
#105
mistralai
voxtral-small-24b-2507
70.75
99.38
42.12
0.902
1.02
610.163 ms
41,840,418
837.25 MB
#106
anthropic
claude-sonnet-4.5
70.74
93.30
48.18
3.491
1.00
6,180.14 ms
43,114,181
899.14 MB
#107
google
gemma-4-31b-it
70.73
90.63
50.82
5.276
1.00
7,081.24 ms
56,772,354
1,050.70 MB
#108
nvidia
nemotron-nano-9b-v2
70.70
98.80
42.59
12.717
1.33
483.347 ms
40,823,966
813.29 MB
#109
qwen
qwen3-30b-a3b-instruct-2507
70.57
99.69
41.44
2.806
1.09
308 ms
31,184,916
374.88 MB
#110
mistralai
magistral-small-2506
70.03
99.10
40.96
12.425
1.18
522.531 ms
39,072,130
681.69 MB
#111
openai
gpt-5.4-nano
69.88
97.32
42.45
1.792
1.14
1,843.9 ms
45,378,797
1,138.96 MB
#112
google
gemini-2.5-flash-preview
69.77
99.64
39.90
1.622
1.00
384.551 ms
42,309,547
735.32 MB
#113
openai
gpt-5-nano
69.71
97.30
42.12
27.104
1.08
1,015.46 ms
45,844,074
1,126.98 MB
#114
openai
gpt-3.5-turbo-16k
69.51
99.73
39.30
1.272
1.33
673.958 ms
37,688,322
273.55 MB
#115
mistralai
ministral-14b-2512
69.22
99.49
38.95
1.251
1.04
1,188.128 ms
45,278,108
285.40 MB
#116
mistralai
devstral-small
69.18
98.78
39.58
2.412
1.09
372.696 ms
44,597,846
757.05 MB
#117
mistralai
mistral-small-2603
68.61
96.68
40.54
0.913
1.07
841.891 ms
34,809,725
941.72 MB
#118
google
gemini-2.5-flash-lite
68.35
99.87
36.83
1.512
1.04
850.96 ms
39,578,736
261.53 MB
#119
anthropic
claude-opus-4
67.85
98.62
37.07
4.243
1.97
352.457 ms
28,250,540
112.14 MB
#120
mistralai
ministral-8b-2512
67.55
97.06
38.05
1.397
1.21
1,240.521 ms
47,439,900
1,011.42 MB
#121
openai
gpt-5-mini
67.36
97.52
37.20
20.092
1.02
906.12 ms
61,641,565
1,386.39 MB
#122
mistralai
ministral-8b
67.29
99.73
34.86
0.925
1.00
385.911 ms
40,043,041
257.63 MB
#123
anthropic
claude-3-haiku
66.14
90.04
42.24
1.544
1.31
4,766.958 ms
34,045,851
774.89 MB
#124
mistralai
mistral-small-3.1-24b-instruct
65.97
87.92
44.02
1.809
1.00
376.5 ms
37,893,118
912.60 MB
#125
mistralai
mistral-nemo
64.74
99.48
30.00
3.307
1.09
680.644 ms
48,641,279
222.69 MB
#126
openai
gpt-4.1-nano
63.93
99.93
27.93
1.538
1.06
445.694 ms
52,428,071
239.26 MB
#127
openai
gpt-3.5-turbo
59.22
95.05
23.39
1.150
1.80
1,126.878 ms
36,206,482
218.84 MB
#128
meta-llama
llama-3.3-70b-instruct
56.31
77.07
35.56
3.501
1.21
445.242 ms
38,658,489
992.39 MB
#129
openai
gpt-5-pro
55.97
79.10
32.84
288.492
1.05
660.364 ms
47,670,686
1,025.31 MB
#130
openrouter
andromeda-alpha
54.66
94.32
15.00
28.858
1.82
358.176 ms
37,959,195
240.96 MB
#131
moonshotai
kimi-linear-48b-a3b-instruct
53.81
82.34
25.27
1.305
1.57
348.686 ms
18,091,302
755.15 MB
#132
liquid
lfm-2.2-6b
52.41
98.82
6.00
1.393
1.07
428.357 ms
77,197,005
224.10 MB
#133
liquid
lfm2-8b-a1b
50.73
99.46
2.00
1.389
1.08
294.462 ms
37,560,298
133.52 MB
#134
qwen
qwen2.5-coder-7b-instruct
50.46
98.62
2.30
5.664
1.00
264 ms
29,050,068
205.58 MB
#135
deepseek
deepseek-v3.1-terminus:exacto
50.28
58.65
41.91
3.206
1.56
950.439 ms
53,138,787
1,064.91 MB
#136
inclusionai
ring-1t
50.05
98.10
2.00
26.389
1.00
293 ms
7,321,600
116.24 MB
#138
openai
gpt-4-turbo
49.25
98.49
0.00
4.652
1.08
1,130.64 ms
40,869,345
918.65 MB
#139
openrouter
sherlock-dash-alpha
45.73
53.21
38.24
1.237
1.22
932.811 ms
56,016,230
1,309.17 MB
#140
alibaba
tongyi-deepresearch-30b-a3b
45.05
57.82
32.29
20.128
1.13
553.026 ms
57,570,886
1,153.90 MB
#141
baidu
ernie-4.5-21b-a3b-thinking
34.63
38.57
30.68
54.468
1.15
847.854 ms
48,091,065
1,108.87 MB
#142
perplexity
sonar-pro-search
20.33
5.71
34.95
3.717
2.45
683.182 ms
56,964,923
1,347.49 MB
#143
nvidia
nemotron-nano-12b-v2-vl:free
9.01
0.00
18.03
23.357
1.86
487.333 ms
57,185,740
1,558.26 MB