SoHoT benchmarks

Benchmarks

Each table is one exam on one machine: every run in it asked the same cases under the same conditions, and each candidate shows its latest result there. A changed case set starts a new table, newest first. Reference rows are a hosted model run as a ceiling and are never adopted. Wall-clock compares only within one machine; across machines, compare pass rates.

image
code
web
svg
video
music
decide
agent
ocr
retrieval
pii
claims
repo
extract
stt
tts

image

Mac Studio M5 Ultra 96 GB · 584 cases · 2 runs, latest 2026-11-12

candidate	pass	median s	first s	peak GB	metrics	run
mflux/dev-q8	80/81 0.99	28.91	37.10	20.6	staging 0.988	2026-11-12

This exam has one candidate and no reference row to compare against.

Earlier exams (4)

Mac Studio M5 Ultra 96 GB · 3 cases · case text 3b767bf2 · 1 run, 2026-10-19

candidate	pass	median s	first s	peak GB	metrics	run
mflux/dev-q8	3/3 1.00	34.91	35.37	20.4	cer 0.000	2026-10-19
diffusers/prxpixel-t2i	3/3 1.00	65.41	73.73	29.8	cer 0.000	2026-10-19

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases x 3 repeats · case text 2d7a8767 · 1 run, 2026-10-13

candidate	pass	median s	first s	peak GB	metrics	run
mflux/dev-q8	6/9 0.67	22.20	35.38	20.6		2026-10-13

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases x 3 repeats · case text a7c60c4c · 3 runs, latest 2026-10-06

candidate	pass	median s	first s	peak GB	metrics	run
mflux/filipstrand/Z-Image-Turbo-mflux-4bit-q8 4bit, by filipstrand source	9/9 1.00	8.30	25.14	10.1	cer 0.000	2026-10-06
mflux/dev-q8	9/9 1.00	14.15	106.03	20.5	cer 0.000	2026-10-05
mflux/flux2-klein-4b-q8 4B params	9/9 1.00	27.29	43.55	15.0	cer 0.000	2026-10-06
mflux/Tongyi-MAI/Z-Image-q8 by Tongyi-MAI source	7/9 0.78	16.91	74.70	17.9	cer 0.667	2026-10-06

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases · case text c4659450 · 1 run, 2026-10-05

candidate	pass	median s	first s	peak GB	metrics	run
mflux/flux2-klein-4b-q8 4B params	3/3 1.00	2.92	42.38	15.0	cer 0.000	2026-10-05

This exam has one candidate and no reference row to compare against.

Serving here: mflux:dev (adopted by-hand)

code

Mac Studio M5 Ultra 96 GB · 135 cases · llamacpp:Ornith-1.5-35B-Q4_K_M ctx 27648, llamacpp:Ornith-1.5-35B-Q4_K_M slots 2, llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M ctx 262144 · 1 run, 2026-11-12

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	19/24 0.79	3.04	0.99	0.0	co_tenant 1.000 code_pass 0.833 completion_tokens 983.375 tokens_per_s 128.725	2026-11-12
llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M family Qwen/Qwen3.6-35B-A3B, 35B (3B active) params, Q4_K_M, by unsloth source	18/23 0.78	32.28	9.42	0.0	code_pass 0.864 completion_tokens 3796.727 tokens_per_s 130.673	2026-11-12

This exam has no reference row to compare against.

Earlier exams (14)

Mac Studio M5 Ultra 96 GB · 135 cases · llamacpp:Ornith-1.5-35B-Q4_K_M ctx 27648, llamacpp:Ornith-1.5-35B-Q4_K_M slots 2, llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M ctx 221184 · 1 run, 2026-11-11

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M family Qwen/Qwen3.6-35B-A3B, 35B (3B active) params, Q4_K_M, by unsloth source	26/31 0.84	34.18	11.44	0.0	code_pass 0.926 completion_tokens 5090.677 tokens_per_s 131.513	2026-11-11
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	24/31 0.77	5.63	0.99	0.0	co_tenant 1.000 code_pass 0.814 completion_tokens 1474.161 tokens_per_s 133.339	2026-11-11

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 135 cases · llamacpp:Ornith-1.5-35B-Q4_K_M ctx 262144, llamacpp:Ornith-1.5-35B-Q4_K_M slots 4, llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M ctx not recorded · 26 runs, latest 2026-11-07

candidate	pass	median s	first s	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	31/36 0.86	3.59	139.88	code_pass 1.000 completion_tokens 534.936 tokens_per_s 131.142	2026-11-05
best-of-3/llamacpp:Ornith-1.5-35B-Q4_K_M: llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	112/135 0.83	46.23	5.48	code_pass 0.931 completion_tokens 11451.750 method_calls 3.000 method_chosen 1.137 method_samples 3.000	2026-10-12
llamacpp:ukisai_Swift-Qwen3.8-27b-Q4_K_M family ukisai/Swift-Qwen3.8-27b, 27B params, Q4_K_M, by bartowski source	111/135 0.82	40.43	8.96	code_pass 0.949 completion_tokens 4223.463 tokens_per_s 43.687	2026-10-29
llamacpp:Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M family Qwen/Qwen3.6-35B-A3B, 35B (3B active) params, Q4_K_M, by HauhauCS source	65/80 0.81	40.50	7.30	code_pass 0.914 completion_tokens 6964.513 tokens_per_s 138.054	2026-10-20
neutral/llamacpp:Ornith-1.5-35B-Q4_K_M: llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	59/75 0.79	11.39	2.43	code_pass 0.965 completion_tokens 2522.903 method_calls 1.000	2026-10-20
llamacpp:Qwen3.8-27B-Humanlike-Chat-Q4_K_M family huihui-ai/Huihui-Qwen3.8-27B-abliterated, 27B params, Q4_K_M, by LessThanThreeAI source	28/36 0.78	65.01	156.07	code_pass 0.929 completion_tokens 4761.871 tokens_per_s 45.061	2026-10-22
WeiboAI/VibeThinker-3B family Qwen/Qwen2.5-Coder-3B, 3B params, by WeiboAI source	61/80 0.76	35.43	5.28	code_pass 0.906 completion_tokens 6536.014 tokens_per_s 108.656	2026-10-17
plan/llamacpp:Ornith-1.5-35B-Q4_K_M: llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	96/135 0.71	17.92	1.89	code_pass 0.901 completion_tokens 4621.035 method_calls 2.000	2026-10-26
deepen/llamacpp:Ornith-1.5-35B-Q4_K_M: llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	64/90 0.71	43.04	43.88	code_pass 0.859 completion_tokens 7947.308 method_calls 2.000 tail_entropy 4.923 tail_repeat 0.000	2026-10-22
openbmb/MiniCPM5-2B family llama, 2B params, by openbmb source	36/53 0.68	41.69	2.86	code_pass 0.878 completion_tokens 6476.689 tokens_per_s 122.671	2026-10-15
mlx-community/NeoHorse-1-4B-4bit family TokenRhythm/NeoHorse-1-4B, 4B params, 4bit, by mlx-community source	14/29 0.48	43.85	6.65	code_pass 0.750 completion_tokens 4567.700 tokens_per_s 167.230	2026-11-07
openbmb/MiniCPM5-1B family llama, 1B params, by openbmb source	14/32 0.44	55.39	2.31	code_pass 0.748 completion_tokens 6583.524 tokens_per_s 214.271	2026-11-05
LiquidAI/LFM2.5-8B-A1B family LiquidAI/LFM2.5-8B-A1B-Base, 8B (1B active) params, by LiquidAI source	50/116 0.43	39.82	2.04	code_pass 0.709 completion_tokens 6549.533 tokens_per_s 165.756	2026-10-16
llamacpp:Qwen_Qwen3.5-4B-Q4_K_M family Qwen/Qwen3.5-4B, 4B params, Q4_K_M, by bartowski source	9/22 0.41	89.79	146.33	code_pass 0.855 completion_tokens 3233.364 tokens_per_s 162.973	2026-11-05
llamacpp:Qwen-Drive-1.0-4B-Q4_K_M family Qwen/Qwen-Drive-1.0-4B, 4B params, Q4_K_M, by Hob-forge source	18/77 0.23	1.53	0.78	code_pass 0.453 completion_tokens 883.894 tokens_per_s 149.241	2026-11-05
LiquidAI/LFM2.5-2.6B family LiquidAI/LFM2.5-2.6B-Base, 2.6B params, by LiquidAI source	18/135 0.13	0.01	4.19	code_pass 0.756 completion_tokens 4174.923 tokens_per_s 138.385	2026-10-19
llamacpp:Bonsai-8B-Q1_0 family prism-ml/Bonsai-8B-unpacked, 8B params, Q1_0, by prism-ml source	2/20 0.10	0.94	0.83	code_pass 0.280 completion_tokens 202.650 tokens_per_s 182.450	2026-11-03
llamacpp:microsoft_Fara-7B-Q4_K_M family microsoft/Fara-7B, 7B params, Q4_K_M, by bartowski source	10/135 0.07	0.94	0.53	code_pass 0.223 completion_tokens 380.985 tokens_per_s 127.304	2026-11-07
mlx-community/gemma-2-2b-it-4bit family gemma2, 2B params, 4bit, by mlx-community source	9/135 0.07	0.71	0.61	code_pass 0.244 completion_tokens 123.231 tokens_per_s 152.066	2026-11-07
mlx-community/Qwen2-VL-2B-Instruct-4bit family qwen2_vl, 2B params, 4bit, by mlx-community source	5/135 0.04	0.50	0.25	code_pass 0.170 completion_tokens 104.977 tokens_per_s 180.681	2026-11-05
Qwen/Qwen3.5-0.8B family Qwen/Qwen3.5-0.8B-Base, 0.8B params, by Qwen source	1/33 0.03	142.31	141.53	code_pass 1.000 completion_tokens 875.000 tokens_per_s 252.000	2026-10-16
LiquidAI/LFM2.5-350M family LiquidAI/LFM2.5-350M-Base, 350M params, by LiquidAI source	3/135 0.02	0.30	0.28	code_pass 0.147 completion_tokens 87.030 tokens_per_s 267.918	2026-11-05
llamacpp:LiquidAI_LFM2.5-350M-Base_1781204855.Q4_K_M family LiquidAI/LFM2.5-350M, 350M params, Q4_K_M, by SupraLabs source	0/41 0.00	0.26	0.09	code_pass 0.088 completion_tokens 2486.659 tail_entropy 3.827 tail_repeat 0.983 tokens_per_s 366.900	2026-11-02
llamacpp:LFM2.5-VL-3B-Q4_K_M family LiquidAI/LFM2.5-VL-3B, 3B params, Q4_K_M, by unsloth source	0/10 0.00	0.87	0.86	code_pass 0.040 completion_tokens 223.500 tokens_per_s 241.480	2026-11-03
llamacpp:Qwen3.5-0.8B-Q4_K_M family Qwen/Qwen3.5-0.8B, 0.8B params, Q4_K_M, by unsloth source	0/10 0.00	69.43	106.61		2026-11-05
llamacpp:Gemma-4-12B-OBLITERATED-Q4_K_M family google/gemma-4-12B-it, 12B params, Q4_K_M, by OBLITERATUS source	0/7 0.00	273.59	273.53		2026-11-07

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 316 cases · 2 runs, latest 2026-11-02

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:ukisai_Swift-Qwen3.8-27b-Q4_K_M family ukisai/Swift-Qwen3.8-27b, 27B params, Q4_K_M, by bartowski source	174/205 0.85	40.21	6.45	0.0	code_pass 0.947 completion_tokens 3451.166 tokens_per_s 43.422	2026-10-31
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	241/316 0.76	11.70	1.83	0.0	code_pass 0.938 completion_tokens 2866.214 tokens_per_s 139.317	2026-11-02

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 118 cases · reply budget 65536 tokens · 1 run, 2026-10-09

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	96/118 0.81	15.59	941.20	0.0	code_pass 0.918 completion_tokens 5263.148 tokens_per_s 136.938	2026-10-09

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 118 cases · reply budget 32000 tokens · 1 run, 2026-10-09

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	87/118 0.74	14.95	238.83	0.0	code_pass 0.914 completion_tokens 4197.722 tokens_per_s 137.896	2026-10-09

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 118 cases · reply budget 16000 tokens · 1 run, 2026-10-09

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	79/118 0.67	15.48	114.51	0.0	code_pass 0.917 completion_tokens 2864.034 tokens_per_s 138.285	2026-10-09

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 118 cases · reply budget 4000 tokens · 1 run, 2026-10-09

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	68/118 0.58	14.83	51.35	0.0	code_pass 0.943 completion_tokens 1138.151 tokens_per_s 137.251	2026-10-09

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 119 cases · reply budget 16000 tokens · 1 run, 2026-10-08

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	68/119 0.57	20.96	0.83	0.0	code_pass 0.844 completion_tokens 2875.430 tokens_per_s 139.342	2026-10-08

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 119 cases · reply budget 4000 tokens · 1 run, 2026-10-08

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	56/119 0.47	20.73	23.36	0.0	code_pass 0.830 completion_tokens 1148.638 tokens_per_s 135.280	2026-10-08

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 451 cases x 2 repeats · 1 run, 2026-10-08

candidate	pass	median s	first s	peak GB	metrics	run
plan/llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	504/902 0.56	14.25	1.93	0.0	code_pass 0.875 completion_tokens 1483.864 method_calls 2.000	2026-10-08
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	443/902 0.49	12.32	0.98	0.0	code_pass 0.787 completion_tokens 1130.946 tokens_per_s 138.303	2026-10-08

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 50 cases x 3 repeats · 3 runs, latest 2026-10-07

candidate	pass	median s	first s	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	116/150 0.77	2.50	0.90	code_pass 0.835 completion_tokens 657.299 tokens_per_s 135.608	2026-10-07
ibm-granite/granite-4.1-8b family granite, 8B params, by ibm-granite source	104/150 0.69	1.96	49.72	code_pass 0.808 completion_tokens 141.720 tokens_per_s 51.189	2026-10-07
zhengmh/OmniVTG-7B family qwen2_5_vl, 7B params, by zhengmh source	25/150 0.17	1.06	0.93	code_pass 0.291 completion_tokens 99.533 tokens_per_s 62.414	2026-10-07
LiquidAI/LFM2.5-350M family LiquidAI/LFM2.5-350M-Base, 350M params, by LiquidAI source	6/150 0.04	0.28	2.63	code_pass 0.125 completion_tokens 83.927 tokens_per_s 235.943	2026-10-07

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 14 cases x 3 repeats · 17 runs, latest 2026-10-06

candidate	pass	median s	first s	metrics	run
llamacpp:ornith15-trained-head family ornith-ai/Ornith-1.5-35B-A3B, 35B (3B active) params, by conFIGur8tor source	38/42 0.91	4.01	3.56	code_pass 0.956 completion_tokens 901.857 tokens_per_s 134.395	2026-10-06
llamacpp:Tiel-Coder-35B-A3B-UD-Q4_K_S family ornith-ai/Ornith-1.5-35B-A3B, 35B (3B active) params, Q4_K_S, by peculiar-ragdoll source	37/42 0.88	5.23	78.23	code_pass 0.975 completion_tokens 897.250 tokens_per_s 122.888	2026-10-06
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	35/42 0.83	2.79	80.17	code_pass 0.925 completion_tokens 779.659 tokens_per_s 135.954	2026-10-06
llamacpp:Swift-Qwen3.8-27B-Q4_K_M family ukisai/Swift-Qwen3.8-27b, 27B params, Q4_K_M, by ukisai source	33/42 0.79	19.90	12.38	code_pass 0.978 completion_tokens 1063.314 tokens_per_s 44.506	2026-10-06
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (served as q3-coder) 30B (3B active) params, 4bit, by mlx-community source	32/42 0.76	1.10	51.74	code_pass 0.901 completion_tokens 211.833 tokens_per_s 120.921	2026-10-06
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	30/42 0.71	0.86	49.99	code_pass 0.859 completion_tokens 214.381 tokens_per_s 120.517	2026-10-06
llamacpp:command-a-plus-05-2026-IQ1_S family CohereLabs/command-a-plus-05-2026-bf16, IQ1_S, by bartowski source	27/42 0.64	36.97	36.21	code_pass 0.993 completion_tokens 1386.321 tokens_per_s 56.450	2026-10-06
llamacpp:Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp family Qwen/Qwen3.8-27B, 27B params, IQ3_S, by ISTA-DASLab source	27/42 0.64	47.76	12.67	code_pass 0.964 completion_tokens 1618.857 tokens_per_s 48.082	2026-10-06
llamacpp:Nemotron3-Nano-4B-Uncensored-HauhauCS-Aggressive-Q4_K_M family nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16, 4B params, Q4_K_M, by HauhauCS source	20/42 0.48	2.27	1.61	code_pass 0.774 completion_tokens 678.744 tokens_per_s 162.523	2026-10-06
Qwen/Qwen3-8B family Qwen/Qwen3-8B-Base, 8B params, by Qwen source	19/42 0.45	67.60	15.33	code_pass 0.964 completion_tokens 1613.381 tokens_per_s 58.052	2026-10-06
mlx-community/Qwen3-8B-4bit family Qwen/Qwen3-8B, 8B params, 4bit, by mlx-community source	17/42 0.41	28.55	11.87	code_pass 0.979 completion_tokens 1990.889 tokens_per_s 140.783	2026-10-06
llamacpp:Qwen3.5-9B-DeepSeek-V4-Flash-Q4_K_M family unsloth/Qwen3.5-9B, 9B params, Q4_K_M, by Jackrong source	17/42 0.41	31.49	4.79	code_pass 1.000 completion_tokens 914.235 tokens_per_s 124.371	2026-10-06
LiquidAI/LFM2.5-1.2B-Instruct family LiquidAI/LFM2.5-1.2B-Base, 1.2B params, by LiquidAI source	15/42 0.36	0.39	1.02	code_pass 0.566 completion_tokens 157.286 tokens_per_s 283.098	2026-10-06
mlx-community/gemma-2-2b-it-4bit family gemma2, 2B params, 4bit, by mlx-community source	11/42 0.26	0.92	1.08	code_pass 0.578 completion_tokens 211.262 tokens_per_s 113.524	2026-10-06
llamacpp:supergemma4-26b-uncensored-fast-v2-Q4_K_M family google/gemma-4-26B-A4B-it, 26B (4B active) params, Q4_K_M, by Jiunsong source	8/42 0.19	28.84	30.88	code_pass 1.000 completion_tokens 2070.375 tokens_per_s 139.500	2026-10-06
mlx-community/Qwen2-VL-2B-Instruct-4bit family qwen2_vl, 2B params, 4bit, by mlx-community source	6/42 0.14	0.43	3.33	code_pass 0.408 completion_tokens 258.881 tokens_per_s 136.552	2026-10-06
Qwen/Qwen3-0.6B 0.6B params, by Qwen source	6/42 0.14	14.90	4.99	code_pass 0.621 completion_tokens 1517.941 tokens_per_s 220.276	2026-10-06
Qwen/Qwen3.5-0.8B family Qwen/Qwen3.5-0.8B-Base, 0.8B params, by Qwen source	3/42 0.07	14.89	1.80	code_pass 0.596 completion_tokens 714.250 tokens_per_s 140.637	2026-10-06
claude-code:claude-opus-5-5 reference, not adoptable	42/42 1.00	2.88	4.59	code_pass 1.000 completion_tokens 309.810 tokens_per_s 64.162	2026-10-06

Mac Studio M5 Ultra 96 GB · 9 cases x 3 repeats · 7 runs, latest 2026-10-06

candidate	pass	median s	first s	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	27/27 1.00	2.35	2.80	code_pass 1.000 completion_tokens 296.741 tokens_per_s 134.159	2026-10-06
llamacpp:Tiel-Coder-35B-A3B-UD-Q4_K_S family ornith-ai/Ornith-1.5-35B-A3B, 35B (3B active) params, Q4_K_S, by peculiar-ragdoll source	26/27 0.96	2.33	55.46	code_pass 0.982 completion_tokens 376.778 tokens_per_s 122.115	2026-10-06
llamacpp:ornith15-trained-head family ornith-ai/Ornith-1.5-35B-A3B, 35B (3B active) params, by conFIGur8tor source	25/27 0.93	2.48	2.40	code_pass 0.969 completion_tokens 526.593 tokens_per_s 133.648	2026-10-06
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (served as q3-coder) 30B (3B active) params, 4bit, by mlx-community source	24/27 0.89	0.56	51.38	code_pass 0.982 completion_tokens 82.185 tokens_per_s 110.796	2026-10-05
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	24/27 0.89	0.60	47.88	code_pass 0.926 completion_tokens 80.815 tokens_per_s 111.978	2026-10-05
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	19/27 0.70	0.73	7.80	code_pass 0.886 completion_tokens 96.556 tokens_per_s 100.630	2026-10-06
Qwen/Qwen3.5-0.8B family Qwen/Qwen3.5-0.8B-Base, 0.8B params, by Qwen source	7/27 0.26	8.09	15.64	code_pass 0.672 completion_tokens 881.556 tokens_per_s 121.433	2026-10-06
claude-code:claude-opus-5-5 reference, not adoptable	27/27 1.00	2.58	2.58	code_pass 1.000 completion_tokens 149.741 tokens_per_s 49.333	2026-10-05

Mac Studio M5 Ultra 96 GB · 9 cases · 2 runs, latest 2026-10-05

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	7/9 0.78	0.73	7.63	0.0	code_pass 0.911 completion_tokens 97.444 tokens_per_s 132.900	2026-10-05
claude-code:claude-opus-5-5 reference, not adoptable	9/9 1.00	5.73	5.71	0.0	code_pass 1.000 completion_tokens 144.556 tokens_per_s 24.333	2026-10-05

Serving here: llamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai (adopted measured)

web

Mac Studio M5 Ultra 96 GB · 1305 cases · 5 runs, latest 2026-11-12

candidate	pass	median s	first s	metrics	run
best-of-3/q3-4b: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	124/195 0.64	2.94	11.89	completion_tokens 610.774 ink 0.084 method_calls 1.810 method_chosen 1.041 method_samples 3.000	2026-11-12
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	119/194 0.61	2.21	13.25	completion_tokens 330.299 ink 0.075 tokens_per_s 117.028	2026-11-12
neutral/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	142/683 0.21	2.01	0.99	completion_tokens 279.954 ink 0.046 method_calls 1.000	2026-11-09

This exam has no reference row to compare against.

Earlier exams (8)

Mac Studio M5 Ultra 96 GB · 405 cases · 3 runs, latest 2026-11-10

candidate	pass	median s	first s	metrics	run
deepen/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	247/405 0.61	4.88	8.90	completion_tokens 829.658 ink 0.131 method_calls 2.000	2026-11-09
best-of-3/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	235/405 0.58	3.23	1.65	completion_tokens 622.775 ink 0.050 method_calls 1.963 method_chosen 1.096 method_samples 3.000	2026-11-09
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	219/405 0.54	1.71	1.00	completion_tokens 295.017 ink 0.053 tokens_per_s 145.650	2026-11-10
plan/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	206/405 0.51	2.50	2.08	completion_tokens 395.361 ink 0.032 method_calls 2.000	2026-11-10

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 900 cases · 2 runs, latest 2026-11-09

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	7/7 1.00	1.52	9.46	completion_tokens 459.143 ink 0.110 tokens_per_s 124.243	2026-11-09
best-of-3/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	6/6 1.00	2.46	8.04	completion_tokens 360.667 ink 0.116 method_calls 1.000 method_chosen 1.000 method_samples 3.000	2026-11-09
deepen/mlx-community/Qwen3-4B-Instruct-2507-4bit: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0/1 0.00	18.27	18.27		2026-11-09

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 1305 cases x 2 repeats · 1 run, 2026-11-02

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	1464/2610 0.56	1.66	8.24	0.0	completion_tokens 300.541 ink 0.055 tokens_per_s 148.543	2026-11-02

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 5 cases x 2 repeats · 3 runs, latest 2026-10-30

candidate	pass	median s	first s	metrics	run
best-of-3/q3-4b: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	10/10 1.00	6.74	21.88	completion_tokens 1289.100 ink 0.433 method_calls 1.000 method_chosen 1.000 method_samples 3.000	2026-10-19
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	10/10 1.00	7.58	7.28	completion_tokens 1228.800 ink 0.399 tokens_per_s 159.790	2026-10-30
plan/q3-4b: mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	10/10 1.00	9.05	16.87	completion_tokens 1383.900 ink 0.391 method_calls 2.000	2026-10-19

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 5 cases · case text 3fbc8167 · 1 run, 2026-10-12

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	5/5 1.00	12.51	7.23	0.0	completion_tokens 1267.800 ink 0.494 tokens_per_s 105.220	2026-10-12

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 5 cases x 3 repeats · case text 0bf94a70 · 2 runs, latest 2026-10-11

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	15/15 1.00	7.97	8.10	completion_tokens 1279.933 ink 0.438 tokens_per_s 167.300	2026-10-11
best-of-3/q3-4b 4B params	15/15 1.00	33.58	43.45	completion_tokens 3787.400 ink 0.364 method_calls 3.000 method_chosen 1.000 method_samples 3.000	2026-10-11
plan/q3-4b 4B params	14/15 0.93	6.83	9.39	completion_tokens 1447.067 ink 0.437 method_calls 2.000	2026-10-11

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 5 cases x 3 repeats · case text 3ff5f64a · 1 run, 2026-10-05

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	15/15 1.00	7.30	15.46	completion_tokens 1308.133 ink 0.450 tokens_per_s 163.667	2026-10-05
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	12/15 0.80	7.31	10.12	completion_tokens 1056.667 ink 0.582 tokens_per_s 118.267	2026-10-05
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (served as q3-coder) 30B (3B active) params, 4bit, by mlx-community source	12/15 0.80	7.77	11.44	completion_tokens 1028.800 ink 0.455 tokens_per_s 118.907	2026-10-05
claude-code:claude-opus-5-5 reference, not adoptable	14/15 0.93	17.84	22.57	completion_tokens 2805.857 ink 0.623 tokens_per_s 126.150	2026-10-05

Mac Studio M5 Ultra 96 GB · 5 cases · case text f2469983 · 1 run, 2026-10-05

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	5/5 1.00	11.07	8.26	0.0	completion_tokens 1311.200 ink 0.399 tokens_per_s 129.840	2026-10-05

This exam has one candidate and no reference row to compare against.

Serving here: mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b); 4B params, 4bit, by mlx-community (adopted measured)

svg

Mac Studio M5 Ultra 96 GB · 628 cases · 2 runs, latest 2026-11-11

candidate	pass	median s	first s	peak GB	metrics	run
plan/q3-30b: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	454/628 0.72	4.53	6.17	0.0	completion_tokens 567.452 ink 0.385 method_calls 2.000 svg_bytes 654.382	2026-11-11
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	434/628 0.69	2.45	1.74	0.0	completion_tokens 328.292 ink 0.326 svg_bytes 591.624 tokens_per_s 113.984	2026-11-11

This exam has no reference row to compare against.

Earlier exams (7)

Mac Studio M5 Ultra 96 GB · 295 cases · 2 runs, latest 2026-11-10

candidate	pass	median s	first s	peak GB	metrics	run
plan/q3-30b: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	217/295 0.74	4.58	2.94	0.0	completion_tokens 584.864 ink 0.375 method_calls 2.000 svg_bytes 691.014	2026-11-10
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	200/295 0.68	2.57	1.76	0.0	completion_tokens 338.680 ink 0.308 svg_bytes 578.368 tokens_per_s 114.466	2026-11-10

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 923 cases · 2 runs, latest 2026-11-10

candidate	pass	median s	first s	metrics	run
best-of-3/q3-30b: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	607/923 0.66	3.03	3.39	completion_tokens 486.848 ink 0.309 method_calls 1.615 method_chosen 1.022 method_samples 3.000 svg_bytes 545.358	2026-11-09
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	379/923 0.41	1.58	4.24	completion_tokens 329.049 ink 0.321 svg_bytes 588.694 tokens_per_s 109.292	2026-11-10
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	291/923 0.32	1.34	2.58	completion_tokens 319.072 ink 0.406 svg_bytes 564.378 tokens_per_s 125.835	2026-11-10

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 923 cases x 2 repeats · 3 runs, latest 2026-11-09

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	263/388 0.68	2.77	4.72	0.0	busy_wait_s 5.078 completion_tokens 347.331 ink 0.338 svg_bytes 613.395 tokens_per_s 108.160	2026-11-09
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	206/380 0.54	2.54	2.28	0.0	completion_tokens 334.795 ink 0.421 svg_bytes 568.864 tokens_per_s 110.093	2026-11-09

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases · 3 runs, latest 2026-10-19

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	3/3 1.00	2.95	4.13	0.0	completion_tokens 259.000 ink 0.181 svg_bytes 568.667 tokens_per_s 94.267	2026-10-19
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	3/3 1.00	3.48	4.45	0.0	completion_tokens 303.000 ink 0.199 svg_bytes 697.667 tokens_per_s 101.800	2026-10-19
trace/mflux/dev-q8	2/2 1.00	37.44	35.52	20.4	ink 0.227 svg_bytes 30014.500	2026-10-19
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	2/3 0.67	1.36	2.94	0.0	completion_tokens 241.333 ink 0.342 svg_bytes 629.500 tokens_per_s 143.300	2026-10-05
omnisvg:8B	1/2 0.50	173.58	172.27	47.7	ink 0.660 svg_bytes 5065.000	2026-10-19

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases x 2 repeats · 2 runs, latest 2026-10-19

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	6/6 1.00	1.34	4.48	0.0	completion_tokens 270.667 ink 0.163 svg_bytes 615.667 tokens_per_s 116.800	2026-10-19
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	6/6 1.00	2.15	3.75	0.0	completion_tokens 306.333 ink 0.223 svg_bytes 703.333 tokens_per_s 113.000	2026-10-19
best-of-3/q3-30b: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit 30B (3B active) params, 4bit, by mlx-community source	6/6 1.00	2.73	5.94	0.0	completion_tokens 367.000 ink 0.215 method_calls 1.000 method_chosen 1.000 method_samples 3.000 svg_bytes 814.167	2026-10-19
omnisvg:4B	2/4 0.50	75.67	100.84	22.4	ink 0.755 svg_bytes 6594.500	2026-10-19

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases x 8 repeats · 2 runs, latest 2026-10-14

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	24/24 1.00	1.60	4.29	0.0	completion_tokens 272.417 ink 0.208 svg_bytes 625.500 tokens_per_s 110.763	2026-10-14
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	11/24 0.46	1.11	1.97	0.0	completion_tokens 213.500 ink 0.143 svg_bytes 625.818 tokens_per_s 138.217	2026-10-14

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 3 cases x 3 repeats · 2 runs, latest 2026-10-06

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	9/9 1.00	1.56	53.88	completion_tokens 286.778 ink 0.208 svg_bytes 640.556 tokens_per_s 110.989	2026-10-06
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	5/9 0.56	2.09	15.25	completion_tokens 197.111 ink 0.113 svg_bytes 520.600 tokens_per_s 80.589	2026-10-06
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (served as q3-coder) 30B (3B active) params, 4bit, by mlx-community source	5/9 0.56	2.42	4.47	completion_tokens 679.889 ink 0.193 svg_bytes 428.800 tokens_per_s 117.289	2026-10-05
claude-code:claude-opus-5-5 reference, not adoptable	9/9 1.00	9.72	5.99	completion_tokens 1192.333 ink 0.312 svg_bytes 659.667 tokens_per_s 97.089	2026-10-05

Serving here: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b); 30B (3B active) params, 4bit, by mlx-community (adopted measured)

video

Mac Studio M5 Ultra 96 GB · 1 case x 2 repeats · 1 run, 2026-11-09

candidate	pass	median s	first s	peak GB	metrics	run
diffusers-video/Motif-Video-2B 2B params	2/2 1.00	50.76	52.75	21.5	motion 0.083	2026-11-09
h3/minimax-h3	2/2 1.00	171.53	170.73	3.5	motion 0.038	2026-11-09

This exam has no reference row to compare against.

Earlier exams (2)

Mac Studio M5 Ultra 96 GB · 1 case · case text e110870e · 2 runs, latest 2026-11-02

candidate	pass	median s	first s	peak GB	metrics	run
h3/minimax-h3	1/1 1.00	102.51	102.51	3.2	motion 0.042	2026-11-02

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 1 case · case text 06d7862d · 7 runs, latest 2026-10-20

candidate	pass	median s	first s	peak GB	metrics	run
h3/minimax-h3	1/1 1.00	27.84	27.84	5.7	motion 0.042	2026-10-20
h3/minimax-h3@ssd_streaming=false	1/1 1.00	28.73	28.73	18.4	motion 0.034	2026-10-05
diffusers-video/h3	0/1 0.00	7.29	7.29	0.3		2026-10-05

This exam has no reference row to compare against.

Serving here: h3

music

Mac Studio M5 Ultra 96 GB · 4 cases · 2 runs, latest 2026-11-02

candidate	pass	median s	first s	peak GB	metrics	run
acestep/acestep-v15-turbo	4/4 1.00	28.42	53.48	46.0	wer 0.339 duration_error_s 0.000 seconds 30.000	2026-11-02

This exam has one candidate and no reference row to compare against.

Earlier exams (1)

Mac Studio M5 Ultra 96 GB · 4 cases · 6 runs, latest 2026-10-22

candidate	pass	median s	first s	peak GB	metrics	run
acestep/acestep-v15-turbo	4/4 1.00	45.81	54.68	45.6	wer 0.321 duration_error_s 0.000 seconds 30.000	2026-10-22
acestep/acestep-v15-turbo@steps=8	3/4 0.75	53.05	38.97	46.0	wer 0.615 duration_error_s 0.000 seconds 30.000	2026-10-05

This exam has no reference row to compare against.

Serving here: acestep:acestep-v15-turbo

decide

Mac Studio M5 Ultra 96 GB · 661 cases · 3 runs, latest 2026-11-12

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
llamacpp:imajev-4b-Q8_0 4B params, Q8_0, by mindchain source	0.823	241/529 0.46	0.46	0.58	0.0	decide_accuracy 0.823 decide_brier 0.258 decide_ece 0.030 calibrated 1.000 co_tenant 1.000 completion_tokens 36.461 tokens_per_s 77.903	2026-11-12
sky7350/Mica-v0.5-4B family Qwen/Qwen3.5-4B, 4B params, by sky7350 source	0.778	255/661 0.39	0.38	0.38	0.0	decide_accuracy 0.778 decide_brier 0.318 decide_ece 0.072 calibrated 1.000 completion_tokens 36.342 tokens_per_s 94.349	2026-11-05
nimble/Bespoke-Nimble-9B 9B params	0.764	43/196 0.22	4.35	14.19	18.9	decide_accuracy 0.764 decide_brier 0.375 decide_ece 0.125 calibrated 1.000	2026-11-09
bespokelabs/Bespoke-Nimble-9B family Qwen/Qwen3.5-9B, 9B params, by bespokelabs source	not run (harness_error)		--	--	--		2026-11-12
imajev-4b-Q8_0 4B params, Q8_0, by mindchain source serving	not run (harness_error)		--	--	--		2026-11-09

This exam has no reference row to compare against.

Earlier exams (5)

Mac Studio M5 Ultra 96 GB · 132 cases · 4 runs, latest 2026-11-11

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
llamacpp:imajev-4b-Q8_0 4B params, Q8_0, by mindchain source	0.790	54/132 0.41	0.47	0.72	0.0	decide_accuracy 0.790 decide_brier 0.319 decide_ece 0.078 calibrated 1.000 co_tenant 1.000 completion_tokens 36.818 tokens_per_s 74.882	2026-11-11
nimble/Bespoke-Nimble-9B-v2 9B params	0.742	49/132 0.37	4.25	10.70	20.2	decide_accuracy 0.742 decide_brier 0.317 decide_ece 0.052 calibrated 1.000	2026-11-10
bespokelabs/Bespoke-Nimble-9B family Qwen/Qwen3.5-9B, 9B params, by bespokelabs source	not run (harness_error)		--	--	--		2026-11-11
imajev-4b-Q8_0 4B params, Q8_0, by mindchain source serving	not run (harness_error)		--	--	--		2026-11-10

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 134 cases · 1 run, 2026-10-28

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
nimble/Bespoke-Nimble-9B 9B params	0.795	56/134 0.42	5.33	13.69	20.2	decide_accuracy 0.795 decide_brier 0.296 decide_ece 0.063 calibrated 1.000	2026-10-28
imajev-4b-Q8_0 4B params, Q8_0, by mindchain source serving	0.792	55/134 0.41	0.53	0.79	0.0	decide_accuracy 0.792 decide_brier 0.315 decide_ece 0.076 calibrated 1.000 completion_tokens 36.798 tokens_per_s 69.078	2026-10-28
llamacpp:APUS-OpenJev-v1-4B-Q8_0 4B params, Q8_0, by apus-ailab source	0.760	47/134 0.35	0.48	0.60	0.0	decide_accuracy 0.760 decide_brier 0.380 decide_ece 0.143 calibrated 1.000 completion_tokens 39.366 tokens_per_s 78.522	2026-10-28

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 476 cases · 2 runs, latest 2026-10-24

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
imajev-4b-Q8_0 4B params, Q8_0, by mindchain source serving	0.842	221/476 0.46	0.47	0.55	0.0	decide_accuracy 0.842 decide_brier 0.240 decide_ece 0.033 calibrated 1.000 completion_tokens 36.115 tokens_per_s 77.213	2026-10-24

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 137 cases · 2 runs, latest 2026-10-15

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
imajev-4b-Q8_0 (served as eval-imajev-4b) 4B params, Q8_0, by mindchain source	0.843	61/137 0.45	0.42	0.55	0.0	decide_accuracy 0.843 decide_brier 0.240 decide_ece 0.042 calibrated 1.000 completion_tokens 36.182 tokens_per_s 83.692	2026-10-08
imajev-4b-Q8_0 4B params, Q8_0, by mindchain source serving	0.843	61/137 0.45	0.49	0.76	0.0	decide_accuracy 0.843 decide_brier 0.240 decide_ece 0.042 calibrated 1.000 completion_tokens 36.182 tokens_per_s 73.913	2026-10-15

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 36 cases · 4 runs, latest 2026-10-07

candidate	decide_accuracy	all fields right	median s	first s	peak GB	metrics	run
imajev-4b-Q8_0 (served as eval-imajev-4b) 4B params, Q8_0, by mindchain source	0.762	8/36 0.22	0.39	0.57	0.0	decide_accuracy 0.762 decide_brier 0.359 decide_ece 0.140 calibrated 1.000 completion_tokens 37.528 tokens_per_s 90.717	2026-10-07
nimble/Bespoke-Nimble-9B 9B params	0.756	11/36 0.31	3.31	5.31	20.1	decide_accuracy 0.756 decide_brier 0.382 decide_ece 0.106 calibrated 1.000	2026-10-07
llamacpp:plumb-4b-v5-Q8_0 4B params, Q8_0, by crh225 source	0.744	8/36 0.22	0.45	17.88	0.0	decide_accuracy 0.744 decide_brier 0.415 decide_ece 0.159 calibrated 1.000 completion_tokens 33.167 tokens_per_s 74.381	2026-10-07
llamacpp:APUS-OpenJev-v1-4B-Q8_0 4B params, Q8_0, by apus-ailab source	0.726	9/36 0.25	0.51	16.49	0.0	decide_accuracy 0.726 decide_brier 0.447 decide_ece 0.188 calibrated 1.000 completion_tokens 38.722 tokens_per_s 76.667	2026-10-07
decider/strands-decider-2B-hobson-v21 2B params	0.714	9/36 0.25	2.53	14.98	5.8	decide_accuracy 0.714 decide_brier 0.391 decide_ece 0.087 calibrated 1.000	2026-10-07
llamacpp:decider-4b-v2.1-Q8_0 4B params, Q8_0, by Mapika source	0.702	5/36 0.14	0.56	17.05	0.0	decide_accuracy 0.702 decide_brier 0.479 decide_ece 0.178 calibrated 1.000 completion_tokens 37.611 tokens_per_s 66.400	2026-10-07
llamacpp:decider-4b.v2-Q4_K_M 4B params, Q4_K_M, by mindchain source	0.655	6/36 0.17	0.52	11.17	0.0	decide_accuracy 0.655 decide_brier 0.503 decide_ece 0.215 calibrated 1.000 completion_tokens 36.889 tokens_per_s 72.086	2026-10-07
apus-ailab/APUS-OpenJev-v1-4B-GGUF 4B params, by apus-ailab source	not run (refused_by_gateway)		--	--	--		2026-10-07
crh225/plumb-4b-GGUF 4B params, by crh225 source	not run (refused_by_gateway)		--	--	--		2026-10-07
Mapika/decider-4b-GGUF 4B params, by Mapika source	not run (refused_by_gateway)		--	--	--		2026-10-07
mindchain/decider-4b-v2-GGUF 4B params, by mindchain source	not run (refused_by_gateway)		--	--	--		2026-10-07
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	not run (refused_by_gateway)		--	--	--		2026-10-07

This exam has no reference row to compare against.

Serving here: imajev-4b-Q8_0; 4B params, Q8_0, by mindchain

agent

Mac Studio M5 Ultra 96 GB · 374 cases · 1 run, 2026-11-05

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	356/374 0.95	9.17	6.81	0.0	agent_ctx 262144.000 agent_model_s 11.044 agent_no_tool_steps 0.000 agent_steps 8.265 agent_tool_s 0.151 agent_ttft_sum_s 5.444 agent_valid_call_rate 1.000 agent_wall_s 11.259 completion_tokens 1435.979 prompt_tokens 14245.134	2026-11-05
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	52/374 0.14	2.22	2.62	0.0	agent_model_s 4.514 agent_no_tool_steps 0.024 agent_steps 5.660 agent_tool_s 0.028 agent_ttft_sum_s 1.500 agent_valid_call_rate 1.000 agent_wall_s 4.583 completion_tokens 667.869 prompt_tokens 6416.305	2026-11-05

This exam has no reference row to compare against.

Earlier exams (2)

Mac Studio M5 Ultra 96 GB · 14 cases · 5 runs, latest 2026-11-02

candidate	pass	median s	first s	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	14/14 1.00	8.27	5.91	agent_ctx 262144.000 agent_model_s 9.271 agent_no_tool_steps 0.000 agent_steps 5.643 agent_tool_s 0.073 agent_ttft_sum_s 5.940 agent_valid_call_rate 1.000 agent_wall_s 9.514 completion_tokens 878.000 prompt_tokens 43328.143	2026-11-02
sohot-code (model unknown)	14/14 1.00	9.19	7.46	agent_ctx 262144.000 agent_model_s 10.113 agent_no_tool_steps 0.000 agent_steps 5.786 agent_tool_s 0.099 agent_ttft_sum_s 6.493 agent_valid_call_rate 1.000 agent_wall_s 10.268 completion_tokens 889.000 prompt_tokens 34362.429	2026-10-06
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	4/14 0.29	2.57	2.23	agent_model_s 4.610 agent_no_tool_steps 0.143 agent_steps 3.714 agent_tool_s 0.001 agent_ttft_sum_s 2.631 agent_valid_call_rate 1.000 agent_wall_s 4.678 completion_tokens 325.571 prompt_tokens 12585.000	2026-11-02

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 14 cases x 3 repeats · 1 run, 2026-10-07

candidate	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	42/42 1.00	8.21	87.92	0.0	agent_ctx 262144.000 agent_model_s 10.749 agent_no_tool_steps 0.000 agent_steps 5.714 agent_tool_s 0.102 agent_ttft_sum_s 7.488 agent_valid_call_rate 1.000 agent_wall_s 10.902 completion_tokens 751.333 prompt_tokens 35016.357	2026-10-07
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b) 4B params, 4bit, by mlx-community source	14/42 0.33	4.31	9.99	0.0	agent_model_s 6.096 agent_no_tool_steps 0.071 agent_steps 3.976 agent_tool_s 0.008 agent_ttft_sum_s 3.404 agent_valid_call_rate 1.000 agent_wall_s 6.156 completion_tokens 377.333 prompt_tokens 14492.095	2026-10-07

This exam has no reference row to compare against.

Serving here: llamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai (adopted measured)

ocr

Mac Studio M5 Ultra 96 GB · 448 cases · 3 runs, latest 2026-11-11

candidate	pass	median s	first s	peak GB	metrics	run
hf-ocr/PaddleOCR-VL-1.6	371/448 0.83	7.64	18.15	0.0	cer 0.646	2026-11-11
osocr/auto	323/448 0.72	0.30	0.42	0.2	cer 0.170	2026-11-11

This exam has no reference row to compare against.

Earlier exams (4)

Mac Studio M5 Ultra 96 GB · 450 cases · 1 run, 2026-10-26

candidate	pass	median s	first s	peak GB	metrics	run
osocr/auto	448/450 1.00	0.28	24.31	0.3	cer 0.011	2026-10-26

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 444 cases · 1 run, 2026-10-26

candidate	pass	median s	first s	peak GB	metrics	run
osocr/auto	442/444 0.99	0.80	1.16	0.3	cer 0.011	2026-10-26

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 140 cases · 3 runs, latest 2026-10-22

candidate	pass	median s	first s	peak GB	metrics	run
osocr/auto	100/140 0.71	0.32	0.41	0.2	cer 0.190	2026-10-22

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 8 cases · 2 runs, latest 2026-10-07

candidate	pass	median s	first s	peak GB	metrics	run
osocr/auto	8/8 1.00	0.33	23.05	0.3	cer 0.004	2026-10-07
hf-ocr/PaddleOCR-VL-1.6	8/8 1.00	8.82	7.94	0.0	cer 0.000	2026-10-07

This exam has no reference row to compare against.

Serving here: hf-ocr:PaddlePaddle/PaddleOCR-VL-1.6 (adopted measured)

retrieval

Mac Studio M5 Ultra 96 GB · 189 cases · 6 runs, latest 2026-11-10

candidate	pass	median s	first s	peak GB	metrics	run
bm25/okapi	148/189 0.78	0.02	0.03	0.0	retrieval_ndcg 0.880 retrieval_recall 0.783	2026-11-10

This exam has one candidate and no reference row to compare against.

Earlier exams (1)

Mac Studio M5 Ultra 96 GB · 14 cases · 2 runs, latest 2026-10-07

candidate	pass	median s	first s	peak GB	metrics	run
rerank/ettin-reranker-1b-v1 1B params	13/14 0.93	3.24	3.78	5.6	retrieval_ndcg 0.946 retrieval_recall 0.964	2026-10-07
bm25/okapi	10/14 0.71	0.03	0.03	0.0	retrieval_ndcg 0.662 retrieval_recall 0.786	2026-10-07

This exam has no reference row to compare against.

Serving here: bm25

pii

Mac Studio M5 Ultra 96 GB · 556 cases · 2 runs, latest 2026-11-09

candidate	pass	median s	first s	peak GB	metrics	run
hf-pii/privacy-filter	462/556 0.83	3.92	13.55	4.0	pii_f1 0.890 pii_recall 0.871 pii_precision 0.910	2026-11-09
pii-regex/patterns	411/556 0.74	0.03	0.03	0.0	pii_f1 0.796 pii_recall 0.662 pii_precision 1.000	2026-11-09
hf-pii/Shield-82M 82M params	357/556 0.64	2.10	5.82	1.6	pii_f1 0.826 pii_recall 0.944 pii_precision 0.735	2026-11-05

This exam has no reference row to compare against.

Earlier exams (2)

Mac Studio M5 Ultra 96 GB · 166 cases · 5 runs, latest 2026-11-02

candidate	pass	median s	first s	peak GB	metrics	run
pii-regex/patterns	123/166 0.74	0.02	0.02	0.0	pii_f1 0.778 pii_recall 0.636 pii_precision 1.000	2026-11-02

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 16 cases · 2 runs, latest 2026-10-07

candidate	pass	median s	first s	peak GB	metrics	run
hf-pii/privacy-filter	16/16 1.00	4.18	4.06	4.0	pii_f1 0.977 pii_recall 1.000 pii_precision 0.955	2026-10-07
pii-regex/patterns	11/16 0.69	0.03	0.03	0.0	pii_f1 0.745 pii_recall 0.594 pii_precision 1.000	2026-10-07

This exam has no reference row to compare against.

Serving here: pii-regex

claims

Mac Studio M5 Ultra 96 GB · 299 cases · case text 360e1b7d · 4 runs, latest 2026-11-02

candidate	claims_balanced	pass	median s	first s	peak GB	metrics	run
Qwen2.5-7B-Instruct-Q4_K_M 7B params, Q4_K_M, by bartowski source serving	0.593	192/299 0.64	0.14	0.26	0.0	claims_balanced 0.593 claims_case_recall 0.521 claims_case_specificity 0.665 claims_empty_rate 0.668 claims_schema_valid 1.000 claims_bad_matched_cited_only 0.091 claims_bad_matched_conversation 0.000 claims_emitted 0.875 claims_judge_calibrated 0.000 claims_matcher_version 2.000 claims_near_miss_cited_only 0.182 claims_near_miss_conversation 0.067 claims_pass_cited_only 0.318 claims_pass_conversation 0.800 claims_precision_cited_only 0.821 claims_precision_conversation 0.922 claims_recall_cited_only 0.287 claims_recall_conversation 0.657 claims_unreviewed_cited_only 1.727 claims_unreviewed_conversation 1.467 claims_verbatim_cited_only 0.273 claims_verbatim_conversation 0.978 completion_tokens 27.260 tokens_per_s 65.195	2026-11-02

This exam has one candidate and no reference row to compare against.

Earlier exams (2)

Mac Studio M5 Ultra 96 GB · 299 cases · case text d82b0e17 · 1 run, 2026-10-08

candidate	claims_balanced	pass	median s	first s	peak GB	metrics	run
Qwen3-4B-Instruct-2507-Q4_K_M (served as eval-4b) 4B params, Q4_K_M, by unsloth source	0.644	258/299 0.86	0.12	3.73	0.0	claims_balanced 0.644 claims_case_recall 0.319 claims_case_specificity 0.968 claims_empty_rate 0.968 claims_schema_valid 0.980 claims_bad_matched_cited_only 0.095 claims_bad_matched_conversation 0.000 claims_emitted 0.648 claims_judge_calibrated 0.000 claims_near_miss_cited_only 0.571 claims_near_miss_conversation 0.405 claims_pass_cited_only 0.381 claims_pass_conversation 0.476 claims_precision_cited_only 0.853 claims_precision_conversation 0.970 claims_recall_cited_only 0.392 claims_recall_conversation 0.330 claims_unreviewed_cited_only 2.286 claims_unreviewed_conversation 2.905 claims_verbatim_cited_only 0.048 claims_verbatim_conversation 0.048 completion_tokens 34.261 tokens_per_s 62.132	2026-10-08
Qwen2.5-7B-Instruct-Q4_K_M (served as eval-7b) 7B params, Q4_K_M, by bartowski source	0.599	192/299 0.64	0.13	5.87	0.0	claims_balanced 0.599 claims_case_recall 0.532 claims_case_specificity 0.665 claims_empty_rate 0.665 claims_schema_valid 0.997 claims_bad_matched_cited_only 0.087 claims_bad_matched_conversation 0.000 claims_emitted 0.926 claims_judge_calibrated 0.000 claims_near_miss_cited_only 0.304 claims_near_miss_conversation 0.065 claims_pass_cited_only 0.348 claims_pass_conversation 0.783 claims_precision_cited_only 0.857 claims_precision_conversation 0.924 claims_recall_cited_only 0.300 claims_recall_conversation 0.652 claims_unreviewed_cited_only 1.696 claims_unreviewed_conversation 1.522 claims_verbatim_cited_only 0.174 claims_verbatim_conversation 0.913 completion_tokens 29.719 tokens_per_s 69.903	2026-10-08

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 274 cases · 1 run, 2026-10-08

candidate	claims_balanced	pass	median s	first s	peak GB	metrics	run
Qwen3-4B-Instruct-2507-Q4_K_M (served as eval-4b) 4B params, Q4_K_M, by unsloth source	0.643	250/274 0.91	0.12	3.71	0.0	claims_balanced 0.643 claims_case_recall 0.318 claims_case_specificity 0.968 claims_empty_rate 0.968 claims_schema_valid 0.985 claims_bad_matched 0.100 claims_emitted 0.433 claims_judge_calibrated 0.000 claims_precision 0.853 claims_recall 0.392 claims_unreviewed 2.400 completion_tokens 25.183 tokens_per_s 52.309	2026-10-08
Qwen2.5-7B-Instruct-Q4_K_M (served as eval-7b) 7B params, Q4_K_M, by bartowski source	0.492	174/274 0.64	0.13	0.65	0.0	claims_balanced 0.492 claims_case_recall 0.318 claims_case_specificity 0.665 claims_empty_rate 0.665 claims_schema_valid 0.996 claims_bad_matched 0.091 claims_emitted 0.722 claims_judge_calibrated 0.000 claims_precision 0.909 claims_recall 0.300 claims_unreviewed 1.773 completion_tokens 23.942 tokens_per_s 62.721	2026-10-08

This exam has no reference row to compare against.

Serving here: Qwen2.5-7B-Instruct-Q4_K_M; 7B params, Q4_K_M, by bartowski

repo

Mac Studio M5 Ultra 96 GB · 72 cases · reply budget 24576 tokens, case text 23b1e116 · 3 runs, latest 2026-11-07

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.333	4/12 0.33	36.57	21.19	resolved 0.333 patch_applied 0.333 completion_tokens 825.400 test_s 1.888 tokens_per_s 32.380 wall_s 28.733	2026-11-07
llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.200	1/5 0.20	592.50	642.91	resolved 0.200 patch_applied 0.400 completion_tokens 13413.000 test_s 36.203 tokens_per_s 33.967 wall_s 424.803	2026-11-07
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.100	1/10 0.10	201.81	205.59	resolved 0.100 patch_applied 0.100 completion_tokens 11054.000 test_s 122.386 tokens_per_s 86.700 wall_s 250.780	2026-11-07

This exam has no reference row to compare against.

Earlier exams (22)

Mac Studio M5 Ultra 96 GB · 72 cases · reply budget 32768 tokens, case text 23b1e116 · 5 runs, latest 2026-11-07

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.067	1/15 0.07	146.94	83.53	0.0	resolved 0.067 patch_applied 0.133 completion_tokens 13478.429 test_s 0.258 tokens_per_s 134.471 wall_s 101.678	2026-11-07
llamacpp:Qwopus3.5-27B-v3-Q4_K_M family unsloth/Qwen3.5-27B, 27B params, Q4_K_M, by Jackrong source	0.061	2/33 0.06	150.27	134.46	0.0	resolved 0.061 patch_applied 0.091 completion_tokens 8881.250 test_s 0.296 tokens_per_s 40.837 wall_s 233.513	2026-11-07
Qwen/Qwen3-0.6B 0.6B params, by Qwen source	0.000	0/14 0.00	0.00	0.00	0.0	resolved 0.000 patch_applied 0.000 completion_tokens 439.000 test_s 0.000 tokens_per_s 163.600 wall_s 3.022	2026-11-07
mlx-community/Qwen3-8B-4bit family Qwen/Qwen3-8B, 8B params, 4bit, by mlx-community source	not run (server_dead)		--	--	--		2026-11-06

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 72 cases · reply budget 32768 tokens, case text c21e484d · 3 runs, latest 2026-10-30

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.200	8/40 0.20	41.63	30.22	resolved 0.200 patch_applied 0.225 completion_tokens 1041.364 test_s 1.148 tokens_per_s 32.318 wall_s 34.590	2026-10-26
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.062	1/16 0.06	138.31	20.92	resolved 0.062 patch_applied 0.125 completion_tokens 8710.667 test_s 0.282 tokens_per_s 124.017 wall_s 73.321	2026-10-30
Qwen/Qwen3-0.6B 0.6B params, by Qwen source	0.000	0/15 0.00	5.66	202.01	resolved 0.000 patch_applied 0.000 completion_tokens 3075.867 test_s 0.000 tokens_per_s 149.620 wall_s 21.073	2026-10-30
llamacpp:Qwopus3.5-27B-v3-Q4_K_M family unsloth/Qwen3.5-27B, 27B params, Q4_K_M, by Jackrong source	0.000	0/4 0.00	264.31	212.12	resolved 0.000 patch_applied 0.000	2026-10-26

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 24 cases · 2 runs, latest 2026-10-30

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.333	8/24 0.33	170.89	309.08	0.0	resolved 0.333 patch_applied 0.542 completion_tokens 5989.765 test_s 1.810 tokens_per_s 35.688 wall_s 166.354	2026-10-25
llamacpp:ukisai_Swift-Qwen3.8-27b-Q4_K_M family ukisai/Swift-Qwen3.8-27b, 27B params, Q4_K_M, by bartowski source	0.292	7/24 0.29	455.85	843.80	0.0	resolved 0.292 patch_applied 0.333 completion_tokens 13070.000 test_s 2.514 tokens_per_s 39.830 wall_s 327.593	2026-10-30

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 108 cases · reply budget 32768 tokens · 4 runs, latest 2026-10-29

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.455	10/22 0.46	139.09	138.34	resolved 0.455 patch_applied 0.636 completion_tokens 5995.789 test_s 0.594 tokens_per_s 32.947 wall_s 190.069	2026-10-29
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.111	12/108 0.11	141.98	25.42	resolved 0.111 patch_applied 0.139 completion_tokens 10457.481 test_s 1.484 tokens_per_s 124.222 wall_s 90.401	2026-10-23
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.056	6/108 0.06	46.08	20.97	resolved 0.056 patch_applied 0.074 completion_tokens 3689.823 test_s 0.449 tokens_per_s 35.612 wall_s 113.850	2026-10-24
claude-code:claude-opus-5-5 reference, not adoptable	0.685	74/108 0.69	28.70	7.86	resolved 0.685 patch_applied 0.907 cache_creation_tokens 31339.846 cache_read_tokens 0.000 completion_tokens 3637.240 prompt_tokens 2.038 test_s 1.863 tokens_per_s 112.650 turns 1.019 wall_s 32.481	2026-10-23

Mac Studio M5 Ultra 96 GB · 54 cases · 1 run, 2026-10-29

candidate	resolved	pass	median s	first s	peak GB	metrics	run
repair/llamacpp:Qwen3.6-27B-Q4_K_M: llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.463	25/54 0.46	231.03	54.75	0.0	resolved 0.463 patch_applied 0.667 agent_valid_call_rate 1.000 agent_ctx 262144.000 agent_model_s 394.193 agent_no_tool_steps 0.000 agent_steps 18.444 agent_tool_s 5.204 agent_ttft_sum_s 154.285 agent_wall_s 401.546 completion_tokens 14233.630 prompt_tokens 486654.870 repair_calls_cut 1.278 repair_generated_tokens 14233.630 repair_test_runs 2.815 test_s 1.130 wall_s 401.546	2026-10-29

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 28 cases · 3 runs, latest 2026-10-29

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.607	17/28 0.61	54.20	109.39	resolved 0.607 patch_applied 0.857 completion_tokens 2654.000 test_s 0.874 tokens_per_s 35.346 wall_s 80.891	2026-10-29
repair/llamacpp:Qwen3.6-27B-Q4_K_M: llamacpp:Qwen3.6-27B-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source	0.571	16/28 0.57	130.71	38.42	resolved 0.571 patch_applied 0.821 agent_valid_call_rate 1.000 agent_ctx 262144.000 agent_model_s 334.974 agent_no_tool_steps 0.000 agent_steps 14.714 agent_tool_s 5.028 agent_ttft_sum_s 154.956 agent_wall_s 342.395 completion_tokens 11933.786 prompt_tokens 343419.286 repair_calls_cut 0.000 repair_generated_tokens 11933.786 repair_test_runs 2.500 test_s 1.624 wall_s 342.395	2026-10-25
repair/llamacpp:Ornith-1.5-35B-Q4_K_M: llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source	0.250	7/28 0.25	268.80	113.88	resolved 0.250 patch_applied 0.250 agent_valid_call_rate 1.000 agent_ctx 262144.000 agent_model_s 256.714 agent_no_tool_steps 0.000 agent_steps 19.714 agent_tool_s 6.341 agent_ttft_sum_s 130.279 agent_wall_s 264.320 completion_tokens 28117.464 prompt_tokens 559791.000 repair_calls_cut 0.000 repair_generated_tokens 28117.464 repair_test_runs 5.607 test_s 0.152 wall_s 264.320	2026-10-25

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 108 cases · reply budget 16384 tokens · 1 run, 2026-10-24

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.083	9/108 0.08	46.24	31.59	0.0	resolved 0.083 patch_applied 0.083 completion_tokens 1664.688 test_s 0.566 tokens_per_s 31.819 wall_s 59.865	2026-10-24

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 108 cases · reply budget 8000 tokens · 1 run, 2026-10-24

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.083	9/108 0.08	46.59	19.50	0.0	resolved 0.083 patch_applied 0.102 completion_tokens 805.625 test_s 0.685 tokens_per_s 34.769 wall_s 25.240	2026-10-24

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 108 cases · reply budget 4000 tokens · 1 run, 2026-10-24

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	not run (refused_by_gateway)		--	--	--		2026-10-24

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 108 cases · reply budget 2000 tokens · 1 run, 2026-10-24

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	not run (refused_by_gateway)		--	--	--		2026-10-24

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 80 cases · 2 runs, latest 2026-10-23

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Qwopus3.6-27B-v2-MTP-Q4_K_M family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by Jackrong source	0.500	3/6 0.50	78.57	104.03	resolved 0.500 patch_applied 0.667 completion_tokens 2064.250 test_s 0.779 tokens_per_s 34.275 wall_s 58.230	2026-10-23
llamacpp:Qwopus3.5-27B-v3-Q4_K_M family unsloth/Qwen3.5-27B, 27B params, Q4_K_M, by Jackrong source	0.143	2/14 0.14	477.78	88.88	resolved 0.143 patch_applied 0.500 completion_tokens 6263.857 test_s 0.865 tokens_per_s 40.029 wall_s 159.231	2026-10-23
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/6 0.00	3.11	3.03	resolved 0.000 patch_applied 0.500 completion_tokens 485.833 test_s 0.376 tokens_per_s 103.167 wall_s 5.997	2026-10-23

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 128 cases · 2 runs, latest 2026-10-23

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.023	3/128 0.02	255.38	27.48	0.0	resolved 0.023 patch_applied 0.078 completion_tokens 10075.382 test_s 0.130 tokens_per_s 104.365 wall_s 88.535	2026-10-23
claude-code:claude-opus-5-5 reference, not adoptable	0.266	34/128 0.27	30.59	12.88	0.0	resolved 0.266 patch_applied 0.945 cache_creation_tokens 31181.374 cache_read_tokens 0.000 completion_tokens 3786.968 prompt_tokens 2.098 test_s 2.081 tokens_per_s 110.666 turns 1.049 wall_s 34.476	2026-10-23

Mac Studio M5 Ultra 96 GB · 103 cases · 1 run, 2026-10-22

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.125	2/16 0.12	240.34	49.65	0.0	resolved 0.125 patch_applied 0.250 completion_tokens 16865.556 test_s 0.430 tokens_per_s 116.489 wall_s 153.185	2026-10-22

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 65536 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	5.94	2.71	0.0	resolved 0.000 patch_applied 0.205 completion_tokens 5916.744 test_s 0.479 tokens_per_s 90.272 wall_s 135.002	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 32768 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	5.60	2.98	0.0	resolved 0.000 patch_applied 0.231 completion_tokens 1669.744 test_s 0.459 tokens_per_s 87.664 wall_s 32.165	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 16384 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	6.04	3.76	0.0	resolved 0.000 patch_applied 0.231 completion_tokens 1312.744 test_s 0.307 tokens_per_s 90.721 wall_s 22.756	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 8000 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	6.35	2.75	0.0	resolved 0.000 patch_applied 0.154 completion_tokens 1358.821 test_s 0.677 tokens_per_s 87.041 wall_s 21.736	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 4000 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	6.15	3.17	0.0	resolved 0.000 patch_applied 0.256 completion_tokens 824.667 test_s 0.768 tokens_per_s 85.900 wall_s 14.950	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 2000 tokens, temperature 0.2, split all v1+tasks-995 · 1 run, 2026-10-19

candidate	resolved	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/39 0.00	6.82	3.98	0.0	resolved 0.000 patch_applied 0.256 completion_tokens 741.128 test_s 0.491 tokens_per_s 85.992 wall_s 13.679	2026-10-19

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 65536 tokens, temperature 1.0, min_p 0.0, presence_penalty 0.0, top_k 20.0, top_p 0.95, split all v1 · 1 run, 2026-10-18

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.128	5/39 0.13	86.03	33.53	0.0	resolved 0.128 patch_applied 0.333 completion_tokens 13221.343 test_s 0.611 tokens_per_s 120.020 wall_s 114.965	2026-10-18

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 39 cases · reply budget 32768 tokens, temperature 0.2, split all v1 · 2 runs, latest 2026-10-18

candidate	resolved	pass	median s	first s	peak GB	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.051	2/39 0.05	6.38	2.59	0.0	resolved 0.051 patch_applied 0.179 completion_tokens 1337.949 test_s 0.272 tokens_per_s 70.680 wall_s 17.679	2026-10-18

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 41 cases · 3 runs, latest 2026-10-17

candidate	resolved	pass	median s	first s	metrics	run
llamacpp:Ornith-1.5-35B-Q4_K_M 35B (3B active) params, Q4_K_M, by ornith-ai source serving	0.108	4/41 0.10	189.55	21.60	resolved 0.108 patch_applied 0.216 completion_tokens 8470.000 test_s 0.335 tokens_per_s 123.872 wall_s 71.743	2026-10-17
mlx-community/Qwen3-4B-Instruct-2507-4bit 4B params, 4bit, by mlx-community source	0.000	0/41 0.00	6.35	3.19	resolved 0.000 patch_applied 0.243 completion_tokens 975.297 test_s 0.497 tokens_per_s 84.519 wall_s 16.640	2026-10-17
claude-code:claude-opus-5-5 reference, not adoptable	0.324	12/41 0.29	17.54	15.78	resolved 0.324 patch_applied 0.811 completion_tokens 2126.054 test_s 1.238 tokens_per_s 96.889 wall_s 22.210	2026-10-17

Serving here: llamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai (adopted measured)

extract

Mac Studio M5 Ultra 96 GB · 450 cases · 4 runs, latest 2026-10-22

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen2.5-7B-Instruct-4bit 7B params, 4bit, by mlx-community source serving	325/450 0.72	0.13	0.13	completion_tokens 6.649 tokens_per_s 40.871	2026-10-22
neutral/mlx-community/Qwen2.5-7B-Instruct-4bit: mlx-community/Qwen2.5-7B-Instruct-4bit 7B params, 4bit, by mlx-community source	315/450 0.70	0.12	0.11	completion_tokens 6.442 method_calls 1.000	2026-10-22
plan/mlx-community/Qwen2.5-7B-Instruct-4bit: mlx-community/Qwen2.5-7B-Instruct-4bit 7B params, 4bit, by mlx-community source	284/450 0.63	0.26	0.51	completion_tokens 19.829 method_calls 2.000	2026-10-22
plan/local-large: mlx-community/Qwen2.5-7B-Instruct-4bit 7B params, 4bit, by mlx-community source	282/450 0.63	0.26	0.52	completion_tokens 19.693 method_calls 2.000	2026-10-22
deepen/mlx-community/Qwen2.5-7B-Instruct-4bit: mlx-community/Qwen2.5-7B-Instruct-4bit 7B params, 4bit, by mlx-community source	164/450 0.36	0.43	2.52	completion_tokens 76.724 method_calls 2.000	2026-10-22

This exam has no reference row to compare against.

Earlier exams (2)

Mac Studio M5 Ultra 96 GB · 152 cases · 1 run, 2026-10-08

candidate	pass	median s	first s	peak GB	metrics	run
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	113/152 0.74	0.13	6.08	0.0	completion_tokens 5.697 tokens_per_s 39.794	2026-10-08

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 10 cases · 3 runs, latest 2026-10-06

candidate	pass	median s	first s	metrics	run
mlx-community/Qwen2.5-7B-Instruct-4bit (served as local-large) 7B params, 4bit, by mlx-community source	8/10 0.80	0.14	13.59	completion_tokens 9.100 tokens_per_s 59.160	2026-10-06
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b) 30B (3B active) params, 4bit, by mlx-community source	8/10 0.80	0.16	47.77	completion_tokens 9.800 tokens_per_s 50.080	2026-10-06
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (served as q3-coder) 30B (3B active) params, 4bit, by mlx-community source	8/10 0.80	0.17	51.28	completion_tokens 9.800 tokens_per_s 50.110	2026-10-06
claude-code:claude-opus-5-5 reference, not adoptable	10/10 1.00	2.12	4.04	completion_tokens 22.800 tokens_per_s 7.020	2026-10-06

Serving here: mlx-community/Qwen2.5-7B-Instruct-4bit; 7B params, 4bit, by mlx-community

stt

Mac Studio M5 Ultra 96 GB · 480 cases · case text cd7fc8fe · 3 runs, latest 2026-11-09

candidate	wer	pass	median s	first s	metrics	run
Confucius4-R2T2	0.014	475/480 0.99	0.23	0.68	wer 0.014	2026-11-09
parakeet-tdt-0.6b-v2 0.6B params	0.015	473/480 0.98	0.09	0.45	wer 0.015	2026-11-09
nemotron-3.5-asr-streaming-0.6b-8bit 0.6B params, 8bit	0.033	464/480 0.97	0.15	1.28	wer 0.033	2026-11-03
whisper-tiny	0.092	391/480 0.81	0.09	3.57	wer 0.092	2026-11-02

This exam has no reference row to compare against.

Earlier exams (3)

Mac Studio M5 Ultra 96 GB · 480 cases · case text 11b166b2 · 4 runs, latest 2026-10-26

candidate	wer	pass	median s	first s	metrics	run
parakeet-tdt-0.6b-v2 0.6B params	0.022	395/480 0.82	0.10	0.13	wer 0.022	2026-10-26
Qwen3-ASR-0.6B-8bit 0.6B params, 8bit	0.026	381/480 0.79	0.13	0.13	wer 0.026	2026-10-19
whisper-large-v3-mlx	0.026	384/480 0.80	0.23	3.80	wer 0.026	2026-10-26

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 133 cases · 2 runs, latest 2026-10-19

candidate	wer	pass	median s	first s	peak GB	metrics	run
parakeet-tdt-0.6b-v2 0.6B params	0.020	110/133 0.83	0.10	3.95	0.0	wer 0.020	2026-10-19
whisper-large-v3-mlx	0.022	112/133 0.84	0.25	4.02	0.0	wer 0.022	2026-10-19

This exam has no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 300 cases · 1 run, 2026-10-05

candidate	wer	pass	median s	first s	peak GB	metrics	run
parakeet-tdt-0.6b-v2 0.6B params	0.016	300/300 1.00	0.07	0.14	0.0	wer 0.016	2026-10-05

This exam has one candidate and no reference row to compare against.

Serving here: mlx-community/parakeet-tdt-0.6b-v2; 0.6B params, by mlx-community

tts

Mac Studio M5 Ultra 96 GB · 470 cases · 1 run, 2026-11-09

candidate	pass	median s	first s	peak GB	metrics	run
fish-audio-s2-pro	398/465 0.86	3.33	3.92	0.0	wer 0.100	2026-11-09
Kokoro-82M-bf16/bm_george 82M params, bf16, by Kokoro-82M-bf16 source	397/465 0.85	0.16	0.33	0.0	wer 0.097	2026-11-09

This exam has no reference row to compare against.

Earlier exams (3)

Mac Studio M5 Ultra 96 GB · 460 cases · 3 runs, latest 2026-10-26

candidate	pass	median s	first s	peak GB	metrics	run
Kokoro-82M-bf16/bm_george 82M params, bf16, by Kokoro-82M-bf16 source	409/460 0.89	0.15	2.83	0.0	wer 0.084	2026-10-26

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 135 cases · 1 run, 2026-10-08

candidate	pass	median s	first s	peak GB	metrics	run
Kokoro-82M-bf16/bm_george 82M params, bf16, by Kokoro-82M-bf16 source	115/133 0.86	0.14	19.90	0.0	wer 0.082	2026-10-08

This exam has one candidate and no reference row to compare against.

Mac Studio M5 Ultra 96 GB · 10 cases · 2 runs, latest 2026-10-06

candidate	pass	median s	first s	peak GB	metrics	run
Kokoro-82M-bf16/bm_george 82M params, bf16, by Kokoro-82M-bf16 source	5/5 1.00	0.13	2.79	0.0	wer 0.000	2026-10-06
Breeze-TTS-2	5/5 1.00	9.76	16.51	0.0	wer 0.000	2026-10-06

This exam has no reference row to compare against.

Serving here: mlx-community/Kokoro-82M-bf16; family yl4579/StyleTTS2-LJSpeech, 82M params, bf16, by mlx-community