Lane report

What each lane serves on each machine, the receipt behind it, and the latest run's full comparison.

Machines

machine last publishedageschemaexport
Mac Studio M5 Ultra 96 GB2026-11-12T04:30:40Z0.0d984current

Each machine publishes its own measurements. One that has not published in 7 days is marked stale, and its numbers are as old as its publish date.

Lanes across machines

laneMac Studio M5 Ultra 96 GB
imagemflux:dev
0.99 pass, 28.91s, 2026-11-12
codellamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai
0.79 pass, 3.04s, 2026-11-12
webmlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b); 4B params, 4bit, by mlx-community
0.64 pass, 2.94s, 2026-11-12
svgmlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b); 30B (3B active) params, 4bit, by mlx-community
0.72 pass, 4.53s, 2026-11-11
videoh3
1.00 pass, 171.53s, 2026-11-09
musicacestep:acestep-v15-turbo
1.00 pass, 28.42s, 2026-11-02 stale
decideimajev-4b-Q8_0; 4B params, Q8_0, by mindchain not run (harness_error)
agentllamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai
0.95 pass, 9.17s, 2026-11-05 stale
ocrhf-ocr:PaddlePaddle/PaddleOCR-VL-1.6
0.83 pass, 7.64s, 2026-11-11
retrievalbm25
0.78 pass, 0.02s, 2026-11-10
piipii-regex
0.74 pass, 0.03s, 2026-11-09
claimsQwen2.5-7B-Instruct-Q4_K_M; 7B params, Q4_K_M, by bartowski
claims_balanced 0.593, pass 192/299 0.64, 0.14s, 2026-11-02 stale
repollamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai
resolved 0.100, pass 1/10 0.10, 201.81s, 2026-11-07
extractmlx-community/Qwen2.5-7B-Instruct-4bit; 7B params, 4bit, by mlx-community
0.72 pass, 0.13s, 2026-10-22 stale
sttmlx-community/parakeet-tdt-0.6b-v2; 0.6B params, by mlx-community
wer 0.015, pass 473/480 0.98, 0.09s, 2026-11-09
ttsmlx-community/Kokoro-82M-bf16; family yl4579/StyleTTS2-LJSpeech, 82M params, bf16, by mlx-community
0.85 pass, 0.16s, 2026-11-09

Wall-clock is only comparable within one machine. Across machines compare pass rates and what each lane serves.

Mac Studio M5 Ultra 96 GB

macOS 27.2 · arm64 · cpu, ds4, llamacpp, mlx · published 2026-11-12T04:30:40Z · schema 98

laneservespassmedian s TTFT med / p95metricmeasured
imagemflux:dev0.9928.91--staging 0.9882026-11-12adopted by-hand
codellamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai0.793.042.58s / 18.35sco_tenant 1.000
code_pass 0.833
completion_tokens 983.375
tokens_per_s 128.725		2026-11-12adopted measured
webmlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b); 4B params, 4bit, by mlx-community0.642.94--completion_tokens 610.774
ink 0.084
method_calls 1.810
method_chosen 1.041
method_samples 3.000		2026-11-12adopted measured
svgmlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b); 30B (3B active) params, 4bit, by mlx-community0.724.53--completion_tokens 567.452
ink 0.385
method_calls 2.000
svg_bytes 654.382		2026-11-11adopted measured
videoh31.00171.53--motion 0.0382026-11-09
musicacestep:acestep-v15-turbo1.0028.42--wer 0.339
duration_error_s 0.000
seconds 30.000		2026-11-02stale
decideimajev-4b-Q8_0; 4B params, Q8_0, by mindchaindecide_accuracy --, all fields right ------2026-11-10not run (harness_error)
agentllamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-ai0.959.170.30s / 0.42sagent_ctx 262144.000
agent_model_s 11.044
agent_no_tool_steps 0.000
agent_steps 8.265
agent_tool_s 0.151
agent_ttft_sum_s 5.444
agent_valid_call_rate 1.000
agent_wall_s 11.259
completion_tokens 1435.979
prompt_tokens 14245.134		2026-11-05adopted measured stale
ocrhf-ocr:PaddlePaddle/PaddleOCR-VL-1.60.837.64--cer 0.6462026-11-11adopted measured
retrievalbm250.780.02--retrieval_ndcg 0.880
retrieval_recall 0.783		2026-11-10
piipii-regex0.740.03--pii_f1 0.796
pii_recall 0.662
pii_precision 1.000		2026-11-09
claimsQwen2.5-7B-Instruct-Q4_K_M; 7B params, Q4_K_M, by bartowskiclaims_balanced 0.593, pass 192/299 0.640.140.08s / 0.20sclaims_balanced 0.593
claims_case_recall 0.521
claims_case_specificity 0.665
claims_empty_rate 0.668
claims_schema_valid 1.000
claims_bad_matched_cited_only 0.091
claims_bad_matched_conversation 0.000
claims_emitted 0.875
claims_judge_calibrated 0.000
claims_matcher_version 2.000
claims_near_miss_cited_only 0.182
claims_near_miss_conversation 0.067
claims_pass_cited_only 0.318
claims_pass_conversation 0.800
claims_precision_cited_only 0.821
claims_precision_conversation 0.922
claims_recall_cited_only 0.287
claims_recall_conversation 0.657
claims_unreviewed_cited_only 1.727
claims_unreviewed_conversation 1.467
claims_verbatim_cited_only 0.273
claims_verbatim_conversation 0.978
completion_tokens 27.260
tokens_per_s 65.195		2026-11-02stale
repollamacpp:Ornith-1.5-35B-Q4_K_M; 35B (3B active) params, Q4_K_M, by ornith-airesolved 0.100, pass 1/10 0.10201.81116.81s / 116.81sresolved 0.100
patch_applied 0.100
completion_tokens 11054.000
test_s 122.386
tokens_per_s 86.700
wall_s 250.780		2026-11-07adopted measured
extractmlx-community/Qwen2.5-7B-Instruct-4bit; 7B params, 4bit, by mlx-community0.720.130.10s / 0.14scompletion_tokens 6.649
tokens_per_s 40.871		2026-10-22stale
sttmlx-community/parakeet-tdt-0.6b-v2; 0.6B params, by mlx-communitywer 0.015, pass 473/480 0.980.09--wer 0.0152026-11-09
ttsmlx-community/Kokoro-82M-bf16; family yl4579/StyleTTS2-LJSpeech, 82M params, bf16, by mlx-community0.850.16--wer 0.0972026-11-09

image latest run 2026-11-12

candidatepasspassmedian sTTFT med / p95peak GBmetric
mflux/dev-q880/810.9928.91--20.6staging 0.988

code latest run 2026-11-12

candidatepasspassmedian sTTFT med / p95peak GBmetric
llamacpp:Ornith-1.5-35B-Q4_K_M
35B (3B active) params, Q4_K_M, by ornith-ai source
19/240.793.042.58s / 18.35s0.0co_tenant 1.000
code_pass 0.833
completion_tokens 983.375
tokens_per_s 128.725
llamacpp:Qwen3.6-35B-A3B-UD-Q4_K_M
family Qwen/Qwen3.6-35B-A3B, 35B (3B active) params, Q4_K_M, by unsloth source
18/230.7832.2828.08s / 54.67s0.0code_pass 0.864
completion_tokens 3796.727
tokens_per_s 130.673

web latest run 2026-11-12

candidatepasspassmedian sTTFT med / p95peak GBmetric
best-of-3/q3-4b: mlx-community/Qwen3-4B-Instruct-2507-4bit
4B params, 4bit, by mlx-community source
124/1950.642.94--0.0completion_tokens 610.774
ink 0.084
method_calls 1.810
method_chosen 1.041
method_samples 3.000
mlx-community/Qwen3-4B-Instruct-2507-4bit
4B params, 4bit, by mlx-community source
119/1940.612.210.13s / 0.16s0.0completion_tokens 330.299
ink 0.075
tokens_per_s 117.028

svg latest run 2026-11-11

candidatepasspassmedian sTTFT med / p95peak GBmetric
plan/q3-30b: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit
30B (3B active) params, 4bit, by mlx-community source
454/6280.724.53--0.0completion_tokens 567.452
ink 0.385
method_calls 2.000
svg_bytes 654.382
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (served as q3-30b)
30B (3B active) params, 4bit, by mlx-community source
434/6280.692.450.15s / 0.18s0.0completion_tokens 328.292
ink 0.326
svg_bytes 591.624
tokens_per_s 113.984

video latest run 2026-11-09

candidatepasspassmedian sTTFT med / p95peak GBmetric
diffusers-video/Motif-Video-2B
2B params
2/21.0050.76--21.5motion 0.083
h3/minimax-h32/21.00171.53--3.5motion 0.038

music latest run 2026-11-02

candidatepasspassmedian sTTFT med / p95peak GBmetric
acestep/acestep-v15-turbo4/41.0028.42--46.0wer 0.339
duration_error_s 0.000
seconds 30.000

decide latest run 2026-11-12

candidatedecide_accuracyall fields rightpassmedian sTTFT med / p95peak GBmetric
llamacpp:imajev-4b-Q8_0
4B params, Q8_0, by mindchain source
0.823241/5290.460.46--0.0decide_accuracy 0.823
decide_brier 0.258
decide_ece 0.030
calibrated 1.000
co_tenant 1.000
completion_tokens 36.461
tokens_per_s 77.903
bespokelabs/Bespoke-Nimble-9B
family Qwen/Qwen3.5-9B, 9B params, by bespokelabs source
not run (harness_error)------

agent latest run 2026-11-05

candidatepasspassmedian sTTFT med / p95peak GBmetric
llamacpp:Ornith-1.5-35B-Q4_K_M
35B (3B active) params, Q4_K_M, by ornith-ai source
356/3740.959.170.30s / 0.42s0.0agent_ctx 262144.000
agent_model_s 11.044
agent_no_tool_steps 0.000
agent_steps 8.265
agent_tool_s 0.151
agent_ttft_sum_s 5.444
agent_valid_call_rate 1.000
agent_wall_s 11.259
completion_tokens 1435.979
prompt_tokens 14245.134
mlx-community/Qwen3-4B-Instruct-2507-4bit (served as q3-4b)
4B params, 4bit, by mlx-community source
52/3740.142.220.18s / 0.25s0.0agent_model_s 4.514
agent_no_tool_steps 0.024
agent_steps 5.660
agent_tool_s 0.028
agent_ttft_sum_s 1.500
agent_valid_call_rate 1.000
agent_wall_s 4.583
completion_tokens 667.869
prompt_tokens 6416.305

ocr latest run 2026-11-11

candidatepasspassmedian sTTFT med / p95peak GBmetric
hf-ocr/PaddleOCR-VL-1.6371/4480.837.64--0.0cer 0.646
osocr/auto323/4480.720.30--0.2cer 0.170

retrieval latest run 2026-11-10

candidatepasspassmedian sTTFT med / p95peak GBmetric
bm25/okapi148/1890.780.02--0.0retrieval_ndcg 0.880
retrieval_recall 0.783

pii latest run 2026-11-09

candidatepasspassmedian sTTFT med / p95peak GBmetric
hf-pii/privacy-filter462/5560.833.92--4.0pii_f1 0.890
pii_recall 0.871
pii_precision 0.910
pii-regex/patterns411/5560.740.03--0.0pii_f1 0.796
pii_recall 0.662
pii_precision 1.000

claims latest run 2026-11-02

candidateclaims_balancedpasspassmedian sTTFT med / p95peak GBmetric
Qwen2.5-7B-Instruct-Q4_K_M
7B params, Q4_K_M, by bartowski source
0.593192/2990.640.140.08s / 0.20s0.0claims_balanced 0.593
claims_case_recall 0.521
claims_case_specificity 0.665
claims_empty_rate 0.668
claims_schema_valid 1.000
claims_bad_matched_cited_only 0.091
claims_bad_matched_conversation 0.000
claims_emitted 0.875
claims_judge_calibrated 0.000
claims_matcher_version 2.000
claims_near_miss_cited_only 0.182
claims_near_miss_conversation 0.067
claims_pass_cited_only 0.318
claims_pass_conversation 0.800
claims_precision_cited_only 0.821
claims_precision_conversation 0.922
claims_recall_cited_only 0.287
claims_recall_conversation 0.657
claims_unreviewed_cited_only 1.727
claims_unreviewed_conversation 1.467
claims_verbatim_cited_only 0.273
claims_verbatim_conversation 0.978
completion_tokens 27.260
tokens_per_s 65.195

repo latest run 2026-11-07

candidateresolvedpasspassmedian sTTFT med / p95peak GBmetric
llamacpp:Qwen3.6-27B-Q4_K_M
family Qwen/Qwen3.6-27B, 27B params, Q4_K_M, by unsloth source
0.2001/50.20592.50280.09s / 524.58s0.0resolved 0.200
patch_applied 0.400
completion_tokens 13413.000
test_s 36.203
tokens_per_s 33.967
wall_s 424.803
llamacpp:Ornith-1.5-35B-Q4_K_M
35B (3B active) params, Q4_K_M, by ornith-ai source
0.1001/100.10201.81116.81s / 116.81s0.0resolved 0.100
patch_applied 0.100
completion_tokens 11054.000
test_s 122.386
tokens_per_s 86.700
wall_s 250.780

extract latest run 2026-10-22

candidatepasspassmedian sTTFT med / p95peak GBmetric
mlx-community/Qwen2.5-7B-Instruct-4bit
7B params, 4bit, by mlx-community source
325/4500.720.130.10s / 0.14s0.0completion_tokens 6.649
tokens_per_s 40.871
plan/local-large: mlx-community/Qwen2.5-7B-Instruct-4bit
7B params, 4bit, by mlx-community source
282/4500.630.26--0.0completion_tokens 19.693
method_calls 2.000

stt latest run 2026-11-09

candidatewerpasspassmedian sTTFT med / p95peak GBmetric
Confucius4-R2T20.014475/4800.990.23--0.0wer 0.014
parakeet-tdt-0.6b-v2
0.6B params
0.015473/4800.980.09--0.0wer 0.015

tts latest run 2026-11-09

candidatepasspassmedian sTTFT med / p95peak GBmetric
fish-audio-s2-pro398/4650.863.33--0.0wer 0.100
Kokoro-82M-bf16/bm_george
82M params, bf16, by Kokoro-82M-bf16 source
397/4650.850.16--0.0wer 0.097

Real use last 7 days through the gateway

aliasrequeststokens in / outTTFT p50 / p95errorsbad tool calls
sohot-agent17289 / 34--0.0%--
sohot-claims57325124 / 55387--0.3%--
sohot-code48714832 / 89310.35s / 0.52s0.8%--
sohot-decide951119357 / 109440.04s / 0.04s0.0%--
sohot-extract1191494683 / 486140.59s / 0.70s0.0%--

Adoptions

laneadoptedreplacedhowwhen
codellamacpp:Ornith-1.5-35B-Q4_K_M--measured2026-10-06
imagemflux:devmflux:flux2-klein-4bby-hand2026-10-06
svgq3-30blocal-largemeasured2026-10-06
agentllamacpp:Ornith-1.5-35B-Q4_K_Mq3-4bmeasured2026-10-07
repollamacpp:Ornith-1.5-35B-Q4_K_Mmlx-community/Qwen3-4B-Instruct-2507-4bitmeasured2026-10-24
ocrhf-ocr:PaddlePaddle/PaddleOCR-VL-1.6osocr:automeasured2026-11-07
svgplan:q3-30bq3-30bmeasured2026-11-10
webbest-of:3:q3-4bmlx-community/Qwen3-4B-Instruct-2507-4bitmeasured2026-11-11