SoHoT
So hot right now.
It keeps swapping in better local models for each job, and only on evidence.
SoHoT runs this loop on its own, on a timer. Each step is cheaper than the next and only passes on what survived, and every winner becomes the model the next challenger has to beat. Tap or click a box to read the code behind it.
Models on your own machine, doing 16 jobs. It makes images, video, music, speech, SVG and web pages. It writes code, edits a repository through tools as an agent and fixes real bugs in real repositories. And it does the reading: transcribing speech, reading the text in an image, pulling answers out of logs and listings, extracting the claims a text makes, finding personal data, ranking the passages that answer a query and checking which questions a passage answers.
Like the ship of Theseus, every part of this harness gets replaced over time while the harness stays itself. The parts are the models serving each lane. SoHoT goes looking for challengers on its own, measures each one against the model it would replace on identical cases on your hardware, and swaps it in only on evidence. Nothing here was adopted because it was popular.
No cloud account, no provider API key, no per-token bill, and no model retired out from under you. The one key is the gateway's own, which SoHoT makes on first use and keeps on the machine that serves.
Each tier costs more than the last and only sees what survived the one before, so measurement is spent where it counts.
- sweep
Reads the model registries, the places practitioners talk, and what the people who build your tools star on GitHub. - inspect
Reads each candidate's source or model card, without downloading weights or running anything, and drops what cannot run on this machine. - fetch
Downloads what inspect queued, inside its own disk budget: 80 GiB a night unless you set another. - screen
One minimal run that answers one question: did it run and emit anything. - measure
The lane's cases, paired against the model it would replace. When dev and held-out cases will not both fit the time budget, it runs the held-out ones. - adopt
Replaces the incumbent on a significant paired win on held-out cases, or by hand on at least 10 votes from people comparing the two.
A challenger and the incumbent run the same held-out cases. The harness works out how many repeats the paired test needs to see a real gain, and a null result with too little power is recorded as underpowered, not as no better. Cases are split by content digest so nothing tunes itself on the cases that decide.
A lane is one job with its own cases and checks, and each serves its own winner:
- image
- code
- web
- svg
- video
- music
- decide
- agent
- ocr
- retrieval
- pii
- claims
- repo
- extract
- stt
- tts
In the teal lanes (music, svg) no metric can pick a winner, so a person votes.
An OpenAI-compatible gateway serves one alias per text lane: sohot-code, sohot-web, sohot-svg, sohot-extract, sohot-decide, sohot-agent, sohot-claims. An adoption re-points the alias to the winner without anyone editing a config. The gateway restarts to take it once it has been idle for 60 seconds; after 20 minutes it stops taking new requests and lets the ones in flight finish first. Claude Code can use it through the Anthropic messages route.
Every machine keeps its own receipts and publishes its own slice, keyed by a hardware label and never a hostname. Paths are cut, and a privacy scan refuses any export that still names a person or a place. A machine that has not published in 7 days is marked stale.
Every missed defect is traced to the logical failing behind it, which becomes a generic class with a test shape. Later code is tested against every class that has bitten, and CI fails new code that trips a detector until it has a test of its own or a waiver that says why.
Every proposal, sighting and verdict goes into a small database, so a candidate already answered is not offered again. A rejection reopens when the thing that failed it changes, and a screen or measure loss is tried again after 7 days, up to 3 times.
An MCP server, scripts/serve-mcp.sh, gives an agent these tools: prompt, svg, web, code, local_complete, local_decide, delegated, lane_feed, image, video, job_status, job_result. Image and video go on a work queue that runs one GPU job at a time, started when no client is being served. Each lane's state, from what it serves to every candidate's verdict, is one document: soh lane-feed <lane>, or the MCP resource soh://lanes/{lane}.
Install the command, soh, into its own environment:
git clone https://github.com/unxmaal/SoHoT
cd SoHoT
uv tool install --python 3.12 --editable .On Apple Silicon, install the services as launchd agents, which also schedule the nightly loop:
./scripts/launchd.sh install # every service, kept up across rebootsor start them by hand:
./scripts/serve-mlx.sh & # MLX text models
./scripts/serve-eval.sh & # llama.cpp: GGUF models, and decide and claims
./scripts/serve-tts.sh & # speech in and out
./scripts/serve-gateway.sh & # the address clients use, port 4000On a machine with an NVIDIA card, one command starts all three:
./scripts/services.sh start # gateway, text, audioThen make something, and look for something better:
soh svg "a settings gear icon"
soh discover --sweep # read every source family
soh discover --queue # what a screen would teach, best first
soh report # a page of where each lane stands here
soh lane-feed code # one lane's served model and verdictsPoint Claude Code at the gateway. The key comes from the machine that serves:
export SOHOT_GATEWAY_KEY="$(soh gateway key)"
ANTHROPIC_BASE_URL=http://<host>:4000 ANTHROPIC_AUTH_TOKEN="$SOHOT_GATEWAY_KEY" \
claude --model sohot-codeEverything else is in the README.