Fishtest: distributed testing framework for Stockfish
Fishtest
Definition
Fishtest is the distributed testing framework used by the open-source engine Stockfish to evaluate code changes through massive engine-vs-engine matches. Volunteers around the world contribute CPU time (“workers”) that play hundreds of thousands of games between a baseline engine and a patched candidate. Statistical methods—most notably SPRT (Sequential Probability Ratio Test)—determine whether the patch is likely to produce a real Elo improvement and should be merged into Stockfish.
How it works
- Base vs. Patch: Each test pits a known “base” build of Stockfish against a “patch” build that includes a proposed change (e.g., evaluation tweak, search heuristic, or NNUE detail). The goal is to measure the patch’s Elo impact.
- Distributed workers: Contributors run a small program that compiles the engines and plays batches of games on their hardware, reporting results back to the Fishtest server.
- Time controls and modes: Tests typically run at multiple speeds and settings:
- STC (Short Time Control) to quickly detect large effects or regressions.
- LTC (Long Time Control) to confirm subtle gains are robust.
- Fixed-nodes tests that limit computation by nodes instead of time, reducing variance from hardware differences.
- Openings and adjudication: Games begin from curated, balanced opening suites to avoid bias. Adjudication rules (e.g., draw detection, tablebase adjudication) end clearly decided positions early to speed testing.
- Statistics: Fishtest uses SPRT with predefined hypotheses (e.g., “the patch is 0 Elo vs. it is at least a small positive Elo”). A running log-likelihood ratio (LLR) determines whether to accept (merge) or reject the patch with high confidence. Reported numbers include Elo estimate and error bars.
- Tooling: A match manager (commonly cutechess-cli or an equivalent tool) runs the games, while Fishtest coordinates compilation, pairing, color balance, and data aggregation.
Usage in chess
Among engine developers and advanced users, “It passed Fishtest” means a change demonstrated a statistically significant improvement and was accepted into Stockfish’s main code. Content creators and strong players may cite Fishtest results when discussing why the engine evaluates positions differently over time or why a new release is stronger by a few Elo.
Strategic and historical significance
Fishtest has been central to Stockfish’s rise as one of the strongest chess engines. By enabling continuous, rigorous A/B testing, it rewards small, reliable improvements and filters out regressions. This process accelerated progress during major milestones, including the integration of NNUE evaluation (2020), where many subtle code and network updates were verified through large-scale testing. The framework exemplifies how open, community-driven computation can push state-of-the-art engine strength.
Examples
- Typical test lifecycle:
- A contributor proposes a patch that slightly changes king safety evaluation.
- Fishtest runs an STC test. Preliminary results show +1.3 Elo with tight error bars; the SPRT LLR crosses the accept boundary.
- The same patch runs at LTC. After many more games, it still shows a small, consistent gain; SPRT accepts again.
- The patch is merged into Stockfish master.
- Regression caught: A search tweak appears neutral at STC, but at LTC it loses ~0.7 Elo and SPRT rejects it. The change is not merged, saving users from a subtle strength decrease.
- Micro-gains add up: Over months, numerous tiny merges—tuning piece-square tables, pruning thresholds, or move ordering—each worth a fraction of an Elo, accumulate into a noticeable strength jump measurable in real play and engine tournaments.
Interesting facts
- Name and origin: The name “Fishtest” reflects its role testing “Stockfish.” The system was created and is maintained by Stockfish contributors (the project began in the early 2010s) and is open-source, just like the engine.
- Scale: Over the years, the community has contributed the equivalent of many CPU-years of testing, producing hundreds of millions of engine-vs-engine games to vet improvements.
- Color-coding culture: In community discussions, a “green” test means accepted (positive result); “red” means rejected.
- Beyond code: While primarily used for code patches, Fishtest has also been used to evaluate nets and parameter tunings in the NNUE era.
Tips for contributors
- Run workers on stable hardware; disable turbo-boost variability if possible to reduce noise.
- Start with STC tests; only promote to LTC if the idea shows promise.
- Document your changes clearly so other developers can interpret results and replicate.
- Be patient: SPRT may need many games, especially for small expected Elo gains.
Related terms
Anecdote
In the years after the famous AlphaZero vs. Stockfish experiments (2017), the Stockfish team kept merging numerous tiny, Fishtest-validated improvements. When NNUE arrived, Fishtest helped confirm that the hybrid (alpha-beta + neural eval) approach was not only competitive but superior on commodity CPUs—one meticulously tested patch at a time.