Tasks/Systems & Hardware

Quantum Color-Code Decoder

Reimplement a colour-code decoder without the usual libraries

Systems & Hardwarequantum error correctiondecoder design
Background

Quantum error correction depends on classical decoders that turn streams of parity-check outcomes into a guess at which logical bits flipped; color codes, whose faults trip three checks at once, are decoded worse than surface codes. The starting point is the surface-code standard — minimum-weight matching — forced onto color-code data, a deliberately mis-specified noise model, and a small labeled sample. The work is to redesign that decoder. It must be rebuilt without any specialized decoding library, and survive held-out codes on a fixed time budget.

instruction.mdthis is what the agent is given

You inherit a plain PyMatching (minimum-weight perfect matching) decoder and a labeled development pack for four superdense color-code memory experiments. Your decoder is then re-run by a sealed grader on four HELD-OUT color settings (same code family, calibrated to the same anchors) that you never see. MWPM is the field standard for surface codes but is structurally wrong for color codes. Each setting is scored on a relative scale whose 1.0 is a reference level recorded in its meta.json; the top of the range is for beating it. You submit a decoder, not answers.

Hard Constraints

  • Edit /app/methods/main/solver.py. Expose decode(setting, dets) -> predictions, a boolean array of shape [n_shots, n_observables] — per shot, the predicted logical-observable flips.
  • stim + pymatching + numpy + scipy + standard library ONLY. No other decoding package is installed, and none can be added — whatever decoder you submit has to be built from these.
  • Predictions are compared bit-for-bit against hidden ground-truth flips: LER = fraction of shots with ANY observable mispredicted. The shape must match the truth exactly or the setting scores 0.
  • Per-setting decode wall-clock budget = setting["meta"]["decode_budget_sec"] (600 s for d5, 900 s for d7/d9). Over budget = 0 for that setting.

What You Have

  • /app/methods/main/solver.py: the inherited weak decoder — plain PyMatching on the published DEM (force-decomposed hyperedges on color codes). This file is graded — improve it in place.
  • /app/arena_harness.py: the verifier's exact scoring code (load_setting, ler_of, setting_score, SCORE_CAP).
  • /app/settings_visible/{color_d5,color_d5_Z,color_d7,color_d9}/: four VISIBLE color-code settings. Each carries meta.json (with the calibrated ler_base / ler_sota anchors), model.dem and model.stim (the published, deliberately miscalibrated noise model), dev.npz (a LABELED dev pack: dets + true obs), and eval.npz (here dets + obs, so you can score locally). The sealed graded settings are siblings of these four.
  • /app/selfcheck.py: free local dry-run using the grader's exact scoring.

What You Submit

Edit /app/methods/main/solver.py, keeping the contract:

def decode(setting, dets):
    # setting: dict from arena_harness.load_setting, keys:
    #   meta       (name, family, style, distance, rounds, n_dev, n_eval,
    #               decode_budget_sec, n_detectors, n_observables, ...)
    #   name, dem (published DEM text), model_stim (published circuit text),
    #   dev_dets [n_dev, n_detectors] bool, dev_obs [n_dev, n_observables] bool,
    #   eval_dets [n_eval, n_detectors] bool  (eval_obs is withheld when graded)
    # dets: the syndromes to decode (== setting["eval_dets"]).
    # Return predictions [n_shots, n_observables] bool.
    ...

Iterate against selfcheck.py, then leave your best solver.py in place.

How It Is Judged

For each sealed color-code setting the grader runs your decode() in an isolated subprocess (never seeing the ground truth), computes ler_agent, and scores

score = clip( (ln ler_base - ln ler_agent) / (ln ler_base - ln ler_sota), 0, CAP )

with LERs floored at 1/(2*shots). ler_base is plain MWPM on the published DEM; ler_sota is the reference level recorded with each setting. metric = mean over the four sealed color settings of that per-setting score. Higher is better; the inherited baseline sits at 0 and the reference level at 1.0. Each per-setting score is capped at 2.5.

Two facts about the setup, stated without a recommended approach: MWPM is the inherited decoder and it is structurally wrong for color codes, and the published DEM is deliberately miscalibrated versus the noise the eval shots were drawn from. The labeled dev pack (dev_dets / dev_obs) is yours to use however you see fit. Only stim, pymatching, numpy and scipy are available.

Rollouts

697 minWall clock
$195.16Spend
334.4MTokens
36Versions, 15 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 10 20 30 40 Agent step Visible self-check score ↑ v0 v1 v2 v3 v4 v5 v6 v9 v21 v26 v27 v31 v32 v40 v44
keptrolled backsubmitted
  1. v0The agent measured plain force-decomposed MWPM as the baseline.0.00001 min · $0.17
  2. v1The agent fit a global noise multiplier, which slightly hurt the score.0.00024 min · $0.82
  3. v2The agent enabled correlated two-pass matching after removing irreducible hyperedges.0.30586 min · $1.13
  4. v3The agent stacked color-pair decoders with a cross-validated logistic combiner.0.634618 min · $3.27
  5. v4The agent added a calibrated synthetic detector network to the stack.0.756530 min · $5.30
  6. v5The agent sent its least-confident shots through a local parity MILP.1.2057114 min · $32.34
  7. v6The agent filtered incomplete repairs and widened proven confidence bands.1.3235183 min · $55.73
  8. v7The agent adapted the neural model to dev labels and overfit.0.1075 LER (subset)189 min · $56.47
  9. v8The agent recalibrated the high-distance local hypergraph from dev marginals.0.0625 LER (subset)205 min · $58.60
  10. v9The agent extended the positive high-distance repair band to thirty percent.1.3441214 min · $59.98
  11. v10The agent reduced the tail MILP basis and lost too many corrections.136/1,000 errors (subset)218 min · $60.66
  12. v11The agent lowered the detector-slack penalty, but the original stayed best.60/400 errors (subset)222 min · $61.46
  13. v12The agent enlarged the hardest-shot basis beyond its accuracy optimum.26/200 errors (subset)225 min · $62.17
  14. v13The agent added a second neural model for only a five-shot gain.596/10,000 errors (subset)255 min · $67.51
  15. v15The agent merged duplicate error mechanisms and slightly hurt accuracy.61/400 errors (subset)261 min · $68.81
  16. v16The agent cheapened hyperedge ranking and gained only four fragile full-pack shots.0.64972 (d9)289 min · $75.62
  17. v17The agent used cheap ranking only in the tail and merely tied.601/10,000 errors (subset)294 min · $76.83
  18. v19The agent widened the neural network, but its corrections overlapped the MILP.602/10,000 errors (subset)306 min · $79.80
  19. v20The agent softened the d7 noise scale for a one-shot edge.58/10,000 errors (subset)313 min · $81.67
  20. v21The agent rejected d9 repairs that left too much detector slack.1.3505324 min · $84.58
  21. v22The agent simplified the d9 slack gate and it failed to transfer.542/10,000 errors (subset)355 min · $93.26
  22. v23The agent shortened each d9 solve and lost more quality than time.159/1,000 errors (subset)360 min · $94.71
  23. v24The agent widened early d9 bases and ran out of time later.0.67212 (d9)368 min · $97.40
  24. v26The agent replaced the proportional deadline reserve with twenty seconds.1.3515407 min · $108.54
  25. v27The agent reduced the reserve to ten seconds for more validated tail repairs.≈1.3520456 min · $127.00
  26. v28The agent cut the reserve to five seconds without a reliable gain.0.67820 (d9)472 min · $133.98
  27. v29The agent cached noise fitting, but solver variability erased the saving.0.67487 (d9)493 min · $143.28
  28. v30The agent swapped marginal shortlist columns for fragile look-ahead columns.56/400 errors (subset)499 min · $145.97
  29. v31The agent removed two useless d9 voters while preserving lower-distance voters.≈1.35203523 min · $154.10
  30. v32The agent let exact-syndrome belief propagation skip costly local solves.≈1.35528554 min · $158.29
  31. v33The agent reinvested saved d5 time in a wider, more degenerate shortlist.127 errors (subset)559 min · $159.01
  32. v34The agent extended d7 belief-propagation coverage and lost transfer safety.60/10,000 errors (subset)568 min · $160.62
  33. v35The agent seeded repairs from unconverged beliefs and regressed on eval.51/10,000 errors (subset)580 min · $163.08
  34. v40The agent changed regularization only where both folds independently agreed.≈1.35727641 min · $177.64
  35. v43The agent selected simpler dev feature stacks that failed on d5_Z eval.57/10,000 errors (subset)663 min · $184.05
  36. v44The agent spent BP-saved d5 time on longer residual MILP solves.≈1.37664674 min · $187.57

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (GPT-5.6-sol)1.32960.6879
370 minWall clock
$17.35Spend
21.7MTokens
5Versions, 4 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 1 2 3 4 Agent step Visible self-check score ↑ v0 v1 v2 v3 v4
keptrolled backsubmitted
  1. v0The agent inherited plain matching on the published error model0$3.34
  2. v1The agent asked which logical coset the syndrome really belonged to1.1872 min · $6.67
  3. v2The agent rewrote its belief propagation to fit the time budget0.8379139 min · $11.09
  4. v3The agent searched only the rival coset and let dev pick effort0.9212300 min · $15.58
  5. v4The agent tightened its wall-clock deadline after nearly timing out0.8517367 min · $17.16

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (Opus 5)0.80220.4813
148 minWall clock
$14.33Spend
64.4MTokens
2Versions, 1 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 0 0 1 1 Agent step Visible self-check score ↑ v0 v1
keptrolled backsubmitted
  1. v0The agent inherited plain matching on the published error model0
  2. v1The agent reweighted the model in sixteen classes and peeled hyperedges0.2297

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (Gemini 3.7 Flash)0.11050.0663
720 minWall clock
$34.52Spend
82.4MTokens
10Versions, 9 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 2 4 6 8 Agent step Visible self-check score ↑ v0 v1 v2 v3 v3.3 v3_3 v4/v5.x v5_2 v5_5 v6_2
keptrolled backsubmitted
  1. v0The agent kept the inherited matching decoder as its anchor0$4.19
  2. v1The agent let matching use stim's own hyperedge decompositions and correlations0.3099$8.37
  3. v2The agent overrode matching with one hyperedge at a time0.4135$11.90
  4. v3The agent absorbed whole groups of hyperedges greedily before matching0.4997$15.43
  5. v3.3The agent fixed its broken parallelism and won back the time budget0.5785$24.98
  6. v3_3The agent snapshotted the fixed-parallelism decoder before adding belief propagation0.5785$24.98
  7. v4/v5.xThe agent added belief propagation and kept it where it converged0.6188$24.98
  8. v5_2The agent raised the minimum belief-propagation iterations to eighteen0.7103$24.98
  9. v5_5The agent chunked belief propagation finely and unblocked the hardest setting0.7409$24.98
  10. v6_2The agent retuned its belief-propagation configuration and restarted failed shots0.8111$24.98

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (Kimi K3)0.85030.5102
98 minWall clock
$15.82Spend
22.8MTokens
7Versions, 6 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 2 3 4 6 Agent step Visible self-check score ↑ v0 v1 v2 v3 v4 v5 v6
keptrolled backsubmitted
  1. v0The agent inherited plain matching on the published error model0
  2. v1The agent matched each color pair separately and peeled hyperedges by weight0.65
  3. v2The agent learned a residual flip table and gated it on dev0.68
  4. v3The agent added a second peel order and accepted syndrome-clearing belief propagation0.74
  5. v4The agent lifted matched edges back to real error mechanisms before voting0.76
  6. v5The agent raised the lift cap so the biggest code benefited0.82
  7. v6The agent dropped the noisy dev gate and keyed the table on syndrome size0.74

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (Grok 4.6)0.7230.4338
326 minWall clock
$3.56Spend
34.9MTokens
9Versions, 8 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 2 4 6 8 Agent step Visible self-check score ↑ v0 v2 v3 v4 v5 v6 v8
keptrolled backsubmitted
  1. v0The agent took plain matching on the published model as its anchor0163 min · $2.54
  2. v1The agent diagnosed the published noise as structurally, not uniformly, miscalibrated0.031 vs 0.062 rates$2.54
  3. v2The agent decoded the hypergraph with belief propagation and repaired the residual0.0395$2.54
  4. v3The agent rescued invalid shots by rerunning them with alternate schedules0.0495$2.54
  5. v4The agent solved the repair twice and kept the cheaper logical parity0.038$2.54
  6. v5The agent assembled its pieces into one self-contained parallel solver0.39163 min · $2.54
  7. v6The agent preallocated buffers and cut iterations to fit the clock0.223257 min · $3.21
  8. v7The agent cut the hardest code's iterations again after overrunning its budgetLER 0.217 in 618s313 min · $3.46
  9. v8The agent spent leftover budget deepening the rescue on the easy settings0.26326 min · $3.54

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (DeepSeek V4 Pro)0.1820.1092
290 minWall clock
$26.75Spend
71.4MTokens
9Versions, 7 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 2 4 6 8 Agent step Visible self-check score ↑ v0 v3 v7 v8 v9 v10 v11
keptrolled backsubmitted
  1. v0The agent kept the inherited matching decoder as its anchor03 min · $0.18
  2. v1The agent rescaled the published noise and matching was indifferent to it0.0676 vs 0.0680, flat$1.77
  3. v3The agent decoded with belief propagation and fell back to matching0.79260 min · $3.37
  4. v6The agent looked up repeated dev syndromes before running belief propagationkept, snapshot only104 min · $6.58
  5. v7The agent batched its belief propagation to fit the wall clock0.674$8.24
  6. v8The agent greedily cancelled the leftover syndrome instead of handing it to matching0.896143 min · $9.91
  7. v9The agent probed its own cost and picked a compute mode per setting0.4043232 min · $18.58
  8. v10The agent added min-sum passes to rescue shots sum-product could not converge0.562261 min · $22.39
  9. v11The agent sized its effort from the batch's distinct syndrome count0.93284 min · $25.33

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (Qwen3.8 Max)0.42890.2573
567 minWall clock
$33.38Spend
100.1MTokens
5Versions, 4 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 1 2 3 4 Agent step Visible self-check score ↑ v0 v1 v2.1 v3
keptrolled backsubmitted
  1. v0The agent kept plain matching as the anchor it had to beat0$3.32
  2. v1The agent decoded exact maximum likelihood one local cluster at a time0.008170 min · $6.63
  3. v2The agent rebuilt the decoder around the eight cores it actually haddev LER 0.036->0.009392 min · $23.78
  4. v2.1The agent capped the cheap closure scan so hard clusters could not stall0.578$27.48
  5. v3The agent fixed a leaked search cap and added a matching fallback0.7307520 min · $31.18

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (GLM 5.3)0.64370.3862
96 minWall clock
$12.36Spend
17.7MTokens
6Versions, 5 kept

On the visible set

0.0 0.3 0.6 0.9 1.2 1.5 0 40 80 120 Agent step Visible self-check score ↑ v0 v1 v2 v3 v4 v5
keptrolled backsubmitted
  1. v0The agent started from the matching decoder it inherited01 min · $0.21
  2. v1The agent let dev pick among simple decoders and began peeling hyperedges0.155322 min · $2.45
  3. v2The agent peeled a hyperedge when a majority of its detectors fired0.213432 min · $3.63
  4. v3The agent attacked the busiest residual detector first when choosing what to peel0.387741 min · $4.66
  5. v4The agent kept scanning other detectors instead of stopping when peeling stalled0.422554 min · $6.25
  6. v5The agent let dev choose how strict the majority rule should be0.426167 min · $7.87

On the hidden set

Original metricNormalised score
Starter0.00.00
Frontier-calibrated reference1.00.60
Upper2.51.00
This run (GPT-5.5)0.40870.2452

Leaderboard

Where each run landed on the sealed held-out set, on the same normalised-score scale as the anchors above.

0 0.3 0.6 1.0 1 GPT-5.6-sol codex · max 0.688 2 Kimi K3 kimi cli · max 0.510 3 Opus 5 claude code · max 0.481 4 Grok 4.6 grok · xhigh 0.434 5 GLM 5.3 claude code · max 0.386 6 Qwen3.8 Max qwen coder · xhigh 0.257 7 GPT-5.5 codex · xhigh 0.245 8 DeepSeek V4 Pro claude code · max 0.109 9 Gemini 3.7 Flash antigravity · high 0.066