Tasks/Physical Sciences & Engineering

Distribution Feeder Model Calibration from Smart-Meter Data

Jointly infer customer phases, line impedances, and regulator taps on unseen feeders

Physical Sciences & Engineeringpower systemsmodel calibration
Background

Distribution system model calibration — checking a utility's records against the network as built — decides how much rooftop solar a line can accept, yet phase labels, conductor impedances and regulator taps drift from the paperwork. The starting point is a first-pass script grouping meter voltage traces into three phases and reading the tap off a power-flow replay. It is to be rebuilt as a joint inversion of phases, impedances and tap. The unknowns outnumber the magnitude-only readings, and the method is re-run unchanged on unseen feeders.

instruction.mdthis is what the agent is given

You are an AI agent assisting a distribution planning engineer at a mid-size utility with a solar interconnection queue. Before any hosting-capacity or voltage-drop study can be signed off, the feeder model has to be calibrated: the GIS says which phase each customer is on, which conductor each segment uses, and where the regulator tap sits — and all three are wrong at well-documented rates, because crews re-phase laterals after storms, re-conductor spans without closing the paperwork, and move taps in the field. All you have per feeder is two to three days of 15-minute AMI data from most (not all) customers — voltage magnitude only — plus head SCADA and billing averages. A model calibrated too optimistically overstates hosting capacity and customers get voltage complaints; too pessimistically and interconnections are refused for no reason. You inherit the utility's current first-pass script; your job is to build the calibration method the utility will run fleet-wide — it is re-run, unchanged, on feeders whose actual wiring you never see.

Hard Constraints

  • Submit a method: methods/main/solver.py defining exactly calibrate(record) -> dict with keys "phase" (list of one int in 0/1/2 per customer), "code" (list of one conductor class per segment, from the record's catalog), "scale" (list of one positive impedance multiplier per segment), and "tap_steps" (number, in regulator steps). Optionally add "z_rx": a list of [R1_ohm, X1_ohm] per segment, which then overrides code/scale in the impedance part of the score. A pure function of one feeder record.
  • record is a dict with the feeder's network records and its AMI monitoring window — the same schema as the practice records, minus the truth field. Field-by-field documentation: data/practice/DATA_CARD.md.
  • Each call runs in a fresh process under a 300-second wall-clock budget (measured outside your process, with a 15 s grace and a hard kill at 330 s); over-budget, crashed or malformed outputs are scored as the worst case for that feeder.
  • Evaluation budget as a whole: the evaluation runs your method once per assessment feeder, serially, inside a 10800-second wall-clock cap for the entire scoring pass, on 2 CPUs and 512 MB of memory. The assessment fleet is 16 feeders — 1.33× the 12 practice feeders you have — and its members are on average larger (more customers and more segments) than the practice ones. Budget your per-feeder cost accordingly: 16 feeders that each burn the full 300 s do fit, but nothing beyond that does.
  • CPU only, no network. Runtime: Python 3 with numpy, scipy and the OpenDSS engine (import opendssdirect); you may run as many forward power-flow solves as the budget allows. Your own container has 2 CPUs / 1024 MB. /dev/shm is only 64 MiB on both sides, so a multiprocessing / joblib design that passes large arrays through POSIX shared memory will create the segment successfully and then die with SIGBUS on the first write — size any such buffer accordingly. BLAS/OpenMP thread counts are pinned to the same constant on both sides, so a timing you measure with selfcheck.py transfers to the evaluation.
  • Only files under methods/ are collected and re-run: keep everything calibrate() imports inside methods/main/. Do not modify data/, selfcheck.py or run_solver.py.
  • Return plain python types (ints/floats/lists); numpy arrays are fine for z_rx rows, but every value must be finite and code entries must come from the record's catalog. A malformed return is scored as the worst case for that feeder.

What You Have

  • data/practice/instances.json — 12 practice feeders with complete as-operated truth (actual per-customer phases, per-segment conductor classes and impedance multipliers, actual tap), documented in data/practice/DATA_CARD.md. This is your only labeled data; study it in full.
  • data/assessment/records.json — the 16 assessment feeders your method will actually be judged on: same monitoring protocol, no truth. They are unseen feeders drawn from the same population as the practice fleet (a subset weighted toward its hard end: the largest networks and longest segments the population produces).
  • methods/main/solver.py — the inherited starting method: voltage- trace similarity grouping for metered phases, records-as-shipped impedances, and a tap estimate from the average voltage offset of an OpenDSS replay (methods/main/dss_forward.py is its forward engine; python3 methods/main/dss_forward.py runs a round-trip demo). It carries a real signal but is far from what the records support; its level is also the floor you must clearly beat before the evaluation awards any credit.
  • python3 selfcheck.py — free and unlimited: scores your current methods/main/solver.py on the practice fleet against its truth and prints per-feeder component scores and the mean.

What You Submit

Leave your best methods/main/solver.py (plus any helper files it needs inside methods/main/) in place. There is no submit step and no feedback from the assessment fleet: whatever sits in methods/main/ at the end is what the evaluation re-runs.

How It Is Judged

The evaluation re-runs your calibrate() once per assessment feeder, on records byte-identical to data/assessment/records.json, and compares your output to the sealed as-operated truth. Per feeder it computes a score in [0, 1] (HIGHER is better): 0.45 × phase score (label accuracy over metered customers, rescaled so that copying the shipped GIS labels scores 0 and perfect labeling scores 1) + 0.40 × impedance score (how much your cumulative substation-to-customer positive-sequence R and X — the quantity a voltage-drop study consumes — improves on the records as shipped, measured by log-error norm and clipped to [0, 1]) + 0.15 × tap score (1 − |tap step error| / 4, clipped). Returning the records as shipped scores exactly 0 — selfcheck.py computes the identical per-feeder score on practice. Feeder scores are averaged within each assessment group (f00f07 and f08f15), then across the two groups. Your reward rises monotonically with that sealed mean; at or below the shipped starting method's level it is zero.

Rollouts

166 minWall clock
$33.47Spend
47.2MTokens
19Versions, 18 kept

On the visible set

0.4 0.5 0.6 0.7 0.8 0.9 0 8 15 22 30 Agent step Full practice mean over 12 feede ↑ v0 v1 v2 v3 v4 v7 v8 v9 v10 v11 v12 v13 v16 v17 v26 v27 v29
keptrolled backsubmitted
  1. v0The agent started from spectral phases, recorded impedances, and a replay-based tap.full practice mean: 0.43301 min · $0.22
  2. v1The agent replaced spectral clustering with robust leave-one-out phase prototypes.full practice mean: 0.50807 min · $0.94
  3. v2The agent jointly inverted unmetered phases, line impedances, and regulator tap.full practice mean: 0.624124 min · $3.07
  4. v3The agent ensembled trace and physics phase candidates using full-window replay.full practice mean: 0.742940 min · $5.48
  5. v4The agent screened sparse conductor substitutions with conservative impedance shrinkage.full practice mean: 0.778757 min · $8.94
  6. v7The agent ranked conductors out-of-time and blended two impedance estimates.full practice mean: 0.789666 min · $10.70
  7. v8The agent snapped regulator taps to valid integer steps.projected full practice mean: 0.790570 min · $11.51
  8. v9The agent broke replay ties toward candidates correcting more metered phases.full practice mean: 0.838980 min · $13.69
  9. v10The agent removed unmetered compensation when evaluating metered-only hybrids.projected full practice mean: 0.853782 min · $14.35
  10. v11The agent trusted the out-of-time conductor estimate without averaging.full practice mean: 0.863188 min · $15.75
  11. v12The agent capped near-tie phase edits and guarded spectral initialization.full practice mean: 0.8631101 min · $19.66
  12. v13The agent relaxed the phase cap when no bounded candidate replayed closely.full practice mean unchanged: 0.8631105 min · $20.91
  13. v16The agent expanded conductor screening across the entire catalog.full practice mean unchanged: 0.8631106 min · $21.63
  14. v17The agent allowed a heavily penalized third conductor correction.full practice mean unchanged: 0.8631130 min · $25.55
  15. v25The agent added a weak feeder-average impedance-multiplier prior.full impedance score: 0.6577132 min · $25.99
  16. v26The agent combined feeder-average and local priors to break conductor ties.full practice mean: 0.8659139 min · $27.38
  17. v27The agent fixed a loop that accumulated too many conductor changes.full practice mean unchanged: 0.8659151 min · $30.07
  18. v28The agent tested up to five conductor changes, causing implausible compensation.full practice mean unchanged: 0.8659154 min · $30.64
  19. v29The agent added one unmetered-phase candidate for low-coverage feeders.full practice mean: 0.8659166 min · $33.47

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (GPT-5.6-sol)0.6913640.4102
148 minWall clock
$30.95Spend
44.1MTokens
7Versions, 6 kept

On the visible set

0.450 0.525 0.600 0.675 0.750 0.825 0 40 80 120 160 Agent step Mean practice score over 12 feed ↑ v0 v1 v2 v3 v4 v5 v6
keptrolled backsubmitted
  1. v0The agent inherited a trace-clustering baseline with records-as-shipped impedances0.4334 min · $0.82
  2. v1The agent rewrote the solver around its own vectorised power-flow sweep0.692645 min · $11.15
  3. v2The agent ran profiled coordinate descent over all load phase labels0.752349 min · $11.85
  4. v3The agent added a weak GIS-label prior and corrected unmetered profiles0.783972 min · $18.56
  5. v4The agent moved coordinate descent inside the outer alternation loop0.807182 min · $21.07
  6. v5The agent added multi-label perturb-and-relax moves to the phase search0.7862107 min · $24.65
  7. v6The agent deleted dead code and submitted the unchanged pipeline0.7862144 min · $30.24

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (Opus 5)0.7536130.5771
70 minWall clock
$6.36Spend
23.7MTokens
2Versions, 1 kept

On the visible set

0.425 0.450 0.475 0.500 0.525 0.550 0 0 0 1 1 Agent step Mean practice score over 12 feed ↑ v0 v1
keptrolled backsubmitted
  1. v0The agent inherited a trace-clustering baseline with records-as-shipped impedances0.433
  2. v1The agent ensembled two spectral clusterings and inverted a global impedance scale0.5517

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (Gemini 3.7 Flash)0.5532330.0398
358 minWall clock
$55.09Spend
154.7MTokens
20Versions, 19 kept

On the visible set

0.44 0.48 0.52 0.56 0.60 0.64 0 4 8 12 16 Agent step Mean practice score over 12 feed ↑ v1 v2 v3 v4 v5 v6 v6 (joint tap/scale) v7 v7/v8 v8 v9 v10 v11 v12 v13 v13b v14 v16 v16 (snapshot) baseline
keptrolled backsubmitted
  1. v1The agent replaced clustering with shape-ICM phases and Gauss-Newton segment scales0.5233$4.91
  2. v2The agent switched the phase search to plain coordinate descent0.5149$6.87
  3. v3The agent took the better of batch-ICM and coordinate descent by shape-SSE0.5227$8.82
  4. v4The agent capped phase deviations and rounded the tap to integers0.5263$10.78
  5. v5The agent aligned phase candidates to GIS to break rotation symmetry0.5545$11.76
  6. v6The agent estimated the impedance scale from the drop-current slope0.608$14.15
  7. v6 (joint tap/scale)The agent iterated the tap and the impedance scale jointly0.6081$15.86
  8. v7The agent merged the joint tap-scale iteration into the shipped solver0.6081$17.57
  9. v7/v8The agent cleaned up the solver and added a robustness fallback0.6081$19.28
  10. v8The agent stripped the remaining dead code from the solver0.6081$20.99
  11. v9The agent disabled the deviation cap as redundant0.6081$22.70
  12. v10The agent added per-segment slope scales clipped toward the feeder mean0.6199$24.66
  13. v11The agent loosened the clipping threshold to three sigma0.6203$25.68
  14. v12The agent weighted per-segment estimation by impedance-magnitude share0.6217$26.72
  15. v13The agent damped per-segment deviation in the reactance path sums0.6255$38.16
  16. v13bThe agent raised the reactance damping coefficient to 0.150.6256$41.09
  17. v14The agent removed unused dead code0.6256$44.02
  18. v16The agent voted phases across ICM, coordinate descent and spectral clustering0.6326$53.22
  19. v16 (snapshot)The agent re-verified the consensus phase method on the assessment records0.6326$54.15
  20. baselineThe agent inherited a trace-clustering baseline with records-as-shipped impedances0.433$54.15

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (Kimi K3)0.6100560.1922
46 minWall clock
$13.18Spend
20.6MTokens
18Versions, 7 kept

On the visible set

0.44 0.48 0.52 0.56 0.60 0 4 8 12 16 Agent step Mean practice score over 12 feed ↑ v0 v1 v2 v3 v4 v5 v6 v6b v6b (rejected) v7
keptrolled backsubmitted
  1. v0The agent inherited a spectral-phase baseline with recorded impedances0.433
  2. v1The agent tried greedy catalog search with physics phase reassignment0.47
  3. v1bThe agent swept conductor codes by per-segment RMSE under true phasesalready at noise floor
  4. v2The agent restored safe phases and added ridge path-drop inversion0.483
  5. v2bThe agent tried unconstrained least squares for R and Xcollinear P/Q, imp=0
  6. v3The agent matched voltages phase-invariantly to fit the global impedance scale0.505
  7. v3bThe agent tried reassigning every meter across all three phasescascade of phase flips
  8. v4The agent added per-segment scales shrunk toward the global scale0.54
  9. v4bThe agent weakened the ridge shrinkage on the segment scalesp01/p11 collapsed
  10. v5The agent fell back to GIS phases when clustering disagreed0.572
  11. v5bThe agent picked phase labels by whose fitted scale neared onedestroyed p01
  12. v5cThe agent searched trunk-only code changes across both day halvesp04 +0.26, net down
  13. v5dThe agent merged meters whose voltages were nearly identicalp00 up, p03/p04 down
  14. v6The agent applied outlier segment scales fully to R, halfway to X0.583
  15. v6bThe agent clipped the returned impedances at a small floor0.583
  16. v6b (rejected)The agent tried a dual spectral embedding chosen by GIS agreement0.57
  17. v6cThe agent added a second three-phase confirmation pass after impedance fittingp04 0.33 -> 0
  18. v7The agent rescaled unmetered load profiles to track the head SCADA0.584

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (Grok 4.6)0.5670740.0769
78 minWall clock
$2.01Spend
13.7MTokens
2Versions, 1 kept

On the visible set

0.44 0.46 0.48 0.50 0.52 0 0 0 1 1 Agent step Mean practice score over 12 feed ↑ v0 v1
keptrolled backsubmitted
  1. v0The agent inherited a trace-clustering baseline with records-as-shipped impedances0.433$0.99
  2. v1The agent decided disputed phases by OpenDSS exclusion and ridge-fitted per-segment impedances0.522777 min · $1.98

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (DeepSeek V4 Pro)0.564880.0710
114 minWall clock
$13.58Spend
44.4MTokens
2Versions, 1 kept

On the visible set

0.425 0.450 0.475 0.500 0.525 0 40 80 120 160 Agent step Mean practice score over 12 feed ↑ v1 v2
keptrolled backsubmitted
  1. v1The agent inherited a trace-clustering baseline with records-as-shipped impedances0.43364 min · $5.95
  2. v2The agent rebuilt the physics model and shipped one global impedance multiplier0.5373112 min · $13.05

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (Qwen3.8 Max)0.5405190.0057
480 minWall clock
$20.05Spend
69.3MTokens
5Versions, 4 kept

On the visible set

0.450 0.525 0.600 0.675 0 1 2 3 4 Agent step Mean practice score over 12 feed ↑ v0 v1 v2 v3 v4
keptrolled backsubmitted
  1. v0The agent inherited a trace-clustering baseline with records-as-shipped impedances0.43390 min · $8.46
  2. v1The agent rewrote the solver with a vectorized sweep and MAP priors0.6429166 min · $10.97
  3. v2The agent beam-searched the unmetered load phases0.6694346 min · $15.85
  4. v3The agent tracked every configuration reached and returned the best MAP0.6706423 min · $17.84
  5. v4The agent corrected selection bias in the MAP record-change penalties0.7147423 min · $17.84

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (GLM 5.3)0.6828070.3872
34 minWall clock
$7.34Spend
8.7MTokens
7Versions, 6 kept

On the visible set

0.45 0.50 0.55 0.60 0.65 0 20 40 60 80 Agent step Mean practice score over 12 feed ↑ v0 v1 v2 v3 v4 v5 v6
keptrolled backsubmitted
  1. v0The agent inherited a trace-clustering baseline with records-as-shipped impedances0.4331 min · $0.17
  2. v1The agent relabeled phases from a 5D spectral embedding with a margin gate0.542412 min · $1.83
  3. v2The agent rounded the tap and fitted regularized impedance scales0.618318 min · $2.73
  4. v3The agent chose among spectral embeddings by OpenDSS residual0.633725 min · $4.47
  5. v4The agent tried every one-segment catalog change and kept the best0.643526 min · $4.96
  6. v5The agent gated code changes by the fitted scale direction0.66428 min · $5.54
  7. v6The agent kept the default phase labels unless an alternative clearly improved0.66731 min · $6.46

On the hidden set

Original metricNormalised score
Starter0.5383910.0
Frontier-calibrated reference0.7621590.6
Upper1.01.0
This run (GPT-5.5)0.602070.1707

Leaderboard

Where each run landed on the sealed held-out set, on the same normalised-score scale as the anchors above.

0 0.3 0.6 1.0 1 Opus 5 claude code · max 0.577 2 GPT-5.6-sol codex · max 0.410 3 GLM 5.3 claude code · max 0.387 4 Kimi K3 kimi cli · max 0.192 5 GPT-5.5 codex · xhigh 0.171 6 Grok 4.6 grok · xhigh 0.077 7 DeepSeek V4 Pro claude code · max 0.071 8 Gemini 3.7 Flash antigravity · high 0.040 9 Qwen3.8 Max qwen coder · xhigh 0.006