Adyan Ahmad Quazi
  1. Home
  2. Work
  3. ci

Project ci · five shipped projects, re-run on fresh pulls · GitHub Actions

Reproducibility CI

Every figure five of the projects quote, 919 in all, is re-run on a fresh pull of its source every Monday and on every push. Same bytes must give the same number to the precision the memo prints it, or the run fails. The first run found a notebook that could not run on a clean checkout, 42 uncertainty figures the committed notebook no longer printed, and eight sentences the tables did not support as written.

Scope
Projects 01, 02, 03, 04 and 06; 05 excluded by policy while its backfill is three gigabytes
Scale
919 figures across five manifests; 9 sources fingerprinted; 8 drift policies
In the repository
ci · findings memo · ci-reports branch
Status as recorded
Badge on 2026-09-09: 904 of 919 reproduced, 6 drifted inside band, green. The Monday schedule runs from there.

Status as recorded, one mark per figure

Every figure the CI re-runs, one mark eachUnit chart, one square per published figure, in five columns by project, filled from the baseline up. 01: 56 of 56 reproduced exactly. 02: 22 of 22. 03: 334 of 337, 3 declared. 04: 215 of 220, 5 declared. 06: 277 of 284, 6 drifted inside band and 1 declared. In all 919 figures on the badge of 2026-09-09: 904 reproduced exactly, 6 drifted inside band, 9 declared, none failed.01: 56 of 56 reproduced exactly02: 22 of 22 reproduced exactly03: 334 of 337 reproduced exactly, 3 declared04: 215 of 220 reproduced exactly, 5 declared06: 277 of 284 reproduced exactly, 6 drifted inside band, 1 declared0156 of 560222 of 2203334 of 33704215 of 22006277 of 284
Every figure the CI re-runs, one mark eachUnit chart, one square per published figure, in five columns by project, filled from the baseline up. 01: 56 of 56 reproduced exactly. 02: 22 of 22. 03: 334 of 337, 3 declared. 04: 215 of 220, 5 declared. 06: 277 of 284, 6 drifted inside band and 1 declared. In all 919 figures on the badge of 2026-09-09: 904 reproduced exactly, 6 drifted inside band, 9 declared, none failed.01: 56 of 56 reproduced exactly02: 22 of 22 reproduced exactly03: 334 of 337 reproduced exactly, 3 declared04: 215 of 220 reproduced exactly, 5 declared06: 277 of 284 reproduced exactly, 6 drifted inside band, 1 declared0156of 560222of 2203334of 33704215of 22006277of 284
  • reproduced exactly
  • drift inside band
  • declared
  • failed: 0
One square per published figure, a column per project, filled from the baseline up: reproduced first, then drift inside band, then declared. The badge of 2026-09-09: 904 of 919 reproduced exactly, 6 drifted inside band after the IESO rewrote its two 2026 files, 9 declared with a reason, the five design constants of 04 among them.ci · badge of 2026-09-09
Data behind this chart
Figures per project on the badge of 2026-09-09
ProjectListedReproducedDrift in bandDeclared
01 Seller risk565600
02 Funnel222200
03 Defence33733403
04 Wind22021505
06 Demand28427761
Total91990469
01

The question

Several hundred figures were each printed by a notebook cell on the day a memo was written, and nothing in the repo said whether they would come back the same on a new machine, or after the publisher refreshed the file. Do the numbers the READMEs and memos quote come back on a clean checkout and a fresh pull, and when a source changes, how far may each kind of figure move before the memo needs a dated update?

02

What it is

One YAML manifest per project lists every quoted figure with the sentence as printed, the documents that quote it, its kind, the query that reads it back from the tables the project’s own notebook builds, or a regular expression over the notebook’s printed output where the figure never reaches a table, the committed value, and the sources it depends on.

A runner fetches the two Kaggle datasets when they are absent, executes each notebook headless, cell by cell, in its own folder so it re-pulls and rebuilds everything, fingerprints each source, re-measures every figure read-only against the rebuilt DuckDB file, and classifies under two regimes. Identical bytes must give an identical figure at the memo’s precision, or the run fails. Changed bytes are publisher drift, read under a per-source contract of bands by kind. Figures that cannot be checked are declared with a reason and still measured, never dropped.

A GitHub Actions workflow has a plan job that turns the manifest folder into a matrix, one reproduce job per project, and a publish job that writes dated reports, a summary and a badge to an orphan ci-reports branch, so main stays hand-committed.

03

What the first run found

  1. 1The wind-repowering notebook could not run top to bottom on a clean checkout. An exploratory cell opened two downloads that the publisher now answers with a landing page; a one-cell guard fixed it and no number changed.
  2. 2The same memo’s 42 uncertainty statements were not what the committed notebook printed, while every point estimate held. Two clean runs agreed with each other on all 42; the placebo p-value read 0.167 in the memo and 0.268 on a clean run.
  3. 3Eight memo sentences the tables did not support as written, among them a count read from a three-row display where the table holds 35 such hours, a two that is a three, and two interval bounds off by a point.
  4. 4Five demand-forecast figures rounded twice from a one-decimal table. They stay as printed, because the manifest checks the table value behind each.

Each was recorded in its manifest entry and in the findings memo. The first three were closed on 2026-09-09: the repowering notebook run by hand and its 42 figures re-measured, the eight sentences corrected, and the manifests updated to 919 figures with 9 declared.

04

Finding

Fresh-pull run of 2026-09-07, figure by figure

Fresh-pull run of 2026-09-07, figure by figureStacked horizontal bars, figures per project on the fresh-pull run of 7 September 2026. 01: 56 of 56 reproduced. 02: 22 of 22. 03: 334 reproduced and 3 declared, of 337. 04: 173 reproduced and 47 declared, of 220. 06: 275 reproduced, 6 moved inside their drift band after the publisher rewrote its 2026 files, and 1 declared, of 282. In total 917 listed, 860 reproduced, 6 drifted, 51 declared, 0 failed. After the corrections of 9 September: 919 listed, 9 declared.010020030001 Seller risk: 56 reproduced, 0 drift inside band, 0 declared, of 5601 Seller risk56 of 5602 Funnel: 22 reproduced, 0 drift inside band, 0 declared, of 2202 Funnel22 of 2203 Defence: 334 reproduced, 0 drift inside band, 3 declared, of 33703 Defence334 of 33704 Wind: 173 reproduced, 0 drift inside band, 47 declared, of 22004 Wind173 of 22006 Demand: 275 reproduced, 6 drift inside band, 1 declared, of 28206 Demand281 of 282reproduced exactlydrift inside banddeclaredfailed: 0
Fresh-pull run of 2026-09-07, figure by figureStacked horizontal bars, figures per project on the fresh-pull run of 7 September 2026. 01: 56 of 56 reproduced. 02: 22 of 22. 03: 334 reproduced and 3 declared, of 337. 04: 173 reproduced and 47 declared, of 220. 06: 275 reproduced, 6 moved inside their drift band after the publisher rewrote its 2026 files, and 1 declared, of 282. In total 917 listed, 860 reproduced, 6 drifted, 51 declared, 0 failed. After the corrections of 9 September: 919 listed, 9 declared.010020030001 Seller risk: 56 reproduced, 0 drift inside band, 0 declared, of 5601 Seller risk56 of 5602 Funnel: 22 reproduced, 0 drift inside band, 0 declared, of 2202 Funnel22 of 2203 Defence: 334 reproduced, 0 drift inside band, 3 declared, of 33703 Defence334 of 33704 Wind: 173 reproduced, 0 drift inside band, 47 declared, of 22004 Wind173 of 22006 Demand: 275 reproduced, 6 drift inside band, 1 declared, of 28206 Demand281 of 282reproduced exactlydrift inside banddeclaredfailed: 0
The committed reports are the afternoon run, after the IESO rewrote its 2026 files. That morning, on identical bytes, 866 of 917 came back exactly.ci · run 2026-09-07
Data behind this chart
Fresh-pull run of 2026-09-07, figures per project
ProjectListedReproducedDrift in bandDeclared
01 Seller risk565600
02 Funnel222200
03 Defence33733403
04 Wind220173047
06 Demand28227561
Total917860651
  1. 917 figures, five projects.56, 22, 337, 220 and 282 figures in the five manifests, every one tied to the query that reads it back.
  2. Nothing failed.All five sources came back byte-identical to the memos’ pulls, so every figure was held to an exact match: 866 of 917 reproduced, 51 declared, 9 of them design constants and notebook properties and 42 bootstrap-derived.
  3. Drift, seen once on real bytes.That afternoon the IESO rewrote its two 2026 files with one more day. Six whole-series figures moved inside their bands; the 248 scored figures that read the same file held exactly.
  4. 919 after the corrections.With the 42 re-measured and the sentences fixed, the manifests list 919 figures and declare 9.

The workflow’s first run on GitHub, on 9 Sep 2026, completed green in 14 minutes and created the reports branch. With the two Kaggle secrets stored, a dispatched run reproduced every project and the badge read 860 of 917, 6 drifted, green. The push that afternoon, with the 42 figures re-measured and the manifests at 919, took the badge to 904 of 919, 6 drifted. Notebook run times on the committed baseline: 7.6 s, 15.5 s, 7.8 s, 424 s and 84 s.

05

What this cannot tell you

  • Whether a figure is right. It tests agreement between memo and notebook, and that the agreement survives a re-pull.
  • Whether a change in a figure matters. The bands are one person’s judgement, written down.
  • How most sources behave when they drift, since the drift regime is untested until a source actually changes.
  • Whether a stopped notebook is a real failure: a transient network timeout that survives two attempts reads as the notebook stopping.
  • Anything about project 05, which is excluded.
06

Artifacts

  • The runner, reproduce.py: one file, the standard library plus DuckDB, PyYAML, nbformat and nbclient
  • Five manifests with committed fingerprint files
  • The drift contract as policy.yml, with its reasons in a data contract memo covering nine sources
  • A decisions memo of twenty-one decisions, each with what would reverse it, and a findings memo
  • The dated reports, the workflow, and the root README badge and paragraph
07

Sources

  1. The five project sources as fingerprinted: 14 Kaggle files by size, 1 CanadaBuys hash, 1 USGS API hash, 1 archive hash, 13 EIA-923 hashes and 49 IESO hashes.
  2. The first fresh-pull run, 2026-09-07, and the first GitHub run, 2026-09-09, with dated reports on the ci-reports branch.
  3. The findings, decisions and data contract memos in ci/memo.