Column 3: verify.py
Open data, followed all the way down to one person inside it.
Every piece here starts with a public dataset and ends with a story. The numbers can be checked: any figure set in mono opens its own evidence in a column to the right, so you never have to leave the sentence you are reading to find out whether it is true.
open to work
work
files
Every published figure, re-derived from the raw source by something that isn’t the pipeline.
A pipeline that emits its own numbers cannot establish that they are right — it can only be self-consistent. So there is a notebook that reads the raw Kaggle files and the committed data.json, recomputes each claim from scratch, and asserts the two agree.
It deliberately does not import build.py. The pipelines are stdlib; the notebook is pandas and numpy. Where two implementations agree, the number is not an artefact of either one.
It also checks the places where the site admits a weakness, because an admission is a claim too. The scale-dependent residual in piece 01 is recomputed both ways — rank 1 under the shipped scaling, 17 under least squares — and the withdrawn flatness inference in piece 02 is verified as withdrawn: sleep’s profile really is at least as uniform as screen time’s.
The whole notebook is below — all 20 cells, with the output each one actually produced when make_audit.py ran it. Nothing here is transcribed by hand, and no line of it is a screenshot.
Auditing kodata
Every number published on kodata is re-derived here from the raw source, independently of the pipelines that produced it.
Why this exists. The site claims that its figures are traceable. A pipeline that emits its own numbers cannot establish that on its own — it can only be self-consistent. So this notebook reads the raw Kaggle files and the committed `data.json`, recomputes each published claim from scratch, and asserts the two agree.
Why pandas. The pipelines are stdlib-only. This notebook is pandas and numpy. Where they agree, the result is not an artefact of one implementation.
Nothing here imports build.py.
import json, math
from pathlib import Path
import numpy as np
import pandas as pd
import kagglehub
ROOT = Path.cwd()
while not (ROOT / "web" / "content").exists() and ROOT != ROOT.parent:
ROOT = ROOT.parent
# what the site publishes
P1 = json.loads((ROOT / "web/content/01-ai-exposure/data.json").read_text(encoding="utf-8"))
P2 = json.loads((ROOT / "web/content/02-screen-time/data.json").read_text(encoding="utf-8"))
# the raw source, fetched the same way the pipelines fetch it
b1 = kagglehub.dataset_download("kylefengkfeng209/will-ai-take-my-job-exposure-skills-and-wages")
b2 = kagglehub.dataset_download("kylefengkfeng209/screen-time-vs-mental-health-ml-ready")
jobs = pd.read_csv(f"{b1}/ai_job_exposure.csv")
prof = pd.read_csv(f"{b1}/occupation_cognitive_profile.csv")
abil = pd.read_csv(f"{b1}/cognitive_ability_ai_exposure.csv")
teens = pd.read_csv(f"{b2}/screen_time_mental_health.csv")
items = pd.read_csv(f"{b2}/bdi_and_screen_items.csv")
CHECKS = []
def check(label, got, want, tol=5e-4):
ok = (abs(got - want) <= tol) if isinstance(want, (int, float)) else (got == want)
CHECKS.append((label, got, want, ok))
print(f"{'PASS' if ok else 'FAIL'} {label}")
print(f" recomputed {got!r} published {want!r}")
assert ok, label
print(f"pandas {pd.__version__}, numpy {np.__version__}")
print(f"jobs {jobs.shape} profiles {prof.shape} abilities {abil.shape}")
print(f"teenagers {teens.shape} items {items.shape}")pandas 3.0.5, numpy 2.5.1 jobs (271, 23) profiles (271, 23) abilities (21, 3) teenagers (4810, 10) items (4810, 29)
--- ## Piece 01 — What Makes Writing Writing
Claim 1. The dataset scores AI exposure twice, and the two disagree
Exposure is scored per cognitive ability and per occupation. Build the per-occupation prediction from the abilities each job demands, then correlate it against the rating the dataset publishes for that job.
ability_cols = [c for c in prof.columns if c not in ("soc_code", "occupation_title")]
W = abil.set_index("cognitive_ability")["ai_exposure_score"]
assert set(ability_cols) == set(W.index), "ability names differ between files"
lv = prof.set_index("soc_code")[ability_cols].dropna()
# weighted mean of ability exposure, weighted by how strongly the job needs each
pred = (lv * W[ability_cols]).sum(axis=1) / lv.sum(axis=1)
d = jobs.set_index("soc_code").join(pred.rename("pred"), how="inner").dropna(subset=["pred"])
r_layers = d["pred"].corr(d["ai_exposure_llm_human"])
print(f"n = {len(d)}")
check("piece 01 · the two layers correlate", round(float(r_layers), 4), P1["robustness"]["layers_r"])
check("piece 01 · shared variance", round(float(r_layers) ** 2, 3), P1["findings"]["layers_r2"])
# Fisher z interval, independently
z, se = math.atanh(r_layers), 1 / math.sqrt(len(d) - 3)
lo, hi = math.tanh(z - 1.96 * se), math.tanh(z + 1.96 * se)
check("piece 01 · CI low", round(lo, 4), P1["robustness"]["layers_r_ci"][0])
check("piece 01 · CI high", round(hi, 4), P1["robustness"]["layers_r_ci"][1])n = 271
PASS piece 01 · the two layers correlate
recomputed -0.367 published -0.367
PASS piece 01 · shared variance
recomputed 0.135 published 0.135
PASS piece 01 · CI low
recomputed -0.4658 published -0.4658
PASS piece 01 · CI high
recomputed -0.2591 published -0.2591
Claim 2. Writing moves further between the two orderings than any other job
This is the claim the piece leads with, because it needs no scaling: it compares two rank orders and never touches a unit.
d = d.assign(
pr=d["pred"].rank(ascending=False, method="first").astype(int),
ar=d["ai_exposure_llm_human"].rank(ascending=False, method="first").astype(int),
)
d["move"] = d["pr"] - d["ar"]
subj = d[d["occupation_title"] == "Writers and authors"].iloc[0]
check("piece 01 · predicted rank", int(subj["pr"]), P1["subject"]["rank_predicted"])
check("piece 01 · rated rank", int(subj["ar"]), P1["subject"]["rank_actual"])
check("piece 01 · displacement", int(subj["move"]), P1["robustness"]["subject"]["move"])
check("piece 01 · largest displacement of all",
int(d["move"].rank(ascending=False, method="first")[subj.name]),
P1["robustness"]["subject"]["move_rank"])
print()
print(d.nlargest(5, "move")[["occupation_title", "pr", "ar", "move"]].to_string(index=False))PASS piece 01 · predicted rank
recomputed 267 published 267
PASS piece 01 · rated rank
recomputed 4 published 4
PASS piece 01 · displacement
recomputed 263 published 263
PASS piece 01 · largest displacement of all
recomputed 1 published 1
occupation_title pr ar move
Writers and authors 267 4 263
Public relations specialists 258 3 255
Advertising sales agents 252 9 243
Operations research analysts 257 19 238
Urban and regional planners 265 29 236
Claim 3. The residual ranking is not robust — and the site says so
The piece publishes an audit of its own weakest claim: under the shipped rescaling writing is the largest residual of 271, but under ordinary least squares it is seventeenth. Both are recomputed here, because a site that admits a weakness should let you verify the admission.
pm, ps = d["pred"].mean(), d["pred"].std(ddof=0)
am, asd = d["ai_exposure_llm_human"].mean(), d["ai_exposure_llm_human"].std(ddof=0)
# shipped: standardise the prediction onto the rating's mean and spread
fit_shipped = (d["pred"] - pm) / ps * asd + am
res_shipped = d["ai_exposure_llm_human"] - fit_shipped
# textbook: least squares. Because the layers correlate negatively this slopes DOWN.
slope = np.polyfit(d["pred"], d["ai_exposure_llm_human"], 1)[0]
res_ols = d["ai_exposure_llm_human"] - (am + slope * (d["pred"] - pm))
rank_shipped = int(res_shipped.rank(ascending=False, method="first")[subj.name])
rank_ols = int(res_ols.rank(ascending=False, method="first")[subj.name])
check("piece 01 · OLS slope is negative", round(float(slope), 4), P1["robustness"]["ols_slope"])
check("piece 01 · residual rank, shipped", rank_shipped, P1["robustness"]["subject"]["shipped_rank"])
check("piece 01 · residual rank, OLS", rank_ols, P1["robustness"]["subject"]["ols_rank"])
check("piece 01 · admits scale dependence", rank_shipped != rank_ols,
P1["robustness"]["residual_rank_is_scale_dependent"])
print()
print("So: the correlation and the displacement hold under either method.")
print("The residual ranking does not, which is why the piece no longer leads with it.")PASS piece 01 · OLS slope is negative
recomputed -5.1569 published -5.1569
PASS piece 01 · residual rank, shipped
recomputed 1 published 1
PASS piece 01 · residual rank, OLS
recomputed 17 published 17
PASS piece 01 · admits scale dependence
recomputed True published True
So: the correlation and the displacement hold under either method.
The residual ranking does not, which is why the piece no longer leads with it.
Claim 4. Exactly one of the 21 abilities scores negative
neg = abil[abil["ai_exposure_score"] < 0]
check("piece 01 · one negative ability", len(neg), 1)
check("piece 01 · it is Originality", neg.iloc[0]["cognitive_ability"], "Originality")
check("piece 01 · its score", round(float(neg.iloc[0]["ai_exposure_score"]), 3),
P1["findings"]["originality_exposure"])
# the subject's percentile on it, among all occupations
col = "Originality"
vals = prof[col].dropna()
mine = prof.loc[prof["occupation_title"] == "Writers and authors", col].iloc[0]
pct = 100 * (vals < mine).sum() / len(vals)
published = next(p["percentile"] for p in P1["subject_profile"] if p["ability"] == col)
check("piece 01 · writers' percentile on Originality", round(float(pct), 1), published)
print()
print(abil.nlargest(3, "ai_exposure_score")[["cognitive_ability", "ai_exposure_score"]].to_string(index=False))
print(abil.nsmallest(3, "ai_exposure_score")[["cognitive_ability", "ai_exposure_score"]].to_string(index=False))PASS piece 01 · one negative ability
recomputed 1 published 1
PASS piece 01 · it is Originality
recomputed 'Originality' published 'Originality'
PASS piece 01 · its score
recomputed -0.144 published -0.144
PASS piece 01 · writers' percentile on Originality
recomputed 97.4 published 97.4
cognitive_ability ai_exposure_score
Information Ordering 1.908
Memorization 1.686
Speed of Closure 1.377
cognitive_ability ai_exposure_score
Originality -0.144
Spatial Orientation 0.375
Fluency of Ideas 0.401
--- ## Piece 02 — The Line Someone Drew
Claim 1. The screen-time association is real, precise, and tiny — and sleep is eight times larger
r_screen = teens["screen_time_index"].corr(teens["bdi_total"])
r_sleepq = teens["sleep_quality_index"].corr(teens["bdi_total"])
check("piece 02 · screen r", round(float(r_screen), 4), P2["headline"]["screen_r"])
check("piece 02 · screen r2 %", round(100 * float(r_screen) ** 2, 1), P2["headline"]["screen_r2_pct"])
check("piece 02 · sleep quality r2 %", round(100 * float(r_sleepq) ** 2, 1), P2["headline"]["sleepq_r2_pct"])
check("piece 02 · variance ratio", round(float(r_sleepq ** 2 / r_screen ** 2), 1),
P2["headline"]["variance_ratio"], tol=0.06)
# Spearman, because the screen index is ordinal. Done by its definition —
# rank, then Pearson — rather than via pandas' method="spearman", which reaches
# for scipy. Keeps this notebook to pandas + numpy.
rho = teens["screen_time_index"].rank().corr(teens["bdi_total"].rank())
check("piece 02 · spearman", round(float(rho), 4), P2["robustness"]["screen"]["spearman"])
# how much of it travels with hours slept
rxz = teens["screen_time_index"].corr(teens["avg_sleep_hours"])
rzy = teens["avg_sleep_hours"].corr(teens["bdi_total"])
part = (r_screen - rxz * rzy) / math.sqrt((1 - rxz ** 2) * (1 - rzy ** 2))
absorbed = 100 * (1 - abs(part) / abs(r_screen))
pub = next(p for p in P2["partials"] if p["control"] == "avg_sleep_hours")
check("piece 02 · absorbed by hours slept, %", round(float(absorbed), 1), pub["absorbed_pct"], tol=0.06)PASS piece 02 · screen r
recomputed 0.1344 published 0.1344
PASS piece 02 · screen r2 %
recomputed 1.8 published 1.8
PASS piece 02 · sleep quality r2 %
recomputed 14.5 published 14.5
PASS piece 02 · variance ratio
recomputed 8.0 published 8.0
PASS piece 02 · spearman
recomputed 0.1455 published 0.1455
PASS piece 02 · absorbed by hours slept, %
recomputed 46.0 published 46.0
Claim 2. The file's own depressed column is a line at 14
The source ships a yes/no flag without saying what produced it. Search for it rather than assume.
found = [(c, float(((teens["bdi_total"] >= c).astype(float) == teens["depressed"]).mean()))
for c in range(5, 31)]
exact = [c for c, a in found if a == 1.0]
runner = max((t for t in found if t[1] < 1.0), key=lambda t: t[1])
check("piece 02 · exactly one cutoff reproduces the column", len(exact), 1)
check("piece 02 · and it is", exact[0], P2["headline"]["cutoff_verified"])
check("piece 02 · next best cutoff", runner[0], P2["headline"]["cutoff_runner_up"])
check("piece 02 · next best agreement", round(runner[1], 4), P2["headline"]["cutoff_runner_up_agreement"])
print()
print("Nothing in the data forces 14. It is a line somebody drew.")PASS piece 02 · exactly one cutoff reproduces the column
recomputed 1 published 1
PASS piece 02 · and it is
recomputed 14 published 14
PASS piece 02 · next best cutoff
recomputed 13 published 13
PASS piece 02 · next best agreement
recomputed 0.9798 published 0.9798
Nothing in the data forces 14. It is a line somebody drew.
Claim 3. The same teenagers support 130 different headlines, 38 of them indistinguishable from nothing
Two thresholds have to be chosen before “N times more likely to be depressed” can be written. Every combination is recomputed here, with a Katz interval on each.
rows = []
for k in P2["grid"]["thresholds"]:
heavy = teens[teens["screen_time_index"] >= k]
light = teens[teens["screen_time_index"] < k]
for cut in P2["grid"]["cutoffs"]:
a = int((heavy["bdi_total"] >= cut).sum()); n1 = len(heavy)
c = int((light["bdi_total"] >= cut).sum()); n0 = len(light)
rr = (a / n1) / (c / n0)
se = math.sqrt(1 / a - 1 / n1 + 1 / c - 1 / n0)
rows.append({"k": k, "cut": cut, "rr": rr,
"lo": math.exp(math.log(rr) - 1.96 * se),
"solid": math.exp(math.log(rr) - 1.96 * se) > 1})
grid = pd.DataFrame(rows)
check("piece 02 · grid size", len(grid), P2["robustness"]["grid_cells"])
check("piece 02 · lowest ratio", round(float(grid["rr"].min()), 3), P2["grid"]["rr_min"])
check("piece 02 · highest ratio", round(float(grid["rr"].max()), 3), P2["grid"]["rr_max"])
check("piece 02 · intervals clear of 1", int(grid["solid"].sum()), P2["robustness"]["grid_solid"])
print()
print(f"{int((~grid['solid']).sum())} of {len(grid)} are quotable point estimates whose")
print("interval includes 1 — arithmetically true, statistically indistinguishable")
print("from no difference at all.")PASS piece 02 · grid size
recomputed 130 published 130
PASS piece 02 · lowest ratio
recomputed 1.147 published 1.147
PASS piece 02 · highest ratio
recomputed 2.967 published 2.967
PASS piece 02 · intervals clear of 1
recomputed 92 published 92
38 of 130 are quotable point estimates whose
interval includes 1 — arithmetically true, statistically indistinguishable
from no difference at all.
Claim 4. The flat per-item loading says nothing about screens
The piece originally read screen time's even spread across the 21 questionnaire items as evidence against a mechanism. Running the same breakdown for sleep killed that inference, and the withdrawal is published. Verify it.
by_id = items.set_index("subject_id")
merged = teens.set_index("subject_id").join(by_id, rsuffix="_it")
item_cols = [f"bdi_item_{i:02d}" for i in range(1, 22)]
def spread(col):
rs = np.array([merged[col].corr(merged[ic]) for ic in item_cols])
return rs, rs.std(ddof=0) / abs(rs.mean())
_, s_screen = spread("screen_time_index")
_, s_sleep = spread("sleep_quality_index")
check("piece 02 · screen spread ÷ size", round(float(s_screen), 3), P2["flatness_audit"]["screen_spread"])
check("piece 02 · sleep spread ÷ size", round(float(s_sleep), 3), P2["flatness_audit"]["sleep_spread"])
check("piece 02 · sleep is at least as flat", bool(s_sleep <= s_screen), True)
print()
for col in ["screen_time_index", "sleep_quality_index", "avg_sleep_hours",
"midsleep_weekend_hours", "social_jetlag_hours"]:
rs, sp = spread(col)
print(f" {col:24s} mean {rs.mean():+.4f} spread÷size {sp:.3f} "
f"strongest item {int(np.argmax(np.abs(rs))) + 1:02d}")
print()
print("Sleep has eight times the explanatory power and an obvious mechanism, and")
print("its profile is if anything MORE uniform. Flatness is a property of this")
print("questionnaire, not a finding about screens.")PASS piece 02 · screen spread ÷ size
recomputed 0.218 published 0.218
PASS piece 02 · sleep spread ÷ size
recomputed 0.187 published 0.187
PASS piece 02 · sleep is at least as flat
recomputed True published True
screen_time_index mean +0.0799 spread÷size 0.218 strongest item 03
sleep_quality_index mean +0.2247 spread÷size 0.187 strongest item 19
avg_sleep_hours mean -0.1537 spread÷size 0.215 strongest item 19
midsleep_weekend_hours mean +0.0878 spread÷size 0.366 strongest item 19
social_jetlag_hours mean +0.0421 spread÷size 0.581 strongest item 19
Sleep has eight times the explanatory power and an obvious mechanism, and
its profile is if anything MORE uniform. Flatness is a property of this
questionnaire, not a finding about screens.
--- ## Result
summary = pd.DataFrame(CHECKS, columns=["claim", "recomputed", "published", "ok"])
print(summary.to_string(index=False))
print()
print(f"{int(summary['ok'].sum())} of {len(summary)} published claims re-derived "
f"independently, in pandas, from the raw source.")
print()
print("The pipelines are stdlib and this notebook is not, so where the two agree")
print("the number is not an artefact of either. Where the site admits a weakness")
print("— the scale-dependent residual, the withdrawn flatness inference — the")
print("admission checks out too.")
assert summary["ok"].all() claim recomputed published ok
piece 01 · the two layers correlate -0.367 -0.367 True
piece 01 · shared variance 0.135 0.135 True
piece 01 · CI low -0.4658 -0.4658 True
piece 01 · CI high -0.2591 -0.2591 True
piece 01 · predicted rank 267 267 True
piece 01 · rated rank 4 4 True
piece 01 · displacement 263 263 True
piece 01 · largest displacement of all 1 1 True
piece 01 · OLS slope is negative -5.1569 -5.1569 True
piece 01 · residual rank, shipped 1 1 True
piece 01 · residual rank, OLS 17 17 True
piece 01 · admits scale dependence True True True
piece 01 · one negative ability 1 1 True
piece 01 · it is Originality Originality Originality True
piece 01 · its score -0.144 -0.144 True
piece 01 · writers' percentile on Originality 97.4 97.4 True
piece 02 · screen r 0.1344 0.1344 True
piece 02 · screen r2 % 1.8 1.8 True
piece 02 · sleep quality r2 % 14.5 14.5 True
piece 02 · variance ratio 8.0 8.0 True
piece 02 · spearman 0.1455 0.1455 True
piece 02 · absorbed by hours slept, % 46.0 46.0 True
piece 02 · exactly one cutoff reproduces the column 1 1 True
piece 02 · and it is 14 14 True
piece 02 · next best cutoff 13 13 True
piece 02 · next best agreement 0.9798 0.9798 True
piece 02 · grid size 130 130 True
piece 02 · lowest ratio 1.147 1.147 True
piece 02 · highest ratio 2.967 2.967 True
piece 02 · intervals clear of 1 92 92 True
piece 02 · screen spread ÷ size 0.218 0.218 True
piece 02 · sleep spread ÷ size 0.187 0.187 True
piece 02 · sleep is at least as flat True True True
33 of 33 published claims re-derived independently, in pandas, from the raw source.
The pipelines are stdlib and this notebook is not, so where the two agree
the number is not an artefact of either. Where the site admits a weakness
— the scale-dependent residual, the withdrawn flatness inference — the
admission checks out too.
analysis/notebooks/audit.ipynb · generated by make_audit.py · pandas + numpy only, no scipy
Two guards, different jobs: checks that the committed data still comes out of the pipelines; this checks that the sentences still match the data.
The pipelines are never run automatically. This checks them instead.
It would be easy to have a scheduled job re-run both pipelines and commit whatever came out. That would be a mistake. The prose on this site is written around specific findings — “not weak, inverted”, “the only negative one”, two fables — and no automation can rewrite a fable when a number moves. Auto-publishing would put new numbers underneath old sentences, which is the exact failure this whole site is built to avoid.
So nothing here ever writes to web/content. It re-runs both pipelines and reports whether the committed data still reproduces, naming three failures apart:
$ python analysis/verify.py ─── 01-ai-exposure ─── OK 4268 values, identical to the commit ─── 02-screen-time ─── OK 1442 values, identical to the commit PASS every committed data.json reproduces exactly.
analysis/verify.py
Run as a pull-request gate, this catches drift and tampering without ever publishing either. There is no backend and no database anywhere in this project: the data is computed ahead of time, committed as JSON, and served as static files.