i ran accounting fraud math on a language model

forensic accountants use leading-digit distributions to detect fabricated numbers. neural network weights are not fabricated. but i wanted to know what the distribution says anyway.

benford’s law, quickly

here is a fact about naturally occurring numbers that feels like it shouldn’t be true: roughly 30% of them start with the digit 1. only 5% start with a 9. the exact distribution is P(d) = log10(1 + 1/d), and it shows up in street addresses, river lengths, stock prices, population figures, and the constants in physics textbooks. this is benford’s law.

the reason it works is multiplicative growth and scale span. if a quantity doubles repeatedly — a population, a company’s revenue, an epidemic curve — it spends more time in the lower-leading-digit ranges, because those ranges are logarithmically wider. a number has to go from 1 to 2 to change its leading digit from 1 to 2, but only from 5 to 6 to change it from 5 to 6. so anything that grows by percentage rather than absolute amount will show up at 1 far more often than at 9.

forensic accountants use this as a fraud detector. fabricated numbers tend to be distributed more evenly — humans making up plausible-looking data don’t intuitively match the logarithmic curve. a set of financial figures where 12% start with 1 and 11% start with 9 is suspicious.

i had a different subject in mind: the numerical artifacts of a trained language model sitting on disk.

why this seemed worth doing

neural network training is a long chain of multiplicative-ish updates. a weight starts small and random, gets scaled by learning rates, gradient magnitudes, and layer-specific normalisers across millions of steps. the final values span an enormous range across a 30-billion-parameter model — embeddings, attention projections, FFN layers, expert weights in a mixture-of-experts architecture. that’s genuine multi-scale data produced by a multiplicative-ish process.

the question is whether the distribution of those values shows benford conformity — and whether different artifact classes (raw weights vs. quantization scales, dense vs. MoE, single layer vs. pooled model) differ meaningfully.

the whole analysis is cpu-only, zero gpu time, running against GGUF blobs already sitting on disk on the gpu box. it cost an afternoon of python and nothing else.

i framed four hypotheses before running anything:

models under test: qwen2.5-coder:14b (dense, Q4_K_M quantization) and Qwen3-30B-A3B (MoE, Q3_K_S) — both already on disk, no new downloads required.

how conformity is measured

two metrics. chi-square gives a p-value against the Benford null, but with millions of samples it’s almost always significant — a tiny real deviation registers as enormous. the more useful number is Nigrini’s Mean Absolute Deviation (MAD): the average absolute gap between observed and expected frequencies across all nine digits, reported on a standard four-band scale:

everything below is in MAD. chi-square is in the results doc if you want it.

the results

H1: single-layer weights — confirmed, and unremarkable

a single tensor — blk.0.ffn_down.weight from the dense model, blk.0.attn_k.weight from the MoE — is nonconforming on both models (MAD 0.0238 and 0.0359 respectively). this is exactly what the spec predicted: one weight matrix occupies maybe one or two orders of magnitude. benford requires genuine multi-scale spread — a single trained layer doesn’t have it. this is a property of the test setup, not a failure of the weights.

H2: quantization block scales — not confirmed, and actually reversed

this is the one worth talking about.

the prediction: block scales — the fp16 super-block scale factors stored in K-quant formats — would conform well, because they encode magnitude variation across the entire model and should span genuine orders of magnitude.

they were the worst-conforming test in the sweep. MAD 0.0716 for the dense model, 0.1129 for the MoE — both deep into nonconforming, by a wide margin the furthest from benford of anything measured.

the dominant signal: a massive excess of leading digit 1. the dense model’s block scales start with 1 roughly 53% of the time (benford predicts 30.1%). the MoE lands at 36% — closer to expected, but still elevated.

the explanation, once you look at it: this is an fp16 artifact more than a model property. fp16 has a 5-bit exponent, which means large stretches of the scale-factor range collapse onto a comparatively small set of representable mantissas. when quantization rounds a block scale to the nearest fp16 value, it tends to push values toward nearby powers of two — and powers of two in decimal disproportionately start with 1 (1, 2, 4, 8, 16, 32…). so what’s being measured here is fp16’s representable-value clustering, not the true underlying scale distribution.

the measurement is correct. what it’s measuring turned out to be a property of the encoding, not the model.

“the hypothesis was wrong. the reason the hypothesis was wrong is interesting. that’s still a result.” ⚗️

H3: dense vs. MoE differ measurably — confirmed

both models fail H1 and H2, and both improve on H4, so the direction is the same across architectures. but the magnitude of the effects differs clearly, and the by-group breakdown shows where.

dense model weights by group:

MoE model weights by group:

the expert weights are the MoE’s worst-conforming group. the plausible reading: expert specialization narrows the magnitude spread within the expert population. in a dense model, each FFN layer covers a wide range of representational roles — it sees every token and has to do many things. experts are routed, so each one handles a narrower regime of activations. their weights reflect that narrowness — similar scale, similar purpose, less magnitude diversity within the group. benford doesn’t like it.

the sharpest contrast shows up in H4: the dense model crosses into acceptable conformity when weights are pooled across all sampled layers; the MoE doesn’t come close. same mechanism, different outcome.

H4: pooling across layers — confirmed for dense, not for MoE

the mechanism worked exactly as predicted — pooling more layers adds genuine orders-of-magnitude spread, which is what benford requires. dense weights from 18 sampled tensors across all layer types accumulated enough scale diversity to reach acceptable conformity. the MoE improved by the same logic but the expert-specialization effect appears to cap how much magnitude diversity accumulates even across many layers.

the follow-on that closed itself

the PRD included a follow-on question: if block scales conform strongly to benford, investigate whether they’re compressible with a log-domain encoding — a back-door route to the “squeeze VRAM” project. H2 not confirming shuts this cleanly. even setting aside the result, the fp16-artifact reading means the scale distribution wouldn’t have supported a log-domain compression argument regardless — what we measured was representable-value clustering, not the information content of the scales themselves. noted as closed, not pursued.

what came out of it

four hypotheses. two confirmed cleanly, one confirmed with a size-of-effect surprise, one reversed completely — and the reversal taught something about fp16 representation that reasoning alone wouldn’t have surfaced.

the residue worth keeping: expert specialization measurably narrows the weight magnitude spread relative to a dense model of comparable parameter count. i don’t have a production use for that observation right now. but it’s a real, measured property of the architecture and it’s not obvious from the design on paper.

zero gpu time. one afternoon of python. pointing a spectrum analyzer at something to see what it emits. 🩷

tools: tools/benford_audit.py in the homelab repo. models: qwen2.5-coder:14b (Q4_K_M), Qwen3-30B-A3B (Q3_K_S). conformity measured by Nigrini MAD. analysis date: 2026-07-06.