A real Transformer that learns — and the part still small enough to call a seed. 635
parameters, trained by hand-written analytic backprop, reaching 95–98% on a deliberately
non-trivial toy task. This is where the climb is steepest, and most honest.
It actually learns JUST UNIFIED
dtr.c · moe.c
Analytic backpropagation (finite-difference-checked) over the whole 635-parameter graph. A week ago a reviewer found three separate models quietly deciding, returning, and guarding inside one function. As of this week they are one: a single learned forward now routes, answers, and drives the reflex — using all four inputs, not a hardcoded threshold.
Distributed attention — one mind across nodes
dkva.c
Splits a softmax attention across nodes and sums the partials — mathematically identical to single-node attention, on the real inference path. The cluster genuinely computes one attention together.
the per-node KV caches it merges are synthetic warm-up vectors seeded at boot, not memory from a genuinely shared long context. The plumbing is real; the "shared mind" is still primed, not lived.
Collective learning beats the individual
gossip_learn.c
Nodes each train on a disjoint shard (each missing a whole class), then gossip-average their models with no central server — and the averaged swarm provably exceeds any single node's ceiling. Learning survives kills; rejoining nodes catch up.
Self-improvement while idle
ga.c
HONEST WIP
During idle cycles the kernel mutates its own weights and keeps the winners — a real mutate/evaluate/select loop that really runs in its sleep.
it currently optimizes confidence, not accuracy — so today it can make the model more sure rather than more right. A known limitation we're not hiding. Fixing the objective is on the list.
A seed designed to grow
roadmap
Today: 4 tokens, d_model 8, 2 heads, 635 params, 3 nodes. The architecture is shaped to widen with the cluster — but real width (R3) and federation past 32 nodes are designed, not built.
we are not GPT-anything. We're a seed that learns for real. The interesting claim isn't the size — it's that distributed learning genuinely works at all.