Token Identity Routing — Interactive Paper

Interactive architecture study

Token identity as a routing signal for residual MLP experts

Explore how a fixed lexical lookup selects a narrow residual parameter subspace while a shared dense SwiGLU path handles contextual computation.

306.5M parameters 8B training tokens fixed top-2 routing

Interactive companion — not additional experimental evidence.

Token

Contextual hidden state x_t The same contextual vector enters both branches

Token identifier t routing · tokenizer ID 22447

Shared SwiGLU Width 3,840 · all tokens · contextual computation

Fixed route table r_l(t) Selects two of four experts

Residual experts E0 E1 E2 E3

Feed-forward output Shared path + 0.5 E1 + 0.5 E2

Layer-specific deterministic routes

Checkpoint-derived tokenizer IDs and fixed top-2 assignments.

E0E1E2E3 Layer 1—0.50.5— Layer 6——0.50.5 Layer 120.5——0.5 Layer 18——0.50.5

Token “routing” follows a reproducible, layer-specific route. A different occurrence of the same token keeps the same route, while its hidden-state input still changes with context.

Last common evaluation point

Fixed evaluation stream from the FineWeb-Edu training split · lower is better

2.9482

2.9329

Evaluation NLL

1.00×

0.79×

Training throughput

Dense Token-routed

Measurement	Dense	Routed
Parameters	306.5M	306.5M
Evaluation NLL	2.9482	2.9329
NLL difference	—	−0.0153
Training tokens	8B	8B
Seeds	1	1

Single-seed matched-run observation. Evaluation is not held out; downstream multiple-choice differences are within one reported standard error.

Prompt

Dense-306 ready

Dense output will stream here token by token.

TR-MOE-306 ready

Token-routed output will stream here token by token.

Public CPU inference · SSE streaming · qualitative comparison only