frank-primary: measure expert routing; block choice is immaterial

Instrument the live server to answer "can -ot target the most active
layers": AMD IBS op sampling on llama-server during 4 mixed concurrent
generations, sampled load addresses mapped through /proc/<pid>/maps to
backing tensors with extents derived from consecutive GGUF header
offsets (no quant type-size tables - a wrong per-element size silently
mis-buckets tensor tails, first run lost 59% of expert samples to a
junk bucket).

Measured (75 s window, 2.7M expert-tensor samples): per-block read
volume is FLAT, 0.93x-1.08x of mean across all 40 CPU-resident blocks.
MoE top-k routing fires every block every token, so "most active
layers" do not exist at -ot granularity - any 8 GPU-resident blocks
are equivalent, the lever is COUNT, already at the 5/3 VRAM ceiling.
Real skew is per-EXPERT (top 10% of 48x512 cells carry 40.5% of
samples), below tensor-override granularity. No config change.

scripts/frank-primary-expert-activity.py: dump + sample + analyze in
one entry point; --analyze-only re-analyzes saved samples. Instruments
production serving in place; not a benchmark.

serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md

@@ -163,7 +163,8 @@ miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across
 both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
 2,048 MiB on ROCm1.
 
-Hot expert blocks stay GPU-resident. `-ot` granularity is the whole
+Expert blocks stay GPU-resident — and the measurement below shows it
+cannot matter *which* ones. `-ot` granularity is the whole
 per-block expert tensor (512 experts share one tensor), so "critical
 experts" = blocks co-located on the card that holds their layers:
 blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
@@ -189,6 +190,28 @@ delivered, prompt cache reusing 42/64 tokens across requests. All
 three Prometheus fleet targets up; the 5/3 deploy re-verified chat,
 vision and the target on the same battery class.
 
+**Which blocks, measured (same day):** asked whether the `-ot` regex
+could target the *most active* blocks, I instrumented the live server
+instead of guessing — `scripts/frank-primary-expert-activity.py`
+samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
+while 4 mixed concurrent generations run, maps each sampled load
+address through `/proc/<pid>/maps` back to its backing tensor (extents
+derived from consecutive GGUF header offsets — never from a quant
+type-size table, whose errors silently mis-bucket sample tails), and
+aggregates per block and per expert. Verdict, 75 s window / 2.7M
+expert-tensor samples: **per-block read volume is FLAT, 0.93×–1.08× of
+mean across all 40 CPU-resident blocks** (0-4 and 45-47 absent —
+GPU-resident). MoE top-k routing fires every block on every token, so
+"most active layers" do not exist at `-ot` granularity: the regex can
+express any 8-block set, and any set is equivalent to any other — the
+lever is COUNT, already at the 5/3 VRAM ceiling. The real skew is
+per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
+samples; hottest cell 0.1%), below the reach of a tensor-level
+override. Not a benchmark: it instruments production serving in place
+at negligible IBS overhead; re-measure any future checkpoint or
+traffic mix with `frank-primary-expert-activity.py <pid>` (or
+`--analyze-only` against the saved sample file in `/tmp`).
+
 ## Rollback
 
 The lucebox DS4.1 material is fully retained on the host: image