command-center/agent/memories/--home-blank-.omp--/learned.md

• Measuring WHICH card is the bottleneck in a llama.cpp layer-split LLM server on a fleet host (frank-medium, Dankserver 192.168.0.235, 2x R9700 asymmetric PCIe: ROCm0=x16 ~31.5GB/s, ROCm1=x1 ~1.97GB/s) — two traps that produced wrong numbers on 2026-09-18: (1) RUN THE TRANSIENT MEASUREMENT SERVER ON AN ISOLATED PORT, not the production port. If it takes over the prod port (7770), live fleet traffic from other agents hits it and interleaves with your timed request — I measured 35:30 at 2.51 t/s on :7770 vs the true 31.5 t/s; on isolated :7790 (prod stopped to free VRAM, fleet hits the down 7770) it read 31.52 t/s. Override LISTEN_PORT in a temp env copy (sed), point curl at the new port. (2) Use AVERAGE rocm-smi --showuse over a LONG request (2k tokens, ~60s), sampled every 0.5s — PEAK util lies for a layer split because both cards are touched every token and both hit ~100% at peak. Average is the bottleneck indicator. FINDING (durable, specific to frank-medium): the x1 card (ROCm1) is the STRUCTURAL bottleneck at 98% avg util in every config — 35:30/36:29, drafter on ROCm1/ROCm0 all gave util0(x16)=41-43% / util1(x1)=98%, decode ~31.5 t/s in all four. It's COMPUTE-bound, not PCIe-bound (only ~5.2 MB activation crosses the x1 link per 512-token ubatch). The per-step cost is the verify phase (both cards run the full target) + the LOGITS, which are pinned to the card holding the last layers; the MTP drafter is a small nextn head so moving it (SPEC_DRAFT_DEVICE=ROCm0) buys nothing. 35:30 is the optimal split (balanced VRAM 1.27/1.85 GiB free); 36:29 buys 2 util points on the x16 card and erodes its VRAM (fit free 0.45 GiB) for no throughput gain. No llama-server flag moves the lm_head off the last-layer card, so the x1 bottleneck is architectural, not a mis-tune. Recorded in scripts/frank-medium.env (committed 6c6464a).
• placeholder4
• placeholder3
• placeholder2
• placeholder
• frank-medium (Dankserver 192.168.0.235, llama.cpp layer-split 2x R9700 gfx1201) split-balance findings, 2026-09-17: (1) The LAYER SPLIT is NOT a decode-throughput lever at the 393216/MTP shape — 34:31, 35:30 and 36:29 all land in the same ~32 tok/s band (36:29 measured 31.76; 35:30 ~32.4 per env); total work, not imbalance, is the limit. (2) The x1 card (ROCm1, 0b:00.0, Gen4 x1 via B650 chipset ~1.97 GB/s) runs HOT (util ~91-97%) at 35:30 while the x16 card (ROCm0, 03:00.0, Gen4 x16 ~31.5 GB/s) idles (~45%) — because the MTP drafter + output layers + heavier compute buffers are STRUCTURALLY pinned to the card holding the LAST layers, NOT because the PCIe link is the bottleneck (only ~5.2 MB of boundary activation crosses per 512-token ubatch). (3) 36:29 makes the x16 card the utilized worker (util0=42%, util1=3%) at no throughput cost, but spends VRAM headroom (card0 free 0.45 GiB, under the 1.0 GiB floor; 2.22 GiB gap). Frank's standing preference: the card with the FASTEST PCIe lane (x16 = ROCm0) should be the more-utilized card. Deploying 36:29 requires re-fitting the seam test (tests/frank_medium_unit.bats) slack/floor, which currently rejects it.
• niri output config on blankpad (niri 26.04): transform values are COUNTER-CLOCKWISE, so "90" means the panel's top bezel must physically sit on the LEFT ("270" = top bezel right). Two traps when pinning outputs in ~/.config/niri/cfg/display.kdl (symlink into ~/.omp/dotfiles/zz-blankpad/): (1) an output "X" {} block RESETS every property you don't restate to its default — eDP-1 needed scale 1.25 repeated or it would silently drop to 1.0; (2) niri re-runs its auto-layout for any output without a pinned position, so pinning only the externals shoved eDP-1 to the far right (3936,0) — pin eDP-1 at x=0 too if you want the original left-to-right order. Apply with niri validate -c ~/.config/niri/config.kdl then niri msg action load-config-file; confirm with niri msg outputs (shows Logical size swapped to 1200x1920 and "Transform: 90° counter-clockwise"). Live one-off rotation without config: niri msg output NAME transform 90. (context: Rotated both external PM1601 panels (DP-2, HDMI-A-1) to portrait on blankpad 2026-09-15; verified via niri msg outputs, committed 2baf0c0 to dank/command-center.)
• Frank's frank-large ceiling is now 200,000 context in ~/.omp/agent/models.yml (provider llamacpp (dankclaw), committed 60dc0bb to dank/command-center 2026-09-13), while halogen on 192.168.0.143:7770 still serves context=262144 — a deliberate under-serve, not drift. Convention when changing any contextWindow there: read the served value from curl baseUrl/health (it reports context, model, vision.enabled) and check KV-pool admission before/after — admission reserves prompt + max_tokens against the pool (401,408 positions since 2026-09-13), so contextWindow + maxTokens must stay under it or the request hangs the full 3600 s queue timeout before a 503. maxTokens stays pinned at 65536 because HALOGEN_MAX_TOKENS_CAP 400s anything larger. Also: OMP reads models.yml only at startup, so a running session keeps the old ceiling; tell the user new omp invocations pick up the change.
• Frank's terminal font stack on the workstation: package ttf-terminus-nerd (Arch, group nerd-fonts), which provides the TERMINESS outline derivative — families "Terminess Nerd Font" / " Mono" / " Propo". The bitmap original terminus-font is NOT installed, so classic-Terminus crispness depends entirely on 1-bit hinted rendering. Crisp/bitmap look = freetype-load-flags = hinting,monochrome (monochrome = 1-bit, disables AA; the light flag stays in the resolved value but is inert once monochrome is on, per ghostty.5.md). Sizing rule that matters: pixel cell = pt × (96/72) × monitor_scale, and the crisp sweet spot is when that lands on Terminus' native bitmap heights (8x16, 9x18, 10x20, 11x22, 12x24). On Frank's eDP-1 (1920x1080 @ scale 1.25) font-size 12 → 20px → exactly the 10x20 grid; 14pt → 23.3px, fractional and blurry regardless of flags. Check scale with niri msg outputs. Also: ghostty +show-config prints only values differing from default, and Ghostty's default font-size IS 12 — so a 12pt setting vanishes from the diff output; confirm against +show-config --default before concluding the key was ignored.
• NEVER send SIGUSR1 to ghostty on Frank's workstations — it does not reload the config, it kills/respawns the terminal (observed 2026-09-13: PID churn, Frank said "stop killing the terminal"). The old recipe in agent/memories/--home-blank-.omp--/MEMORY.md and skills/dotfiles_git/scripts/reload_ghostty.sh ("pgrep ghostty | while read pid; do kill -USR1 $pid; done") is WRONG for ghostty 1.3.1. Correct way to apply a changed ghostty theme/config without losing shells: press the default keybind ctrl+shift+, (action reload_config) in each window — confirmed as the default in ghostty +show-config --default. For applying theme changes desktop-wide without touching terminals: noctalia msg templates-apply regenerates rendered theme files, and niri msg action load-config-file reloads the compositor (reload-config is NOT a valid niri subcommand). Custom noctalia palettes live at ~/.config/noctalia/palettes/Name.json and are selected with noctalia msg color-scheme-set custom "Name"; that source is immune to the community-palette catalog/md5 sync, unlike editing the cache in ~/.local/state/noctalia/community-palettes/. Also: noctalia msg screenshot-fullscreen saves to ~/Pictures (no need to scrape the PNG out of the clipboard with wl-paste).
• In OMP's ~/.omp/agent/models.yml there are NO per-model sampler keys. Sampling params (temperature, topP, topK, minP, presencePenalty, repetitionPenalty) are GLOBAL settings in config.yml, each defaulting to -1 = "do not send". The only per-model hook is compat.extraBody — a Recordstring, unknown of extra top-level fields merged into every request body. It is applied LAST in the openai-completions builder (applyOpenAIExtraBody runs after applyChatCompletionsCompatPolicy, via Object.assign), so extraBody wins over anything the compat policy or global settings set. Use wire names (top_p, top_k, min_p, presence_penalty, repetition_penalty), not OMP's camelCase setting names. Two traps that cost real debugging time: 1. preserve_thinking is NOT user-settable in models.yml. qwenPreserveThinking is auto-derived inside buildOpenAICompat from the AUTO-DETECTED thinkingFormat plus a local (loopback/RFC1918) baseUrl — and that derivation happens BEFORE applyCompatOverrides. So writing compat.thinkingFormat: qwen in models.yml does NOT turn on preserve_thinking. For a custom-named model (e.g. frank-large, which matches no Qwen id pattern) the only path is extraBody: { preserve_thinking: true }. 2. NEVER put chat_template_kwargs inside extraBody. Object.assign is shallow, so it replaces the whole chat_template_kwargs object and silently drops the reasoning_effort kwarg the compat policy already put there. Top-level preserve_thinking is honoured by halogen-flash-server (its own 400 message says the field "may also be sent as top-level fields"), so top-level is both sufficient and safe. Why it mattered: halogen-flash-server has no server-side sampler defaults — temperature absent or 0 is greedy decode. OMP sent nothing (global -1), so the endpoint was decoding greedy while appearing configured. (context: Verified 2026-09-13 on the Dank fleet while pinning Qwen3.8-Flash-Next recommended sampling (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0) plus preserve_thinking onto frank-large. Ground truth read from the installed bundle: pi-ai/src/providers/openai-completions.ts (applyOpenAIExtraBody applied last), pi-ai/src/providers/openai-shared.ts (applyChatC)
• Frank's Gitea lives on Dankclaw at http://192.168.0.143:3000, single org/user dank. Repo inventory (verified 2026-09-12 via /api/v1/user/repos): agent (default master), command-center (main — this is the ~/.omp dotfiles/KB repo), frank-llm-bench (main), serverAppConfigs (main), vllmScripts (main). LESSON — never clone from a repo name Frank states verbally; resolve the exact full_name via the Gitea API first. He said "serverAppConfig"; the real repo is serverAppConfigs (plural). Cloning the guessed name would 404 with no useful signal. Working procedure: read the dank HTTP token from the existing origin URL in ~/.omp (git remote -v) — do NOT copy that token into memory, docs, or skills; reference its location only. Then GET /api/v1/user/repos and match on the real name/case/plurality before cloning. Local-vs-remote naming: the clone was placed at ~/Projects/serverAppConfig (singular, as Frank explicitly requested) while origin points at the plural repo. The path mismatch is intentional, not drift — pull/push work normally. Repo layout: dankclaw/, dankserver/, frankbook-pro/ (7 quadlet units: sonarr, radarr, lidarr, readarr, prowlarr, qbittorrent, gluetun), shared/ (homepage, alloy, loki, prometheus, searxng), tests/ (47-file E2E suite), serverAppConfigs-docs/ (115-file KB — read its index.md before touching the stack). (context: Frank asked to "pull my serverAppConfig repository into ~/Projects/serverAppConfig"; Gitea API revealed the plural name. Credential handling: token reused from existing ~/.omp remote, deliberately not persisted here.)
• Ghostty 1.3.1 (and the 1.3.x line) has NO font-hinting and NO font-subpixel-render config key. The only font-smoothing key is freetype-load-flags, a comma-separated list: hinting, force-autohint, monochrome, autohint, light (each default-enabled except monochrome/force-autohint; prefix no- to disable). Ground truth: /usr/share/ghostty/doc/ghostty.5.md:598-628. To actually DISABLE anti-aliasing you need freetype-load-flags = hinting,monochrome (docs line 617: monochrome = 1-bit rendering, disables AA). Note hinting,light is a NO-OP — both are already Ghostty defaults. Verification workflow before editing any ghostty key: ghostty +validate-config (exit 0 = valid) plus grep the manpage for the key name; an unknown key silently does nothing useful and +validate-config is the authoritative gate. This matters because a poisoned memory claim about font-subpixel-render/font-hinting caused 3 wasted commits (4c84f02, df33310, 268dcbd) editing a nonexistent key on 2026-09-11. (context: Verified 2026-09-12 against ghostty 1.3.1-arch2.2 manpage and ghostty +validate-config on the workstation. Corrects contradictory entries in agent/memories/--home-blank-.omp--/learned.md (line 2 asserts font-subpixel-render is the key; it is not).)
• Ghostty's font antialiasing is controlled by font-hinting = no-hinting in the config file, not font-subpixel-render = off as the Noctalia reference skill states. When editing ghostty config, check /usr/share/ghostty/doc/ghostty.5.md or src/config/Config.zig — ghostty's --help confirms this. Invalid field causes font-subpixel-render: unknown field on restart. (context: Discovered when user reported "font-subpixel-render: unknown field" after my initial edit to ghostty config)
• Noctalia is a Niri-based desktop environment at /usr/bin/noctalia. Config: ~/.config/noctalia/config.toml. Provides IPC via 'noctalia msg', launcher, lock screen, session menu, and 19 app templates (GTK, Ghostty, Alacritty, Hyprland, Sway, WezTerm, Helix, Btop, Starship, Qt, KDE, Cava, scroll, mango, umbriel, foot, emacs, kitty, alacritty). Templates under /usr/share/noctalia/assets/templates/. Color palette generator: 'noctalia theme [image]'. Text antialiasing: Ghostty uses 'font-subpixel-render' config key (set to 'off' to disable). Alacritty uses 'font.antialias = false' in alacritty.toml.