docs(vault): one fact, one owner — repair the doubled pages, single-source the facts
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
This commit is contained in:
1 parent
3880503057
commit
a5fe045b58
122 files changed
+2412
-2037
No files matched your search
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Container Host Reachability
|
||||
description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
|
||||
description: "Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability."
|
||||
tags: [podman, netavark, networking, rootless, reachability, caddy]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Audit and Posture
|
||||
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
|
||||
description: "Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture."
|
||||
tags: [audit, security, prometheus, caddy, monitoring, git]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Ops Playbook
|
||||
description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
|
||||
description: "Incident-shaped procedures: endpoint down with a GPU holder, stop-first bench windows, service iteration, waking a host."
|
||||
tags: [operations, incidents, gpu, systemd, wol, forensics]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Payload and Disk Reclaim
|
||||
description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification.
|
||||
description: "Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval."
|
||||
tags: [operations, disk, podman, payloads, huggingface, trash]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Remote Job Runner
|
||||
description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline.
|
||||
description: "Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification."
|
||||
tags: [operations, ssh, jobs, sentinel, detached]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: SSH and Remote Access
|
||||
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
|
||||
description: "Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage."
|
||||
tags: [ssh, remote, fish, gui, diagnosis]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
@@ -1,28 +1,15 @@
|
||||
---
|
||||
type: index
|
||||
description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks.
|
||||
tags: [operations, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Operations
|
||||
|
||||
Procedures for operating the fleet. The LLM-specific pages live under
|
||||
[LLM](../LLM/index.md); infrastructure and architecture under the
|
||||
`../Infrastructure/` and `../Architecture/` folders.
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files;
|
||||
why ssh tail-watchers fail
|
||||
- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk
|
||||
procedure, plus the purge-and-refresh protocol
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
- [[Remote Job Runner]] — Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification.
|
||||
- [[Payload and Disk Reclaim]] — Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval.
|
||||
- [[Container Host Reachability]] — Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability.
|
||||
- [[Fleet Ops Playbook]] — Incident-shaped procedures: endpoint down with a GPU holder, stop-first bench windows, service iteration, waking a host.
|
||||
- [[Fleet Audit and Posture]] — Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture.
|
||||
- [[SSH and Remote Access]] — Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage.
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
|
||||
Reference in new issue
Block a user