docs: 2026-08-21 incident (distro upgrade dropped CUDA 12 runtime) + Dankserver now Ubuntu 26.04

docs/Dankserver System Specs.md

@@ -2,7 +2,7 @@
 
 ## Server Overview
 
-Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.
+Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM.
 
 ## Hardware Specifications
 
@@ -54,9 +54,9 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
 
 | Property | Value |
 |----------|-------|
-| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
+| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
-| Kernel | 7.0.0-28-generic |
+| Kernel | 7.0.0-30-generic |
-| Python | 3.12.3 |
+| Python | 3.14.4 |
 
 ### Active Services
 

docs/deployment.md

@@ -160,6 +160,22 @@ If a service segfaults at load with no obvious cause, compare the exported env a
 
 Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
 
+## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21)
+
+On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and `nvidia-cuda-toolkit`. The `llama-server` binary links `libcudart.so.12` and `libcublas.so.12`. On every start the binary exited with code 127. systemd restart-looped the service continuously. The health endpoint on port 7770 stayed down.
+
+Fix: reinstall the runtime packages. The same versions remain in the 26.04 repository.
+
+    sudo apt-get install -y libcudart12 libcublas12 libcublaslt12
+
+Verify that the binary resolves all libraries:
+
+    ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found"
+
+Start the service and check the health endpoint on port 7770.
+
+After any release upgrade on Dankserver, run the `ldd` check above. A missing CUDA 12 library means the packages need a reinstall.
+
 ## See Also
 
 - [AGENTS.md](../AGENTS.md) — the git workflow rules

docs/index.md

@@ -8,7 +8,7 @@ CIRU ROCmFPX llama.cpp on Dankclaw (frank-default-large).
 
 ```mermaid
 graph LR
-    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
+    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
         A[RTX 3090 24GB + 62GB RAM]
         B[upstream llama.cpp b10121
 frank-default-small, port 7770]
@@ -34,7 +34,7 @@ frank-default-large, port 52625]
 
 | Server                                  | GPU                           | VRAM / Memory        | OS           |
 | --------------------------------------- | ----------------------------- | -------------------- | ------------ |
-| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090               | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 |
+| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090               | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 |
 | [[Dankclaw System Specs\|Dankclaw]]     | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
 
 ## Active Deployment