From 577003bed1e1e63fcf9e6eb7d7ca5849a048c2b8 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 21 Aug 2026 19:31:05 -0700 Subject: [PATCH] docs: 2026-08-21 incident (distro upgrade dropped CUDA 12 runtime) + Dankserver now Ubuntu 26.04 --- docs/Dankserver System Specs.md | 8 ++++---- docs/deployment.md | 16 ++++++++++++++++ docs/index.md | 4 ++-- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/Dankserver System Specs.md b/docs/Dankserver System Specs.md index 8fbb861..92c8631 100644 --- a/docs/Dankserver System Specs.md +++ b/docs/Dankserver System Specs.md @@ -2,7 +2,7 @@ ## Server Overview -Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM. +Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM. ## Hardware Specifications @@ -54,9 +54,9 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0 | Property | Value | |----------|-------| -| OS | Ubuntu 24.04.4 LTS (Noble Numbat) | -| Kernel | 7.0.0-28-generic | -| Python | 3.12.3 | +| OS | Ubuntu 26.04 LTS (Resolute Raccoon) | +| Kernel | 7.0.0-30-generic | +| Python | 3.14.4 | ### Active Services diff --git a/docs/deployment.md b/docs/deployment.md index 674854b..1a4bca2 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -160,6 +160,22 @@ If a service segfaults at load with no obvious cause, compare the exported env a Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation. +## Incident: frank-default-small Crash Loop After Distro Upgrade (2026-08-21) + +On 2026-08-21 a release upgrade moved Dankserver from Ubuntu 24.04 to Ubuntu 26.04 (Resolute Raccoon). The upgrade removed the CUDA 12 runtime packages: `libcudart12`, `libcublas12`, `libcublaslt12`, and `nvidia-cuda-toolkit`. The `llama-server` binary links `libcudart.so.12` and `libcublas.so.12`. On every start the binary exited with code 127. systemd restart-looped the service continuously. The health endpoint on port 7770 stayed down. + +Fix: reinstall the runtime packages. The same versions remain in the 26.04 repository. + + sudo apt-get install -y libcudart12 libcublas12 libcublaslt12 + +Verify that the binary resolves all libraries: + + ldd /home/dank/llama.cpp/build/bin/llama-server | grep "not found" + +Start the service and check the health endpoint on port 7770. + +After any release upgrade on Dankserver, run the `ldd` check above. A missing CUDA 12 library means the packages need a reinstall. + ## See Also - [AGENTS.md](../AGENTS.md) — the git workflow rules diff --git a/docs/index.md b/docs/index.md index 82f8c6b..f03b0fc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,7 +8,7 @@ CIRU ROCmFPX llama.cpp on Dankclaw (frank-default-large). ```mermaid graph LR - subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] + subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04] A[RTX 3090 24GB + 62GB RAM] B[upstream llama.cpp b10121 frank-default-small, port 7770] @@ -34,7 +34,7 @@ frank-default-large, port 52625] | Server | GPU | VRAM / Memory | OS | | --------------------------------------- | ----------------------------- | -------------------- | ------------ | -| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 | +| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 | | [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 | ## Active Deployment