chore: track live agent config (config, models, append-system, managed-skills, memories, plugin lock)
This commit is contained in:
1 parent
772607434a
commit
552d1ef8bf
23 files changed
+640
No files matched your search
@@ -0,0 +1,69 @@
|
||||
---
|
||||
name: bench-llama-server-params
|
||||
description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode"
|
||||
---
|
||||
|
||||
# Benchmarking llama.cpp Server-Side Parameters
|
||||
|
||||
## Problem
|
||||
`llama-bench` benchmarks the bench binary itself, not the running server.
|
||||
It cannot measure the effect of server-side parameters like `--threads-batch`,
|
||||
`--ubatch-size`, `--spec-type`, or `--parallel`.
|
||||
|
||||
## Solution: HTTP-Based Benchmarking
|
||||
|
||||
### Prefill throughput (prompt processing)
|
||||
1. Start the server with the target parameter value
|
||||
2. Send a large prompt via the chat API with `max_tokens=1`
|
||||
3. Measure `prompt_tokens / wall_time` from the response usage
|
||||
|
||||
`max_tokens=1` is correct here — you want only prompt processing time,
|
||||
not generation. The server crashes on `max_tokens=0`.
|
||||
|
||||
### Decode throughput (generation)
|
||||
1. Start the server with the target parameter value
|
||||
2. Send a prompt that generates a long response
|
||||
3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode
|
||||
benchmarking. One token tells you nothing about sustained generation rate.
|
||||
4. Measure `predicted_per_second` from the response `timings` object, or
|
||||
compute `completion_tokens / wall_time`
|
||||
|
||||
### Why at least 16k tokens for decode
|
||||
- Decode throughput varies over the generation (warmup, cache growth, MoE
|
||||
routing). A single token is noise.
|
||||
- Speculative decoding acceptance rates stabilize over thousands of tokens.
|
||||
- Short generations finish before the draft head reaches steady state.
|
||||
- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).
|
||||
|
||||
### Key Pattern
|
||||
```python
|
||||
# Decode benchmark — measure generation throughput
|
||||
body = json.dumps({
|
||||
"model": "<alias>",
|
||||
"messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
|
||||
"max_tokens": 16384, # 16k minimum for decode; use max_tokens=1 ONLY for prefill
|
||||
"temperature": 0, # deterministic for reproducibility
|
||||
"stream": False
|
||||
})
|
||||
# Measure wall time from request send to response receipt
|
||||
# Get predicted_per_second from timings, or completion_tokens / wall_time
|
||||
```
|
||||
|
||||
### When to Use
|
||||
- `--threads-batch` (prefill): use `max_tokens=1`
|
||||
- `--ubatch-size` (decode): use `max_tokens=16384+`
|
||||
- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only)
|
||||
- Any server-side parameter that `llama-bench` cannot target
|
||||
|
||||
### Benchmarking policy (MANDATORY)
|
||||
1. **STOP the production service** before running any benchmark
|
||||
2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts
|
||||
`disable` deletes the unit file from `/etc/systemd/system/`)
|
||||
3. Run the bench
|
||||
4. **Unmask + restart** the service when done
|
||||
5. **Verify health** before reporting results
|
||||
|
||||
### Example Scripts
|
||||
- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1)
|
||||
- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+)
|
||||
- `scripts/bench-spec-decode.py` — speculative decoding config sweep
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: quadlet-v1-generator-quirks
|
||||
description: "When writing or deploying Quadlet .container units on podman 5.x (v1 generator, e.g. Dankclaw), avoid keys the v1 generator hard-fails on and verify generated ExecStart before starting; generated units cannot be enabled."
|
||||
---
|
||||
|
||||
# Quadlet v1 generator quirks (podman 5.x)
|
||||
|
||||
The podman 5.x user generator (`/usr/lib/systemd/user-generators/podman-user-generator`, quadlet **v1**) behaves differently from the v4 docs. Verified live on Dankclaw 2026-08-28 while deploying the authentik trio.
|
||||
|
||||
## Hard-fail keys (unit silently fails to generate — `systemctl --user cat <unit>` shows "could not be found")
|
||||
- Any dash-prefixed option line in `[Container]`: `--shm-size=512m` etc. → generator logs `unsupported key '--shm-size' in group 'Container'` and skips the whole unit. Omit them; accept podman defaults.
|
||||
- `Command=` is unsupported — use `Exec=` (that is what this generator accepts).
|
||||
|
||||
## HealthCmd space-splitting
|
||||
The v1 generator splits `HealthCmd` on spaces into the healthcheck argv array (`podman inspect --format {{.Config.Healthcheck.Test}} <ctr>` shows `[CMD-SHELL pg_isready -d ... ]`). A quoted probe like `/bin/sh -c "pg_isready ..."` lands as argv token `"pg_isready` → command-not-found → container reads **unhealthy while the DB is actually up**. Rule: HealthCmd must contain only space-free argv tokens, e.g. `HealthCmd=pg_isready -d authentik -U authentik` or `python /manage.py check --database default`.
|
||||
|
||||
## Generated units cannot be enabled
|
||||
`systemctl --user enable <generated-unit>` fails with "unit is transient or generated". Boot persistence comes from `[Install] WantedBy=default.target` inside the `.container` file, honored by the generator at user-manager start; linger must stay on for the deploy user. Deploy handlers: install unit → `systemctl --user daemon-reload` → `systemctl --user start`; no `enable`.
|
||||
|
||||
## Probe a new key before shipping it
|
||||
Drop a throwaway `probe.container` into `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, then check the generator verdict:
|
||||
|
||||
```bash
|
||||
journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key"
|
||||
systemctl --user cat probe.service # shows the generated ExecStart verbatim
|
||||
rm ~/.config/containers/systemd/probe.container && systemctl --user daemon-reload
|
||||
```
|
||||
|
||||
If a healthcheck ever reads unhealthy but logs look fine, suspect this quirk first: compare the baked `Healthcheck.Test` array against the intended command before believing the status.
|
||||
Reference in new issue
Block a user