dank/command-center
1
0 Fork 0
Code Issues 11 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: rebuild KB in OKF v0.1 with live-verified facts

Browse Source 
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center
knowledge base: concept files promoted to path identity, index listings
stripped of frontmatter, all wikilinks/relative links resolved, every
checkable claim verified against the live fleet (Dankserver = Ryzen 7
7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged.
kb-lint exits 0.
This commit is contained in:
Frank committed 2026-08-28 07:57:13 -07:00
1 parent 552d1ef8bf
commit 02883d3ed2
116 files changed
+13008
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

omp-docs/agentic-web-research/agentic-search-best-practices.md
+121
View File
@@ -0,0 +1,121 @@
---
type: concept
title: Agentic Search Best Practices
description: Query formulation, extraction, validation, retry, and rate limiting for an agent that searches the web.
tags:
- agentic
- search
- rag
- retrieval
- unverified
verified: 2026-08-28
timestamp: 2026-08-11
---
# Agentic Search Best Practices
An agent that searches well does not fire one query and trust one answer. It rewrites queries, reads the strongest sources, and checks its own work.
This page gives the practices. See [Research Effectiveness](research-effectiveness.md) for how to orchestrate the loop.
## The Search Pipeline
```mermaid
graph LR
A[User question] --> B[Query rewriter]
B --> C[Query 1]
B --> C2[Query 2]
B --> C3[Query 3]
C --> D[SearXNG]
C2 --> D
C3 --> D
D --> E[Extract content]
E --> F[Re-rank + dedupe]
F --> G[Validate facts]
G --> H[Synthesize + cite]
```
## Query Formulation
- Rewrite the user question into a search-ready query.
- Generate two or three alternative queries for broader coverage.
- Add synonyms and related terms.
- Batch the queries in one action when the framework allows it.
A code-based agent (CodeAct) can run several searches in one action. This cuts LLM calls by about 30 percent compared with one tool call per search. The ReAct loop (reason, act, observe) is the default pattern.
## Extraction and Processing
- Convert HTML to clean Markdown before you feed the model. Pi does this in `fetch_content`.
- Remove near-duplicate results across engines.
- Re-rank with BM25 relevance scoring.
- Keep a source reference for every extracted fact.
- Truncate long pages. A common cap is 40,000 characters per page.
Use query-aware content filters to cut noise before the model sees text. Crawl4AI ships three filters:
- `BM25ContentFilter` — score page chunks against the query. Keep only relevant snippets.
- `PruningContentFilter` — strip boilerplate by text and link density.
- `LLMContentFilter` — delegate filtering to a model for hard pages.
Run BM25 re-ranking on the search results themselves. The SearXNG BM25 reranker plugin builds a temporary index and fuses BM25 scores with engine rankings using reciprocal rank fusion. This lifts keyword accuracy over raw engine order.
## Respect Rate Limits
Upstream engines and SearXNG block agents that hammer them. Slow down on purpose.
- Respect `Retry-After` headers.
- Use exponential backoff on retries.
- Rotate user-agent strings.
- Add random delays between requests.
- Cap the request rate. A common default is 1 request per second.
- Run a circuit breaker that suspends a failing engine.
- Cache results with a time-to-live value.
```yaml
# SearXNG side: cap timeouts and retries
outgoing:
request_timeout: 2.0
max_request_timeout: 10.0
retries: 3
```
SearXNG suspends failing engines on a timer. For example, a CAPTCHA response suspends the engine for 3600 seconds. Respect that pause instead of retrying into the block.
## Validate the Answer
- Cross-reference each claim against a second independent source.
- Prefer primary sources over summaries.
- Record the authority and freshness of each source.
Score credibility on concrete signals:
- Domain type. Prefer `.edu`, `.gov`, and `.org` over personal `.com` sites.
- Link density. A low text-to-link ratio signals a content farm.
- Metadata. A real `<title>` and author byline signal an authored page.
- Cross-reference. Re-fetch the URL and confirm that the claim appears in the content.
## The Finish Rule
An agentic loop must end. Decide in advance when enough evidence exists:
1. You found consistent answers across two or more sources.
2. Your retrieval budget is spent.
3. The question is answered with the required confidence.
Cap the loop. A common default is 20 steps for the search agent, with a plan re-check every 4 steps. If no source confirms a claim, say so. Do not smooth over the gap.
## Citations
- Smolagents ReAct guide: <https://huggingface.co/docs/smolagents/conceptual_guides/react>
- CodeAct paper: <https://arxiv.org/abs/2402.01030>
- Crawl4AI documentation: <https://docs.crawl4ai.com>
- Search result re-ranking, BM25 approach: <https://github.com/Oaklight/searxng-bm25-reranker>
- SearXNG outgoing settings: <https://docs.searxng.org/admin/settings/settings_outgoing.html>
- SearXNG limiter: <https://docs.searxng.org/admin/searx.limiter.html>
## See Also
- [Research Effectiveness](research-effectiveness.md) — Orchestrate the loop into a report.
- [SearXNG Tuning](searxng-tuning.md) — The backend behind the pipeline.
- [Self-Hosted Web Agents](self-hosted-web-agents.md) — The browser tools in the loop.
omp-docs/agentic-web-research/index.md
+39
View File
@@ -0,0 +1,39 @@
# Agentic Web Research
Agentic web research means that an AI drives the search loop. The agent formulates queries, reads results, decides what to read next, and writes a synthesis. It works best when search, extraction, and reasoning each run in the right place.
> **Your setup:** Pi Web Access calls SearXNG at `192.168.0.143:8081` (dankclaw) and serves models through llama.cpp at `192.168.0.235:7770`, `192.168.0.143:7770`, and `localhost:8080`. The pages below assume that topology.
This section connects four decisions:
1. **Search backend** — make a self-hosted [SearXNG](searxng-tuning.md) instance fast and private.
2. **Network path** — send only SearXNG traffic through a [VPN](vpn-routing.md) and leave the rest alone.
3. **Agent loop** — apply [agentic search best practices](agentic-search-best-practices.md) and [research effectiveness](research-effectiveness.md) so the loop ends, not spins.
4. **Model choice** — pick a self-hostable [open model](open-models.md) strong enough for tool use and [web agents](self-hosted-web-agents.md).
## Quick Map
```mermaid
graph LR
A[User Question] --> B[Agent + Open Model]
B --> C[SearXNG Search]
C -->|VPN only here| D[Upstream Engines]
B --> E[Extraction: Crawl4AI / fetch]
B --> F[Synthesis + Citations]
E --> F
```
## Pages
- [Self-Hosted Web Agents](self-hosted-web-agents.md) — Tools and architectures for an agent that drives a browser.
- [SearXNG Tuning](searxng-tuning.md) — Configuration for speed, privacy, and useful results.
- [VPN Routing for SearXNG](vpn-routing.md) — Route only the SearXNG outbound path through a VPN.
- [Agentic Search Best Practices](agentic-search-best-practices.md) — Query, extract, validate, and retry well.
- [Open Models for Web Use](open-models.md) — Self-hostable models strong at tool and API calling.
- [Research Effectiveness](research-effectiveness.md) — Multi-step, credibility-scored research loops.
- [Model Context Protocol](mcp.md) — The standard that connects the agent to tools.
## Sources
Authoritative references are cited per page under `# Citations`.
omp-docs/agentic-web-research/mcp.md
+95
View File
@@ -0,0 +1,95 @@
---
type: concept
title: Model Context Protocol (MCP) for Web Research
description: The standard way to connect an agent to search, fetch, and tool servers, and how it fits a self-hosted web research stack.
tags:
- mcp
- protocol
- tools
- agentic
- unverified
verified: 2026-08-28
timestamp: 2026-08-11
resource: https://modelcontextprotocol.io
---
# Model Context Protocol (MCP) for Web Research
Model Context Protocol (MCP) is the open standard for connecting an AI model to external tools and data. Use it to give your agent search, page fetching, and database access without wiring each provider by hand.
MCP uses a client-server model. The agent is an MCP client. Each capability runs in an MCP server. The agent calls the tools that the server exposes.
The Linux Foundation hosts the project. The spec lives at `modelcontextprotocol.io`.
## Transports
MCP servers use one of two current transports:
- `stdio` — the server runs as a child process. Good for local, single-client use.
- `Streamable HTTP` — a single HTTP POST endpoint. Good for remote or multi-client use.
The older `HTTP+SSE` transport still works for compatibility. Clients auto-detect it.
## The Model
```mermaid
graph LR
A[Agent / MCP client] --> B[Search server]
A --> C[Fetch server]
A --> D[Database server]
A --> E[Filesystem server]
```
## Official Reference Servers
The `modelcontextprotocol/servers` repository ships seven reference servers:
| Server | Purpose |
|--------|---------|
| Fetch | Fetch and convert a web page |
| Filesystem | Read and write local files |
| Git | Read and manipulate Git repositories |
| Memory | Knowledge-graph persistent memory |
| Sequential Thinking | Reflective problem solving |
| Time | Answer time-related queries |
| Everything | Test and demo server |
## Search Servers
- The official **Brave Search MCP** server (`brave-search-mcp-server`) exposes `brave_web_search` and related tools. It needs a Brave API key.
- Community **SearXNG MCP** servers connect any SearXNG instance to MCP. They work with your self-hosted instance and need no API key.
- **Crawl4AI** ships a built-in MCP bridge in its Docker image that turns crawls into MCP tools.
## Why It Matters Here
- SmolAgents loads MCP tools with `ToolCollection.from_mcp()`.
- CrewAI integrates MCP through `MCPServerStdio`, `MCPServerHTTP`, and `MCPServerSSE`.
- Browser Use exposes MCP via `com.browser-use`.
- Qwen3 does agentic tool calling through the Qwen-Agent framework.
With MCP, the search, fetch, and synthesis steps in this section share one connection standard. The agent does not depend on one vendor's format.
## Run It Securely
Follow these rules for a self-hosted MCP server:
- Bind to `127.0.0.1` for local use.
- Prefer `stdio` transport for a local agent.
- Use Streamable HTTP with auth only for remote use.
- Validate the `Origin` header on HTTP servers. A mismatch must return HTTP 403. A bad check lets a remote site attack a local server.
- Treat tool descriptions as untrusted. They can carry prompt injection.
## Citations
- MCP specification: <https://modelcontextprotocol.io>
- Official MCP reference servers: <https://github.com/modelcontextprotocol/servers>
- Brave Search MCP server: <https://github.com/brave/brave-search-mcp-server>
- SearXNG MCP server (community): <https://github.com/aicrafted/searxng-mcp>
- SmolAgents repository: <https://github.com/huggingface/smolagents>
- Crawl4AI repository: <https://github.com/unclecode/crawl4ai>
## See Also
- [Agentic Search Best Practices](agentic-search-best-practices.md) — The loop MCP connects.
- [Self-Hosted Web Agents](self-hosted-web-agents.md) — Tools that expose MCP.
- [Open Models for Web Use](open-models.md) — Models that drive these tools.
omp-docs/agentic-web-research/open-models.md
+90
View File
@@ -0,0 +1,90 @@
---
type: concept
title: Open Models for Web Use
description: Self-hostable, open-weight models that are strong at tool use, API calling, and internet-assisted tasks.
tags:
- open-models
- tool-use
- function-calling
- llm
- unverified
verified: 2026-08-28
timestamp: 2026-08-11
resource: https://qwenlm.github.io/blog/qwen3/
---
# Open Models for Web Use
The model is the brain behind the agent. For web research the model must call tools, follow instructions, and sometimes understand a page image. This page ranks the best self-hostable open-weight options.
> **Your setup:** Pi already serves local models through llama.cpp. See [Llama.cpp Server](../tools/llama-cpp/index.md) and [Pi Models Configuration](../pi/models.md). You can run any of the picks below on those hosts.
## The Front-Runner: Qwen3 (Alibaba, Apache 2.0)
Qwen3 is the current state of the art for open function calling and tool use. It has a thinking / non-thinking hybrid mode and strong tool-calling. Release date: April 2025.
| Model | Type | Active params | Context | Q4 VRAM |
|-------|------|---------------|---------|---------|
| Qwen3-8B | Dense | 8B | 128K | ~5 GB |
| Qwen3-14B | Dense | 14B | 128K | ~9 GB |
| Qwen3-32B | Dense | 32B | 128K | ~18 GB |
| Qwen3-30B-A3B | MoE | 3B | 128K | ~15 GB |
The 8B and 14B models run on one consumer GPU. The 30B-A3B MoE runs on a single A100-class card or equivalent.
## Best Vision Model: Qwen2.5-VL-7B
Many web tasks need a model that can read a page image, not just text. Qwen2.5-VL-7B is the best open vision model for this. It reads text, charts, icons, and layout in screenshots. It supports computer-use and phone-use agents. Apache 2.0 license. The 7B model fits on a single GPU with 8 GB or more.
Llama 4 Scout is the larger multimodal option. Verify the claims before you trust them:
- "10 million token context" — true, but **Scout only**, not Maverick.
- "Fits on a single H100" — only true with INT4 quantization (about 48 GB).
- "Beats GPT-4o" — Meta's own release benchmarks, not independent runs.
## Choose by Hardware
```mermaid
graph TD
A[Consumer GPU 16-24GB] --> B[Qwen3-8B / 14B]
A --> C[Qwen2.5-VL-7B]
D[Single server GPU 80GB] --> E[Qwen3-30B-A3B]
D --> F[Llama 4 Scout INT4]
G[Multi-GPU cluster] --> H[Qwen3-235B-A22B]
```
## Serving Options
- **llama.cpp** — CPU/GPU inference with GGUF quantization. Best for one consumer GPU. This is what Pi uses.
- **Ollama** — Easy local serving. Does not run in the Pi setup today.
- **vLLM** — High-throughput serving. Best for server and multi-GPU MoE models.
- **SGLang** — Strong for MoE serving.
```bash
# On the Pi llama.cpp host
# Serve a downloaded GGUF with the OpenAI-compatible API
llama-server -m qwen3-8b-q4_k_m.gguf -c 131072 --port 7770
```
## The Trade-Off
| Need | Pick |
|------|------|
| Best tool calling, one consumer GPU | Qwen3-8B or 14B |
| Visually read a web page | Qwen2.5-VL-7B |
| Server-scale reliable agent | Qwen3-235B-A22B |
| Cheapest acceptable agent | Qwen3-8B on a mid GPU |
## Citations
- Qwen3 technical report: <https://qwenlm.github.io/blog/qwen3/>
- Qwen3 repository: <https://github.com/QwenLM/Qwen3>
- Qwen2.5-VL-7B model card: <https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct>
- Meta AI Llama 4 announcement: <https://ai.meta.com/blog/llama-4-multimodal-intelligence/>
- Ollama library: <https://ollama.com/library>
- Browser Use supported models: <https://docs.browser-use.com/open-source/supported-models>
## See Also
- [Self-Hosted Web Agents](self-hosted-web-agents.md) — The tools these models drive.
- [Model Context Protocol (MCP)](mcp.md) — Connect tools to the model.
omp-docs/agentic-web-research/research-effectiveness.md
+130
View File
@@ -0,0 +1,130 @@
---
type: concept
title: Research Effectiveness
description: Orchestrate a multi-step research loop — roles, credibility scoring, and when to stop.
tags:
- agentic
- research
- orchestration
- credibility
- unverified
verified: 2026-08-28
timestamp: 2026-08-11
---
# Research Effectiveness
One model call does not make good research. A research pipeline assigns roles, scores sources, and stops when it has enough. This page explains how to build one.
## Add Structure With Roles
Split the work into roles. Each role has one job.
```mermaid
graph LR
A[Researcher] --> B[Analyst]
B --> C[Writer]
C --> D[Critic]
D -->|gaps found| A
D -->|done| E[Final report]
```
| Role | Job |
|------|-----|
| Researcher | Search, extract pages, rank sources |
| Analyst | Extract facts, evaluate credibility |
| Writer | Synthesize and write the report |
| Critic | Find gaps and request follow-up |
## Use a Manager and a Search Agent
The strongest open recipe is a two-tier design:
- A **manager** agent reasons and plans. It uses code actions and a plan that it re-checks every few steps.
- A **search agent** does the web browsing. It has its own bounded tool set and step cap.
The search agent holds the browser tools: search, visit page, find on page, inspect files, and archive fallback. The manager reads results, decides what to follow next, and asks for a final answer when done.
This pattern powers open deep-research systems. The open reproduction scores about 55 percent on GAIA, below the closed OpenAI version at 67 percent, but it runs fully on your own stack.
## Score Source Credibility
Score each source on a small set of signals:
- Domain authority.
- Content freshness.
- Source diversity (independent sources agree).
- Whether other sources confirm the claim.
Cross-reference each fact. If two independent sources agree, the fact is stronger.
## Keep Context Small
Never pass a full page to the model.
- Convert HTML to Markdown and strip scripts and navigation.
- Split long pages into viewports and show one chunk at a time.
- Cap file text at a fixed size, for example 100,000 characters.
- Filter content before the model sees it.
Smaller context means a faster loop and a lower chance of confusion.
## Stop Hallucinated Citations
An agent can invent a source. Guard against it:
1. The agent returns answers with structured citations.
2. The manager re-fetches each cited URL.
3. If a citation fails, the agent re-searches with a modified query.
Use a web archive as a fallback for dead links.
## Adapt the Search
Do not follow every link. Follow the links that the question needs. Stop when the confidence threshold passes. Tools such as Crawl4AI expose an adaptive crawler that reports its own confidence.
## Choose an Orchestration Tool
| Tool | Best for | Notes |
|------|----------|-------|
| CrewAI | Role-based multi-agent work | Sequential or hierarchical processes |
| LangGraph | Stateful, long-running research | Durable execution, human in the loop |
| SmolAgents | Simple code-based agents | MCP support, manager + search agent |
| Qwen-Agent | Qwen model + MCP tools | Best integration with Qwen3 |
Pi already runs a similar model with its parallel `researcher` and `scout` subagents.
## The Finish Rule
Research must end. Stop when any one is true:
1. Two or more independent sources agree on the answer.
2. The retrieval budget is spent.
3. The required confidence is met for every claim.
Report gaps honestly. Do not invent a source to fill one.
## Failure Modes and Guardrails
Agentic loops fail in predictable ways. Add guardrails up front.
| Failure | Guardrail |
|---------|-----------|
| Hallucinated citation | Cite structured objects, then re-fetch each URL |
| Search engine block | Respect retry-after, use adaptive rate limiting, read the ban time |
| Infinite re-search | Cap search rounds, reject near-duplicate queries, re-plan every few steps |
| Context overflow | Filter content before the model sees it, track the token budget |
## Citations
- Smolagents open deep research: <https://huggingface.co/blog/open-deep-research>
- Smolagents multiagents guide: <https://huggingface.co/docs/smolagents/examples/multiagents>
- CrewAI repository: <https://github.com/crewAIInc/crewAI>
- CrewAI documentation: <https://docs.crewai.com/>
- LangGraph repository: <https://github.com/langchain-ai/langgraph>
- Crawl4AI documentation: <https://docs.crawl4ai.com>
## See Also
- [Agentic Search Best Practices](agentic-search-best-practices.md) — The loop this orchestrates.
- [Open Models for Web Use](open-models.md) — The model that runs each role.
omp-docs/agentic-web-research/searxng-tuning.md
+194
View File
@@ -0,0 +1,194 @@
---
type: concept
title: SearXNG Tuning
description: Configuration decisions for a fast, private, useful self-hosted SearXNG instance.
tags:
- searxng
- search
- privacy
- configuration
- unverified
verified: 2026-08-28
timestamp: 2026-08-11
resource: https://docs.searxng.org
---
# SearXNG Tuning
SearXNG is a metasearch engine. It aggregates 274 or more upstream search services into one result set. Configuration lives in `settings.yml`. The recommended deployment is Docker Compose with a Valkey container.
This page lists the settings that matter for speed, privacy, and useful results.
> **Your setup:** Pi Web Access calls a remote SearXNG at [`192.168.0.143:8081`](../pi/packages/pi-web-access.md) (dankclaw). Apply these settings to that host, not to this machine.
## Plan the Deployment
```mermaid
graph LR
A[Browser or Agent] --> B[SearXNG container]
B --> C[Valkey cache + limiter]
B --> D[Engine request pool]
D --> E[Google News]
D --> F[DuckDuckGo]
D --> G[Wikipedia]
```
Deploy with the official Docker Compose file. It ships a `searxng-valkey` container (`valkey/valkey:9-alpine`). Valkey replaced Redis as the default cache. Mount your config at `./core-config/:/etc/searxng`.
```yaml
services:
searxng:
image: docker.io/searxng/searxng:latest
ports:
- "8081:8080"
volumes:
- ./core-config/:/etc/searxng/:Z
depends_on:
- searxng-valkey
searxng-valkey:
image: docker.io/valkey/valkey:9-alpine
restart: always
```
## Core Settings
```yaml
use_default_settings: true
server:
secret_key: "change-this" # or $SEARXNG_SECRET
base_url: "http://192.168.0.143:8081/" # or $SEARXNG_BASE_URL
limiter: true
method: "POST"
search:
default_lang: "en"
default_locale: "en-US"
safe_search: 0
formats:
- html
- json
```
Change `secret_key`. Set `base_url` to the public address. Use `POST` for privacy. Enable the `json` format for the agent.
## Rate Limiting
Enable the limiter and point it at Valkey. Without a limiter, an upstream engine can block SearXNG's IP.
```yaml
server:
limiter: true
valkey:
url: valkey://searxng-valkey:6379/0
```
The limiter uses a sliding window per IP:
| Window | HTML limit | JSON/API limit |
|--------|-----------|----------------|
| 20 seconds | 15 | 4 per 3600 s |
| 600 seconds | 150 | 4 per 3600 s |
Suspicious IPs get a much lower budget (2 in 20 s, 10 in 600 s). The agent that hits the JSON endpoint repeatedly can trip the 4-per-hour API limit. Back off or make the agent part of a trusted group.
## Outgoing Pool
```yaml
outgoing:
request_timeout: 3.0
max_request_timeout: 10.0
pool_connections: 100
pool_maxsize: 20
enable_http2: true
keepalive_expiry: 5.0
retries: 3
```
A short `request_timeout` keeps results snappy. `enable_http2` multiplexes connections. More `retries` makes results more reliable but slower.
## Engine Selection
Default engine state (current SearXNG):
| Engine | Default state | Note |
|--------|---------------|------|
| DuckDuckGo | Enabled | Reliable general search |
| Wikipedia | Enabled | Encyclopedia lookups |
| Google News | Enabled | News results |
| Google (main) | Inactive | Cloudflare CAPTCHA often blocks it |
| Bing | Disabled | Blocked upstream |
Keep only the engines you need. Fewer engines means faster responses.
```yaml
use_default_settings:
engines:
keep_only:
- duckduckgo
- wikipedia
- google news
```
Make an engine private to one agent with a token:
```yaml
engines:
- name: arch wiki
engine: archlinux
tokens: ['my-secret-token']
```
## Useful Plugins
```yaml
plugins:
searx.plugins.unit_converter.SXNGPlugin:
active: true
searx.plugins.calculator.SXNGPlugin:
active: true
searx.plugins.self_info.SXNGPlugin:
active: true
```
The hostnames plugin filters or rewrites results. It needs a separate `hostnames:` block:
```yaml
hostnames:
remove:
- '(.*\.)?facebook.com$'
high_priority:
- '(.*\.)?wikipedia.org$'
```
## JSON API for the Agent
The agent uses the JSON format. Confirm that it returns structured data:
```bash
curl 'http://192.168.0.143:8081/search?q=test&format=json'
```
The response contains `results`, `query`, `number_of_results`, and `suggestions`. The `/config` endpoint lists engine availability.
## Verify the Instance
1. Open the HTML interface and run a test query.
2. Confirm that the `json` format returns data.
3. Confirm that the limiter page appears on rapid repeats.
## Citations
- SearXNG documentation: <https://docs.searxng.org>
- SearXNG outgoing settings: <https://docs.searxng.org/admin/settings/settings_outgoing.html>
- SearXNG engine settings: <https://docs.searxng.org/admin/settings/settings_engines.html>
- SearXNG Valkey settings: <https://docs.searxng.org/admin/settings/settings_valkey.html>
- SearXNG limiter: <https://docs.searxng.org/admin/searx.limiter.html>
- SearXNG plugins: <https://docs.searxng.org/admin/settings/settings_plugins.html>
- Docker Compose install: <https://docs.searxng.org/admin/installation-docker.html>
## See Also
- [VPN Routing for SearXNG](vpn-routing.md) — Route the outbound path through a VPN.
- [Agentic Search Best Practices](agentic-search-best-practices.md) — Use the JSON format well.
omp-docs/agentic-web-research/self-hosted-web-agents.md
+106
View File
@@ -0,0 +1,106 @@
---
type: concept
title: Self-Hosted Web Agents
description: Tools and architectures that let an AI agent drive a real browser, self-hosted.
tags:
- agentic
- browser
- scraping
- web-agent
- unverified
verified: 2026-08-28
---
# Self-Hosted Web Agents
A web agent is an AI that controls a browser. It sees the page, chooses an action, and repeats until it completes the task. You can run the whole loop on your own hardware.
This page covers the leading open tools and the architecture the agent uses.
## The Main Tools
### Browser Use
Browser Use is a Python library that uses Playwright under the hood. It is MIT-licensed. It supports many model providers, including local OpenAI-compatible endpoints such as llama.cpp.
Key ideas:
- It snapshots the DOM and converts it to Markdown for the model.
- Ten watchdogs handle CAPTCHAs, crashes, popups, downloads, and security.
- Real browser profiles carry saved logins.
**On the "87.4 percent" number:** that result refers to Browser Use's managed Cloud agent (the Hybrid type that blends structured browser tools with screenshots). Peer-reviewed results for vision-only (CUA) models are much lower, about 44.5 percent on the same task type. Treat any single accuracy number as one benchmark, not a universal score.
```python
# Local model via llama.cpp (OpenAI-compatible endpoint)
from browser_use import Agent
from browser_use.llm.openai import ChatOpenAI
agent = Agent(
task="Search for the latest AI news",
llm=ChatOpenAI(
base_url="http://192.168.0.235:7770/v1",
model="frank-default-small",
api_key="EMPTY",
),
)
await agent.run()
```
### Skyvern
Skyvern is an open-source browser agent (AGPL-3.0). It uses a vision model to see the page and Playwright to act. It extends Playwright with `page.act()`, `page.extract()`, and `page.validate()`. It ships a Docker Compose setup. Use it when you need a vision-first agent.
### Crawl4AI
Crawl4AI is an async crawler that returns LLM-friendly Markdown. It is built for retrieval pipelines, not for clicking through interactive pages. Use it to fetch and clean many pages quickly.
Key ideas:
- It produces clean Markdown and a noise-filtered "fit Markdown."
- It supports deep crawling with breadth-first, depth-first, and best-first strategies.
- It can run in Docker with a REST API and MCP endpoints.
```bash
pip install -U crawl4ai
crawl4ai-setup
crwl https://example.com --deep-crawl bfs --max-pages 10
```
## The Loop
```mermaid
graph TD
A[Task] --> B[Model chooses action]
B --> C[Browser or Crawler acts]
C --> D[Extract text / Markdown]
D --> B
D --> E[Task done?]
E -->|No| B
E -->|Yes| F[Return answer + citations]
```
## Which Tool to Use
| Situation | Tool |
|-----------|------|
| Need to click, log in, or fill forms | Browser Use |
| Need vision-based browsing | Skyvern |
| Need to fetch and clean many static pages | Crawl4AI |
| Research pipeline feeds an agent | Crawl4AI first, Browser Use for interactive steps |
A hybrid works best. Use Crawl4AI for mass extraction and Browser Use for the interactive steps.
## Citations
- Browser Use repository: <https://github.com/browser-use/browser-use>
- Browser Use documentation: <https://docs.browser-use.com>
- Skyvern repository: <https://github.com/Skyvern-AI/skyvern>
- Crawl4AI repository: <https://github.com/unclecode/crawl4ai>
- Crawl4AI documentation: <https://docs.crawl4ai.com>
## See Also
- [Agentic Search Best Practices](agentic-search-best-practices.md) — The loop around the tools.
- [Open Models for Web Use](open-models.md) — Models that drive these tools.
- [Research Effectiveness](research-effectiveness.md) — Orchestration around the loop.
omp-docs/agentic-web-research/vpn-routing.md
+189
View File
@@ -0,0 +1,189 @@
---
type: concept
title: VPN Routing for SearXNG
description: Send only SearXNG outbound requests through a VPN while leaving all other local traffic alone.
tags:
- vpn
- searxng
- networking
- proxy
- unverified
verified: 2026-08-28
timestamp: 2026-08-19
resource: https://docs.searxng.org/admin/settings/settings_outgoing.html
---
# VPN Routing for SearXNG
SearXNG talks to many upstream engines. Some of them rate-limit or block the source IP. You can send only that outbound path through a VPN and leave every other process on the host untouched.
This page gives four working patterns. Choose the one that matches how you run SearXNG.
> **Your setup:** SearXNG runs on another host at `192.168.0.143:8081` (dankclaw). Apply the Docker patterns on that host. The proxy pattern works from any host.
## The Goal
```mermaid
graph LR
A[Local agent] --> B[SearXNG]
B -->|through VPN| C[Upstream engines]
D[Other local traffic] --> E[Plain internet]
```
The VPN carries only the SearXNG outbound path. Nothing else changes.
## Pattern 1: SearXNG Proxy Setting
SearXNG supports HTTP and SOCKS proxies natively. Point the outbound settings at a local proxy that exits through the VPN.
```yaml
outgoing:
proxies:
all://:
- socks5://vpn-proxy:1080
extra_proxy_timeout: 10.0
```
Apply the proxy to one engine only:
```yaml
engines:
- name: google
engine: google
proxies:
https:
- socks5://vpn-proxy:1080
```
To force the VPN source IP on a multi-homed host, add `source_ips` and pin the engine to IPv4:
```yaml
outgoing:
source_ips:
- 10.8.0.2
engines:
- name: my-search
engine: duckduckgo
network: ipv4
```
## Pattern 2: Gluetun (Recommended)
Gluetun is a VPN container (WireGuard or OpenVPN). Attach SearXNG to its network namespace. Only SearXNG shares the tunnel. Other containers stay on the default network.
```yaml
services:
gluetun:
image: qmcgaw/gluetun:latest
cap_add: [NET_ADMIN]
devices: [/dev/net/tun:/dev/net/tun]
ports:
- "8081:8080"
environment:
VPN_SERVICE_PROVIDER: mullvad
VPN_TYPE: wireguard
WIREGUARD_PRIVATE_KEY: <key>
restart: always
searxng:
image: searxng/searxng:latest
network_mode: service:gluetun
depends_on:
- gluetun
volumes:
- ./core-config:/etc/searxng
restart: unless-stopped
```
Put the port mapping on the `gluetun` service. The `searxng` container does not get its own ports.
## Pattern 3: Dedicated Internal Network
Create a network that only SearXNG shares with a lighter VPN client.
```yaml
services:
searxng:
image: searxng/searxng:latest
networks:
- searxng-vpn
volumes:
- ./core-config:/etc/searxng
vpn-client:
image: dperson/openvpn-client:latest
networks:
- searxng-vpn
cap_add:
- NET_ADMIN
networks:
searxng-vpn:
internal: true
```
An OpenVPN client container is simpler than Gluetun when your provider uses OpenVPN.
## Pattern 4: Run SearXNG Inside the VPN Container
```yaml
services:
searxng:
image: searxng/searxng:latest
network_mode: "container:vpn-client"
depends_on:
- vpn-client
volumes:
- ./core-config:/etc/searxng
vpn-client:
image: dperson/openvpn-client:latest
cap_add: [NET_ADMIN]
environment:
VPN: server.address;username;password
restart: always
```
This pattern places the whole SearXNG process inside the tunnel namespace. The host sees the tunnel as the only route for SearXNG.
## Compare the Patterns
| Pattern | Complexity | Isolation | Per-engine control |
|---------|-----------|-----------|--------------------|
| Proxy setting | Low | Medium | Yes |
| Gluetun | Medium | High | No |
| Dedicated network | Medium | High | No |
| Inside VPN container | Medium | Highest | No |
## Decisions That Matter
- A SOCKS proxy keeps the per-engine control. Use Pattern 1.
- Gluetun gives the cleanest full-tunnel setup. Use Pattern 2.
- Mullvad ships its own WireGuard keys for Gluetun.
- An OpenVPN-only provider works with Pattern 3 or 4.
## Verify the Route
1. Run the instance with the VPN active.
2. Query an IP-echo page through SearXNG.
3. Confirm that the returned IP differs from your host IP.
4. Confirm that unrelated host traffic keeps the host IP.
```bash
# From inside the tunnel with Gluetun
docker exec gluetun curl -s ifconfig.me
```
## Citations
- SearXNG outgoing settings: <https://docs.searxng.org/admin/settings/settings_outgoing.html>
- SearXNG engine settings: <https://docs.searxng.org/admin/settings/settings_engines.html>
- Gluetun: <https://github.com/qdm12/gluetun>
- Connect a container to Gluetun: <https://github.com/qdm12/gluetun-wiki/blob/main/setup/connect-a-container-to-gluetun.md>
- Docker Compose networking: <https://docs.docker.com/compose/networking/>
## See Also
- [SearXNG Tuning](searxng-tuning.md) — The instance these patterns protect.
- [Agentic Search Best Practices](agentic-search-best-practices.md) — Proxy rotation and retries.