docs: rebuild KB in OKF v0.1 with live-verified facts
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
This commit is contained in:
Frank committed
1 parent
552d1ef8bf
commit
02883d3ed2
116 files changed
+13008
No files matched your search
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: concept
|
||||
title: Research Effectiveness
|
||||
description: Orchestrate a multi-step research loop — roles, credibility scoring, and when to stop.
|
||||
tags:
|
||||
- agentic
|
||||
- research
|
||||
- orchestration
|
||||
- credibility
|
||||
- unverified
|
||||
verified: 2026-08-28
|
||||
timestamp: 2026-08-11
|
||||
---
|
||||
|
||||
# Research Effectiveness
|
||||
|
||||
One model call does not make good research. A research pipeline assigns roles, scores sources, and stops when it has enough. This page explains how to build one.
|
||||
|
||||
## Add Structure With Roles
|
||||
|
||||
Split the work into roles. Each role has one job.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Researcher] --> B[Analyst]
|
||||
B --> C[Writer]
|
||||
C --> D[Critic]
|
||||
D -->|gaps found| A
|
||||
D -->|done| E[Final report]
|
||||
```
|
||||
|
||||
| Role | Job |
|
||||
|------|-----|
|
||||
| Researcher | Search, extract pages, rank sources |
|
||||
| Analyst | Extract facts, evaluate credibility |
|
||||
| Writer | Synthesize and write the report |
|
||||
| Critic | Find gaps and request follow-up |
|
||||
|
||||
## Use a Manager and a Search Agent
|
||||
|
||||
The strongest open recipe is a two-tier design:
|
||||
|
||||
- A **manager** agent reasons and plans. It uses code actions and a plan that it re-checks every few steps.
|
||||
- A **search agent** does the web browsing. It has its own bounded tool set and step cap.
|
||||
|
||||
The search agent holds the browser tools: search, visit page, find on page, inspect files, and archive fallback. The manager reads results, decides what to follow next, and asks for a final answer when done.
|
||||
|
||||
This pattern powers open deep-research systems. The open reproduction scores about 55 percent on GAIA, below the closed OpenAI version at 67 percent, but it runs fully on your own stack.
|
||||
|
||||
## Score Source Credibility
|
||||
|
||||
Score each source on a small set of signals:
|
||||
|
||||
- Domain authority.
|
||||
- Content freshness.
|
||||
- Source diversity (independent sources agree).
|
||||
- Whether other sources confirm the claim.
|
||||
|
||||
Cross-reference each fact. If two independent sources agree, the fact is stronger.
|
||||
|
||||
## Keep Context Small
|
||||
|
||||
Never pass a full page to the model.
|
||||
|
||||
- Convert HTML to Markdown and strip scripts and navigation.
|
||||
- Split long pages into viewports and show one chunk at a time.
|
||||
- Cap file text at a fixed size, for example 100,000 characters.
|
||||
- Filter content before the model sees it.
|
||||
|
||||
Smaller context means a faster loop and a lower chance of confusion.
|
||||
|
||||
## Stop Hallucinated Citations
|
||||
|
||||
An agent can invent a source. Guard against it:
|
||||
|
||||
1. The agent returns answers with structured citations.
|
||||
2. The manager re-fetches each cited URL.
|
||||
3. If a citation fails, the agent re-searches with a modified query.
|
||||
|
||||
Use a web archive as a fallback for dead links.
|
||||
|
||||
## Adapt the Search
|
||||
|
||||
Do not follow every link. Follow the links that the question needs. Stop when the confidence threshold passes. Tools such as Crawl4AI expose an adaptive crawler that reports its own confidence.
|
||||
|
||||
## Choose an Orchestration Tool
|
||||
|
||||
| Tool | Best for | Notes |
|
||||
|------|----------|-------|
|
||||
| CrewAI | Role-based multi-agent work | Sequential or hierarchical processes |
|
||||
| LangGraph | Stateful, long-running research | Durable execution, human in the loop |
|
||||
| SmolAgents | Simple code-based agents | MCP support, manager + search agent |
|
||||
| Qwen-Agent | Qwen model + MCP tools | Best integration with Qwen3 |
|
||||
|
||||
Pi already runs a similar model with its parallel `researcher` and `scout` subagents.
|
||||
|
||||
## The Finish Rule
|
||||
|
||||
Research must end. Stop when any one is true:
|
||||
|
||||
1. Two or more independent sources agree on the answer.
|
||||
2. The retrieval budget is spent.
|
||||
3. The required confidence is met for every claim.
|
||||
|
||||
Report gaps honestly. Do not invent a source to fill one.
|
||||
|
||||
## Failure Modes and Guardrails
|
||||
|
||||
Agentic loops fail in predictable ways. Add guardrails up front.
|
||||
|
||||
| Failure | Guardrail |
|
||||
|---------|-----------|
|
||||
| Hallucinated citation | Cite structured objects, then re-fetch each URL |
|
||||
| Search engine block | Respect retry-after, use adaptive rate limiting, read the ban time |
|
||||
| Infinite re-search | Cap search rounds, reject near-duplicate queries, re-plan every few steps |
|
||||
| Context overflow | Filter content before the model sees it, track the token budget |
|
||||
|
||||
## Citations
|
||||
|
||||
- Smolagents open deep research: <https://huggingface.co/blog/open-deep-research>
|
||||
- Smolagents multiagents guide: <https://huggingface.co/docs/smolagents/examples/multiagents>
|
||||
- CrewAI repository: <https://github.com/crewAIInc/crewAI>
|
||||
- CrewAI documentation: <https://docs.crewai.com/>
|
||||
- LangGraph repository: <https://github.com/langchain-ai/langgraph>
|
||||
- Crawl4AI documentation: <https://docs.crawl4ai.com>
|
||||
|
||||
## See Also
|
||||
|
||||
- [Agentic Search Best Practices](agentic-search-best-practices.md) — The loop this orchestrates.
|
||||
- [Open Models for Web Use](open-models.md) — The model that runs each role.
|
||||
Reference in new issue
Block a user