Frank
8b87b5ae1a
Two independent copies of the same law, and each page cited its own numbers. The finish rule — stop when independent sources agree, when the retrieval budget is spent, or when every claim is met; report gaps, never invent a source — stood in full in `agentic-search-best-practices.md` (with the loop caps: 20 steps, re-plan every 4) and again in `research-effectiveness.md` without them. The operational page owns it; the measurement page keeps the pointer and the one thing that is its business, which is that unbounded loops are why "infinite re-search" is a named failure mode. `tools/gitea/index.md` was a second copy of its own directory: an architecture table, a management command table, a data-persistence path table and a boot-persistence paragraph, every one of them a subset of `architecture.md`, `management.md` and `data-persistence.md` — verified line by line before cutting, including the two facts that looked unique to the index (the `dns_enabled: false` default network, the `/opt/gitea` leftover) and turn out to be owned upstream with more detail. The index now answers "which page owns my question" and gives three one-line orientation facts. Gitea's live version was re-read while I was in there: still 1.27.3. Refs #64
4.8 KiB
type, title, description, tags, verified, timestamp
| type | title | description | tags | verified | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| concept | Research Effectiveness | Orchestrate a multi-step research loop — roles, credibility scoring, and when to stop. |
|
2026-08-28 | 2026-08-11 |
Research Effectiveness
One model call does not make good research. A research pipeline assigns roles, scores sources, and stops when it has enough. This page explains how to build one.
Add Structure With Roles
Split the work into roles. Each role has one job.
graph LR
A[Researcher] --> B[Analyst]
B --> C[Writer]
C --> D[Critic]
D -->|gaps found| A
D -->|done| E[Final report]
| Role | Job |
|---|---|
| Researcher | Search, extract pages, rank sources |
| Analyst | Extract facts, evaluate credibility |
| Writer | Synthesize and write the report |
| Critic | Find gaps and request follow-up |
Use a Manager and a Search Agent
The strongest open recipe is a two-tier design:
- A manager agent reasons and plans. It uses code actions and a plan that it re-checks every few steps.
- A search agent does the web browsing. It has its own bounded tool set and step cap.
The search agent holds the browser tools: search, visit page, find on page, inspect files, and archive fallback. The manager reads results, decides what to follow next, and asks for a final answer when done.
This pattern powers open deep-research systems. The open reproduction scores about 55 percent on GAIA, below the closed OpenAI version at 67 percent, but it runs fully on your own stack.
Score Source Credibility
Score each source on a small set of signals:
- Domain authority.
- Content freshness.
- Source diversity (independent sources agree).
- Whether other sources confirm the claim.
Cross-reference each fact. If two independent sources agree, the fact is stronger.
Keep Context Small
Never pass a full page to the model.
- Convert HTML to Markdown and strip scripts and navigation.
- Split long pages into viewports and show one chunk at a time.
- Cap file text at a fixed size, for example 100,000 characters.
- Filter content before the model sees it.
Smaller context means a faster loop and a lower chance of confusion.
Stop Hallucinated Citations
An agent can invent a source. Guard against it:
- The agent returns answers with structured citations.
- The manager re-fetches each cited URL.
- If a citation fails, the agent re-searches with a modified query.
Use a web archive as a fallback for dead links.
Adapt the Search
Do not follow every link. Follow the links that the question needs. Stop when the confidence threshold passes. Tools such as Crawl4AI expose an adaptive crawler that reports its own confidence.
Choose an Orchestration Tool
| Tool | Best for | Notes |
|---|---|---|
| CrewAI | Role-based multi-agent work | Sequential or hierarchical processes |
| LangGraph | Stateful, long-running research | Durable execution, human in the loop |
| SmolAgents | Simple code-based agents | MCP support, manager + search agent |
| Qwen-Agent | Qwen model + MCP tools | Best integration with Qwen3 |
Pi already runs a similar model with its parallel researcher and scout subagents.
The Finish Rule
The rule itself — stop when independent sources agree, when the retrieval budget is spent, or when every claim has the required confidence, and report the gaps instead of inventing a source — is owned by Agentic Search Best Practices, the page an agent reads before it starts a loop. What belongs here is the measurement: unbounded loops are why "infinite re-search" is a named failure mode in the next section.
Failure Modes and Guardrails
Agentic loops fail in predictable ways. Add guardrails up front.
| Failure | Guardrail |
|---|---|
| Hallucinated citation | Cite structured objects, then re-fetch each URL |
| Search engine block | Respect retry-after, use adaptive rate limiting, read the ban time |
| Infinite re-search | Cap search rounds, reject near-duplicate queries, re-plan every few steps |
| Context overflow | Filter content before the model sees it, track the token budget |
Citations
- Smolagents open deep research: https://huggingface.co/blog/open-deep-research
- Smolagents multiagents guide: https://huggingface.co/docs/smolagents/examples/multiagents
- CrewAI repository: https://github.com/crewAIInc/crewAI
- CrewAI documentation: https://docs.crewai.com/
- LangGraph repository: https://github.com/langchain-ai/langgraph
- Crawl4AI documentation: https://docs.crawl4ai.com
See Also
- Agentic Search Best Practices — The loop this orchestrates.
- Open Models for Web Use — The model that runs each role.