Frank
02883d3ed2
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
131 lines
4.5 KiB
Markdown
131 lines
4.5 KiB
Markdown
---
|
|
type: concept
|
|
title: Research Effectiveness
|
|
description: Orchestrate a multi-step research loop — roles, credibility scoring, and when to stop.
|
|
tags:
|
|
- agentic
|
|
- research
|
|
- orchestration
|
|
- credibility
|
|
- unverified
|
|
verified: 2026-08-28
|
|
timestamp: 2026-08-11
|
|
---
|
|
|
|
# Research Effectiveness
|
|
|
|
One model call does not make good research. A research pipeline assigns roles, scores sources, and stops when it has enough. This page explains how to build one.
|
|
|
|
## Add Structure With Roles
|
|
|
|
Split the work into roles. Each role has one job.
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Researcher] --> B[Analyst]
|
|
B --> C[Writer]
|
|
C --> D[Critic]
|
|
D -->|gaps found| A
|
|
D -->|done| E[Final report]
|
|
```
|
|
|
|
| Role | Job |
|
|
|------|-----|
|
|
| Researcher | Search, extract pages, rank sources |
|
|
| Analyst | Extract facts, evaluate credibility |
|
|
| Writer | Synthesize and write the report |
|
|
| Critic | Find gaps and request follow-up |
|
|
|
|
## Use a Manager and a Search Agent
|
|
|
|
The strongest open recipe is a two-tier design:
|
|
|
|
- A **manager** agent reasons and plans. It uses code actions and a plan that it re-checks every few steps.
|
|
- A **search agent** does the web browsing. It has its own bounded tool set and step cap.
|
|
|
|
The search agent holds the browser tools: search, visit page, find on page, inspect files, and archive fallback. The manager reads results, decides what to follow next, and asks for a final answer when done.
|
|
|
|
This pattern powers open deep-research systems. The open reproduction scores about 55 percent on GAIA, below the closed OpenAI version at 67 percent, but it runs fully on your own stack.
|
|
|
|
## Score Source Credibility
|
|
|
|
Score each source on a small set of signals:
|
|
|
|
- Domain authority.
|
|
- Content freshness.
|
|
- Source diversity (independent sources agree).
|
|
- Whether other sources confirm the claim.
|
|
|
|
Cross-reference each fact. If two independent sources agree, the fact is stronger.
|
|
|
|
## Keep Context Small
|
|
|
|
Never pass a full page to the model.
|
|
|
|
- Convert HTML to Markdown and strip scripts and navigation.
|
|
- Split long pages into viewports and show one chunk at a time.
|
|
- Cap file text at a fixed size, for example 100,000 characters.
|
|
- Filter content before the model sees it.
|
|
|
|
Smaller context means a faster loop and a lower chance of confusion.
|
|
|
|
## Stop Hallucinated Citations
|
|
|
|
An agent can invent a source. Guard against it:
|
|
|
|
1. The agent returns answers with structured citations.
|
|
2. The manager re-fetches each cited URL.
|
|
3. If a citation fails, the agent re-searches with a modified query.
|
|
|
|
Use a web archive as a fallback for dead links.
|
|
|
|
## Adapt the Search
|
|
|
|
Do not follow every link. Follow the links that the question needs. Stop when the confidence threshold passes. Tools such as Crawl4AI expose an adaptive crawler that reports its own confidence.
|
|
|
|
## Choose an Orchestration Tool
|
|
|
|
| Tool | Best for | Notes |
|
|
|------|----------|-------|
|
|
| CrewAI | Role-based multi-agent work | Sequential or hierarchical processes |
|
|
| LangGraph | Stateful, long-running research | Durable execution, human in the loop |
|
|
| SmolAgents | Simple code-based agents | MCP support, manager + search agent |
|
|
| Qwen-Agent | Qwen model + MCP tools | Best integration with Qwen3 |
|
|
|
|
Pi already runs a similar model with its parallel `researcher` and `scout` subagents.
|
|
|
|
## The Finish Rule
|
|
|
|
Research must end. Stop when any one is true:
|
|
|
|
1. Two or more independent sources agree on the answer.
|
|
2. The retrieval budget is spent.
|
|
3. The required confidence is met for every claim.
|
|
|
|
Report gaps honestly. Do not invent a source to fill one.
|
|
|
|
## Failure Modes and Guardrails
|
|
|
|
Agentic loops fail in predictable ways. Add guardrails up front.
|
|
|
|
| Failure | Guardrail |
|
|
|---------|-----------|
|
|
| Hallucinated citation | Cite structured objects, then re-fetch each URL |
|
|
| Search engine block | Respect retry-after, use adaptive rate limiting, read the ban time |
|
|
| Infinite re-search | Cap search rounds, reject near-duplicate queries, re-plan every few steps |
|
|
| Context overflow | Filter content before the model sees it, track the token budget |
|
|
|
|
## Citations
|
|
|
|
- Smolagents open deep research: <https://huggingface.co/blog/open-deep-research>
|
|
- Smolagents multiagents guide: <https://huggingface.co/docs/smolagents/examples/multiagents>
|
|
- CrewAI repository: <https://github.com/crewAIInc/crewAI>
|
|
- CrewAI documentation: <https://docs.crewai.com/>
|
|
- LangGraph repository: <https://github.com/langchain-ai/langgraph>
|
|
- Crawl4AI documentation: <https://docs.crawl4ai.com>
|
|
|
|
## See Also
|
|
|
|
- [Agentic Search Best Practices](agentic-search-best-practices.md) — The loop this orchestrates.
|
|
- [Open Models for Web Use](open-models.md) — The model that runs each role.
|