command-center/omp-docs/agentic-web-research/agentic-search-best-practices.md

type, title, description, tags, verified, timestamp

type	title	description	tags	verified	timestamp
concept	Agentic Search Best Practices	Query formulation, extraction, validation, retry, and rate limiting for an agent that searches the web.	agentic search rag retrieval unverified	2026-08-28	2026-08-11

Agentic Search Best Practices

An agent that searches well does not fire one query and trust one answer. It rewrites queries, reads the strongest sources, and checks its own work.

This page gives the practices. See Research Effectiveness for how to orchestrate the loop.

The Search Pipeline

graph LR
    A[User question] --> B[Query rewriter]
    B --> C[Query 1]
    B --> C2[Query 2]
    B --> C3[Query 3]
    C --> D[SearXNG]
    C2 --> D
    C3 --> D
    D --> E[Extract content]
    E --> F[Re-rank + dedupe]
    F --> G[Validate facts]
    G --> H[Synthesize + cite]

Query Formulation

• Rewrite the user question into a search-ready query.
• Generate two or three alternative queries for broader coverage.
• Add synonyms and related terms.
• Batch the queries in one action when the framework allows it.

A code-based agent (CodeAct) can run several searches in one action. This cuts LLM calls by about 30 percent compared with one tool call per search. The ReAct loop (reason, act, observe) is the default pattern.

Extraction and Processing

• Convert HTML to clean Markdown before you feed the model. Pi does this in fetch_content.
• Remove near-duplicate results across engines.
• Re-rank with BM25 relevance scoring.
• Keep a source reference for every extracted fact.
• Truncate long pages. A common cap is 40,000 characters per page.

Use query-aware content filters to cut noise before the model sees text. Crawl4AI ships three filters:

• BM25ContentFilter — score page chunks against the query. Keep only relevant snippets.
• PruningContentFilter — strip boilerplate by text and link density.
• LLMContentFilter — delegate filtering to a model for hard pages.

Run BM25 re-ranking on the search results themselves. The SearXNG BM25 reranker plugin builds a temporary index and fuses BM25 scores with engine rankings using reciprocal rank fusion. This lifts keyword accuracy over raw engine order.

Respect Rate Limits

Upstream engines and SearXNG block agents that hammer them. Slow down on purpose.

• Respect Retry-After headers.
• Use exponential backoff on retries.
• Rotate user-agent strings.
• Add random delays between requests.
• Cap the request rate. A common default is 1 request per second.
• Run a circuit breaker that suspends a failing engine.
• Cache results with a time-to-live value.

# SearXNG side: cap timeouts and retries
outgoing:
  request_timeout: 2.0
  max_request_timeout: 10.0
  retries: 3

SearXNG suspends failing engines on a timer. For example, a CAPTCHA response suspends the engine for 3600 seconds. Respect that pause instead of retrying into the block.

Validate the Answer

• Cross-reference each claim against a second independent source.
• Prefer primary sources over summaries.
• Record the authority and freshness of each source.

Score credibility on concrete signals:

• Domain type. Prefer .edu, .gov, and .org over personal .com sites.
• Link density. A low text-to-link ratio signals a content farm.
• Metadata. A real <title> and author byline signal an authored page.
• Cross-reference. Re-fetch the URL and confirm that the claim appears in the content.

The Finish Rule

An agentic loop must end. Decide in advance when enough evidence exists:

1. You found consistent answers across two or more sources.
2. Your retrieval budget is spent.
3. The question is answered with the required confidence.

Cap the loop. A common default is 20 steps for the search agent, with a plan re-check every 4 steps. If no source confirms a claim, say so. Do not smooth over the gap.

Citations

• Smolagents ReAct guide: https://huggingface.co/docs/smolagents/conceptual_guides/react
• CodeAct paper: https://arxiv.org/abs/2402.01030
• Crawl4AI documentation: https://docs.crawl4ai.com
• Search result re-ranking, BM25 approach: https://github.com/Oaklight/searxng-bm25-reranker
• SearXNG outgoing settings: https://docs.searxng.org/admin/settings/settings_outgoing.html
• SearXNG limiter: https://docs.searxng.org/admin/searx.limiter.html

See Also

• Research Effectiveness — Orchestrate the loop into a report.
• SearXNG Tuning — The backend behind the pipeline.
• Self-Hosted Web Agents — The browser tools in the loop.