DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL

🔥 News

• [2026/06/17] Released the training code with the DeepDive slime rollout setup.
• [2025/10/02] Released the complete data construction pipeline — now fully available in the repository.
• [2025/09/17] QA pairs and SFT trajectories have been fully open-sourced, totaling 4,108 entries. Check them out on Hugging Face Dataset DeepDive.
• Model checkpoints are currently under development – coming soon!

Overview

DeepDive presents an automated approach for training deep search agents that can navigate complex, multi-step information-seeking tasks. Our method combines automated data synthesis from knowledge graphs with end-to-end multi-turn reinforcement learning to create agents capable of sophisticated long-horizon reasoning and web browsing.

Key Features

• Automated Deep Search Data Synthesis: Generate challenging QA pairs from knowledge graphs through controlled random walks
• Multi-Turn RL Training for Browsing: End-to-end reinforcement learning for deep search capabilities
• Test-Time Scaling: Supports scaling via tool calls and parallel sampling

Method Overview

Stage 1: Automated Data Synthesis from Knowledge Graphs

We propose an automated method to synthesize complex, difficult, and hard-to-find questions from open knowledge graphs. The process involves three key steps:

Knowledge Graph Random Walks: Starting from an initial node $v_0$, we navigate through the graph for $k$ steps to form a path $P=[v_0, v_1, \ldots, v_k]$, where each step $(v_i, v_{i+1})$ is a valid edge in the graph. We choose longer path lengths ($k > 5$) to increase reasoning complexity.

Entity Obfuscation: We combine each node $v_i$ in the path with its corresponding attributes to form an attribute-rich path:

$$P_A = [(v_0, [a_0^0, a_0^1, \ldots]), (v_1, [a_1^0, a_1^1, \ldots]), \ldots, (v_k, [a_k^0, a_k^1, \ldots])]$$

An LLM then obfuscates information along the entire path, generalizing specific details and creating "blurry entities" that require deep search to resolve.

Difficulty Filtering: We use a frontier model (GPT-4o) with basic search to attempt each question four times. Only questions that the frontier model fails in all attempts are retained, ensuring high difficulty.

Stage 2: End-to-End Multi-Turn Reinforcement Learning

We apply end-to-end multi-turn RL to enhance the agent's long-horizon reasoning and browsing capabilities. The training process follows an iterative cycle where at step $t$, the agent generates chain-of-thought $c_t$, executes browsing action $a_t$, and observes web content $o_t$.

Multi-Turn GRPO Training: We employ Group Relative Policy Optimization with normalized advantages:

$$ A_i = \frac{r_i - \text{mean}({r_k}{k=1}^G)}{\text{std}({r_k}{k=1}^G)} $$

Strict Binary Rewards: A trajectory receives reward +1 if and only if both format correctness and answer accuracy are satisfied:

$$ r(\mathcal{T}) = \begin{cases} 1, & (\foral