Search-SWE

Introduction

Search is a core capability in AI agent applications, from retrieving evidence for research assistants to locating code for software agents and grounding enterprise assistants in internal knowledge. Yet building an effective search system—or improving an existing one—requires substantial engineering expertise: choices in query understanding, indexing, retrieval, and ranking must balance relevance with latency and resource use. As coding agents become part of everyday software development, a natural question arises: can they also engineer effective search systems?

We introduce Search-SWE, a benchmark for evaluating coding agents on implementing and optimizing executable search systems. Its design centers on explicit task contracts, reproducible environments with fixed inputs, and task-specific resource budgets. Agents inspect their environment, write and test code, and iterate; a separate verifier evaluates the resulting system on held-out queries, checking functionality, retrieval quality, efficiency, and compliance with task rules. The benchmark measures working systems and the practical quality–efficiency trade-offs made during development.

Explore tasks spanning query understanding, retrieval, ranking, and representation learning.

• Reasoning-intensive Query Rewriting Rewrite complex biology queries to bridge the gap between query intent and relevant documents. →
• Memory-constrained Dense Retrieval Search 1.5M dense vectors within 2 GiB of memory while meeting accuracy and latency requirements. →
• Long-document Reranking Improve long-document reranking through window selection and score aggregation, with a fixed model and candidate set. →
• Code Embedding Fine-tuning Fine-tune a pretrained embedder for code retrieval while preserving its backbone architecture. →

We welcome contributions to tasks, tools, shared environments, tests, and documentation. New tasks should include clear instructions, reproducible environments, fixed inputs and a separate verifier. Start with the agent workflow and the create-searchswe-task skill for authoring and validation.

1. Create a task in your branchCreate the package directly at tasks/<task-name>. Choose a descriptive name of at most five lowercase hyphen-separated words, beginning with a letter. The task- prefix and all are reserved. Check names against the current base and active PRs. A PR may add multiple uniquely named tasks; record actual authors in each package. Keep ordinary task contributions within their package; shared docs and tests discover tasks automatically.
2. Build a reproducible packageSpecify interfaces, metrics, and resource limits; isolate the verifier from the agent. Fixed inputs may reference existing public upstream Hugging Face datasets directly. Document provenance and usage terms, and pin each exact filename, immutable 40-character commit SHA, size and SHA-256. New development assets may use a personal public dataset with redistribution permission. Keep credentials and private evaluation assets outside Git.
3. Test with a coding model and refineRun package checks, unit tests, container checks and verifier positive and negative cases, then an authorized coding-model trial. Use --task <task-name> for downloads and trials. Inspect trajectories, scores and judge health; use the findings to refine the task setting and revalidate substantive changes. The authoring agent prompts contributors to configure any missing API keys locally. Report findings, changes, the final tested revision and unrun checks in the PR, and enable maintainer edits where available.
4. Review, finalize and mergeMaintainers use the maintain-searchswe-task skill to review each task and its validation evidence. Apply review fixes in the same PR, retain verified upstream pins, and verify downloads, hashes and merge-ready checks for the final revision. Official publication is optional; when needed, publish under tasks/<task-name>/ and pin the merged official Hugging Face SHA before merging the task PR. Use a merge commit to preserve author history.

Before scaffolding, inspect one or two closest reviewed tasks already merged into main, chosen by objective, grader, resources/APIs, hardware, and input/artifact layout. Record their paths and relevance, and use them only as structural references: current guides, schemas, and validators take priority. Independently establish task-specific data, thresholds, licenses, Hugging Face pins, names, authors, and access policies rather than copying them. If no close reference exists, say so.

Evaluation

To run an evaluation, follow the repository’s Quick Start ↗: install the host tools, restore task inputs, configure credentials, and launch a task. The walkthrough runs reasoning-query-rewriting on CPU with Pi and DeepSeek Flash; Python 3.12+ and Docker are required.

Search-SWE uses Harbor to evaluate coding agents through the supported Codex CLI, Pi, and Claude Code launchers in isolated environments. Agents develop solutions using supplied data and public checks. Submissions are then tested on held-out queries, with task-specific correctness, resource, and integrity checks determining whether their search-quality scores are valid.

Results. The leaderboard is being prepared.