[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"readme:swe-sweep":3},"\u003Ch1>\u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Ffacebookresearch\u002Fswe-sweep\u002FHEAD\u002Fdocs\u002Fassets\u002Fswe-sweep-hero.png\" alt=\"SWE-sweep logo\" width=\"300\" \u002F>\u003Cbr \u002F>SWE-sweep\u003C\u002Fh1>\u003Cp align=\"center\">\u003Cem>How many bugs can LMs find & fix in large codebases?\u003C\u002Fem>\u003C\u002Fp>\u003Cp align=\"center\">\nGiven a real repository, an agent must discover & repair as many bugs as they can.\u003Cbr \u002F>\nAgents are not given any hint about the type of bug or its location.\n\u003C\u002Fp>\u003Ch2>Links\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>Website: \u003Ca href=\"https:\u002F\u002Fswesweep.com\" rel=\"nofollow ugc noopener\">https:\u002F\u002Fswesweep.com\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>Paper: \u003Ca href=\"https:\u002F\u002Fswesweep.com\u002Fpaper\" rel=\"nofollow ugc noopener\">https:\u002F\u002Fswesweep.com\u002Fpaper\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Quickstart\u003C\u002Fh2>\n\u003Cblockquote>\n\u003Cp>[!note]\nThe code in this repo is only a thin wrapper around harbor, however see the warning below regarding harbor version\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cp>[!warning]\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Harbor version\u003C\u002Fstrong>: Harbor 0.23 does not yet support the separate verifier environments and collect hooks\nused by these tasks. The project therefore pins a compatible revision from Harbor's source\nrepository until Harbor 0.24 is released.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Final score\u003C\u002Fstrong>: We build a micro-average of the bugs resolved (equivalent to a weighted average of the\nindividual task scores). See evaluation notes.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fblockquote>\n\u003Cp>We recommend \u003Ca href=\"https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fgetting-started\u002Finstallation\u002F\" rel=\"nofollow ugc noopener\">uv\u003C\u002Fa> for managing\nPython environments.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">git clone https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fswe-sweep.git\ncd swe-sweep\nuv sync   # or pip install .\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Verify your setup:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">sweep infra doctor\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cdetails>\n  \u003Csummary>Development setup\u003C\u002Fsummary>\u003Cp>  Clone the repository and install the editable package with its dev dependencies:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">git clone https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fswe-sweep.git\ncd swe-sweep\nuv sync --extra test\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>  Run the test suite (matches CI, which tests on Python 3.12 and 3.13):\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">uv run pytest\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>  Run the linting\u002Fformatting hooks:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">uvx pre-commit run --all-files\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fdetails>\u003Ch2>Evaluate a single task\u003C\u002Fh2>\n\u003Cp>Pass a task name and a unified diff against that task's base commit:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">sweep eval <task> <path\u002Fto\u002Fsubmission.diff>  # see raw harbor command below\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cdetails>\n  \u003Csummary>Raw harbor command\u003C\u002Fsummary>\u003Cpre>\u003Ccode class=\"language-bash\">uv run harbor run \\\n--path tasks\u002Fswe-sweep__dateutil \\\n--agent swesweep.patch_agent:PatchAgent \\\n--agent-kwarg \"patch_path=$(realpath \u002Fpath\u002Fto\u002Fmodel.patch)\" \\\n--jobs-dir jobs \\\n--n-concurrent 1 \\\n--yes\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fdetails>\u003Cp>The command runs the corresponding Harbor task with a small patch-applying agent. Harbor\nthen evaluates the resulting checkout in the task's separate verifier environment. Results\nare written under \u003Ccode>jobs\u002F\u003C\u002Fcode> by default.\u003C\u002Fp>\n\u003Cdetails>\n\u003Csummary>How evaluation works\u003C\u002Fsummary>\u003Cp>Evaluation follows the following pseudo-code:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-python\">reset_to_base_commit()\napply(agent_patch)\nreset(test_files)\nbuild_if_needed()\nbase_commit_results = run_visible_suite()  # test -> pass\u002Ffail\n\nsubtask_results = {}   # subtask -> {test -> pass\u002Ffail}\nfor subtask in subtasks:\n  apply(subtask.test_patch)\n  subtask_results[subtask.id] = run(subtask.hidden_tests)\n  revert(subtask.test_patch)\n\nif any_new_failures(base_commit_results):\n  task_score = 0\nelse:\n  task_score = sum(passed_subtasks) \u002F len(subtasks)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fdetails>\u003Ch2>Calculate the final score\u003C\u002Fh2>\n\u003Cp>The benchmark score is calculated as\u003C\u002Fp>\n\u003Cpre>\u003Ccode>total number of bugs solved across tasks \u002F total number of bugs =\n    = mean(number of bugs in task * task score)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Summarize one or more directories containing graded Harbor trials:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">sweep info path\u002Fto\u002Fgraded-solutions --per-task\nsweep info path\u002Fto\u002Fgraded-solutions --json\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Citation\u003C\u002Fh2>\n\u003Cpre>\u003Ccode class=\"language-bibtex\">@misc{lieret2026swesweep,\n  title  = {{SWE-sweep}: Can Agents Autonomously Find and Fix Bugs?},\n  author = {Kilian Lieret and Jeffrey Jian Ma and Rahul Kindi and\n            Yuxiang Wei and Jeremy Ma and Sten Sootla and\n            Parth Thakkar and Chao Beyond Zhou and Pengcheng Yin and\n            Rui Hou and Ofir Press and John Yang},\n  year   = {2026},\n  note   = {Preprint}\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>License\u003C\u002Fh2>\n\u003Cp>SWE-sweep is licensed under the terms of the license f\u003C\u002Fp>\n",1793926099483]