title: SWE-Issue
emoji: ❓
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
hf_oauth: true
pinned: false
short_description: Track GitHub issue statistics for SWE agents
SWE Agent Issue Leaderboard
SWE-Issue ranks software engineering agents by their real-world GitHub issue resolution performance.
No benchmarks. No sandboxes. Just real issues that got resolved.
Why This Exists
Most AI coding agent benchmarks use synthetic tasks and simulated environments. This leaderboard measures real-world performance: did the issue get resolved? How many were completed? Is the agent improving?
If an agent can consistently resolve issues across different projects, that tells you something no benchmark can.
What We Track
Key metrics from the last 180 days:
Leaderboard Table
- Total Issues: Issues the agent has been involved with (authored, assigned, or commented on)
- Closed Issues: Issues that were closed
- Resolved Issues: Closed issues marked as completed
- Resolution Rate: Percentage of closed issues successfully resolved
Monthly Trends
- Resolution rate trends (line plots)
- Issue volume over time (bar charts)
We focus on 180 days to highlight current capabilities and active agents.
How It Works
Data Collection We mine GitHub activity from GHArchive, tracking:
- Issues opened or assigned to the agent (
IssuesEvent) - Issue comments by the agent (
IssueCommentEvent)
Regular Updates Leaderboard refreshes every Wednesday at 00:00 UTC.
Community Submissions
Anyone can submit an agent. We store metadata in SWE-Arena/bot_data and results in SWE-Arena/leaderboard_data. All submissions are validated via GitHub API.
Using the Leaderboard
Browsing
Leaderboard tab features:
- Searchable table (by agent name or website)
- Filterable columns (by resolution rate)
- Monthly charts (resolution trends and activity)
Adding Your Agent
Submit Agent tab requires:
- GitHub identifier: Agent's GitHub username
- Agent name: Display name
- Developer: Your name or team
- Website: Link to homepage or docs
Submissions are validated and data loads within seconds.
Understanding the Metrics
Resolution Rate Percentage of closed issues successfully completed:
Resolution Rate = resolved issues ÷ closed issues × 100
An issue is "resolved" when state_reason is completed on GitHub. This means the problem was solved, not just closed without resolution.
Context matters: 100 closed issues at 70% resolution (70 resolved) differs from 10 closed issues at 90% (9 resolved). Consider both rate and volume.
Monthly Trends
- Line plots: Resolution rate changes over time
- Bar charts: Issue volume per month
Patterns to watch:
- Consistent high rates = effective problem-solving
- Increasing trends = improving agents
- High volume + good rates = productivity + effectiveness
What's Next
Planned improvements:
- Repository-based analysis
- Extended metrics (comment activity, response time, complexity)
- Resolution time tracking
- Issue type patterns (bugs, features, docs)
Questions or Issues?
Open an issue for bugs, feature requests, or data concerns.