RSI-Exam tests whether an AI agent can improve an existing frontier method in a real-world production or scientific setting. Every task is contributed by someone who works on that problem. We are looking for more experts to bring problems from their own fields.
Register first, or submit a task proposal directly. See the task-author process.
Your problem, a weak starter that runs, and a stronger method you can run yourself. We do the packaging.
≈ 18 hRead someone else's task and check the metric, the seal and the anchors. A substantive review earns four contribution credits.
≈ 4 h eachRead one agent trajectory end to end and say whether the score moved for a real reason.
≈ 2 h eachScoring, sealing, difficulty gates, the harness. This is the core-contributor track.
OpenYour field and what you want to do. Once registered, we email you the project’s milestone announcements. Already have a task in mind? Go straight to step 2.
The guide covers what makes a good task, and how to propose one — it helps you frame your own problem more precisely before you write it up.
Fill in the proposal form. Define the problem, the metric, the data split, the anchors, and why the task is challenging for an agent.
We reply within three days. Once the proposal passes you get the authoring kit and start building. Automated checks and a human review keep the task conforming as you go.
If you can, run the full task and keep the trajectory. Self-check it against our rubric first; we then run a human trajectory review and send back feedback for you to revise on.
A peer reviews it. If it passes, it ships with your name on it.
Paper authorship is based on substantial benchmark work. One accepted task qualifies; so do four substantive reviews, or an equivalent mix.
Listed first, ordered by contribution. Three things count:
| what you deliver | credits |
|---|---|
| A task you wrote, accepted into the bank | 15 |
| A task we implemented from your problem, metric and references | 10 |
| A task that fails a gate after real work | Up to 10 |
| A review with a written verdict | 4 |
| A rollout audit | 2 |
| A pipeline or evaluation change we adopt | 2–8 |
| A referral | Acknowledged, 0 |
Each completed benchmark contribution receives credit under the schedule above. We share the contribution record with everyone on it before submission; if a line is wrong, tell us and we fix it.
For planning, the tables below estimate the work that needs your domain judgment. A first task usually fits into a few evenings a week over three or four weeks; we take care of the engineering, probes and compute.
| step | typical time (h) |
|---|---|
| Scope the problem, check the data licence | 2–4 |
| Design the visible / hidden split | 2–5 |
| Get the weak starter running | 2–4 |
| Run the reference method | 4–10 |
| Set the metric and the guards | 1–3 |
| Write the instruction | 1–2 |
| Audit the trajectory, revise | 2–5 |
| Total | 14–33 · usually ≈ 18 |
The reference run sets the 0.60 mark on the score scale. A number from a paper does not set it; the method has to run on your data. A second task usually takes about two thirds as long as the first.
| step | typical time (h) |
|---|---|
| Read the task as if you had to solve it | 0.5–1 |
| Judge the metric and the realism | 0.5–1 |
| Check the leakage report | 0.5–1 |
| Check the anchors and the scale between them | 0.5–1.5 |
| Read one agent trajectory | 0.5–1 |
| Write the verdict | 0.5 |
| Total | 3–6 · usually ≈ 4 |
An automated pass runs the mechanical checks first and hands you the evidence: recomputed anchors, leakage candidates, the trivial-output baseline. You make the calls it cannot.
Yes, and that is the easiest case. You need to be able to run the code today, and the data has to be redistributable or rebuildable offline.
Either way. The bank wants both CPU-only and GPU tasks — say which one yours needs on the proposal form. Agent rollouts can run on our compute if you do not have your own.
Then it does not ship as it stands, and the work is still credited. Calibration catches this before the expensive part. Often the fix is a harder hidden split rather than a new task.
Send them the form and ask them to mention your name when they submit it. We name referrers in the acknowledgements. A referral on its own does not earn credit, since authorship tracks work on the benchmark itself.
To join the paper accompanying RSI-Exam 1.0, complete your contribution and have its credits confirmed by September 15, 2026. The RSI-Exam 2.0 deadline is December 15, 2026. A first task usually takes three to four weeks at a few hours a week; we confirm the schedule when we review your proposal.
Registering takes a few minutes and puts you on the announcement list. If you already have a concrete task in mind, send the proposal instead — that is where a task actually starts.
Questions, partnership enquiries, or unsure whether your task fits? Email contact@rsi-exam.ai.