RSI-Exam
For task authors

What makes a good task.
And how to propose one.

Read this before proposing a task. It is what a reviewer looks at: whether the problem has room left in it, what evidence you can bring, and what a finished task needs.

Open the proposal form Browse the 35 public tasks
What a task is

What the agent gets, and what it is judged on

The agent inherits a working but weak method and a set of visible data it can measure itself on. Whatever it submits is replayed once, from scratch, on data it never touched.

Visible data The starting method

The agent edits it, runs it, measures the result, and repeats — for up to twelve hours.

the agent's container
one artifact
crosses
Hidden data The submitted code, re-run on data the agent never saw

A fresh container runs it against the hidden set with a fixed verifier. That number is the score.

the grading container
The one thing to keep in mind
  • The agent submits code, not answers. The verifier re-runs that code on data the agent never saw, so an improvement that does not transfer earns nothing. Every guideline below follows from this shape.
The bar

Task difficulty

We want growth problems with room left in them. An agent should work for a long time, make genuine progress, and still have distance ahead of it when the budget ends. A run gives it up to twelve hours, and a frontier model gets a lot done in that time.

Distance to the best method

  • The agent works the whole budget and still has not caught the strongest known solution.
  • The usual case, and the tier we most want.

Distance to the ceiling

  • The agent passes the best known solution and is still well short of what the problem allows.
  • Also a good task — often a more informative one.

The difficulty has to come from the problem itself. RSI-Exam is trying to measure research ability — forming a hypothesis, testing it, reading the result, and deciding what to try next. Textbook problems, and problems with a known recipe, no longer pose any difficulty to a frontier agent.

Signs a task may saturatecheck yours against these
  • The published SOTA already scores in the high eighties or nineties. A dataset that has been competed on for years is usually a poor starting point — the agent hits the ceiling almost immediately.
  • A standard library call already reaches most of the metric.
  • One well-known trick is the whole task.
  • The search space is small enough that hyperparameter sweeps exhaust it.
Optional, and it carries weight

Evidence of difficulty

Neither route below is required, but a proposal carrying one is much easier to accept — and either saves you from building a task that turns out to saturate.

ARun it once

Try it on an agent yourself

No container, grader or harness needed. Write the instruction as you would give it to an agent, put your starting method in a scratch directory, and hand both over for a couple of hours. Then tell us what happened.

  • What score did it reach, and how far is that from what you believe is achievable?
  • How many genuinely different things did it try before running out of ideas?
  • Did it plateau, or was it still improving when you stopped it?

Did it find a shortcut you had not thought of? The most valuable outcome here — far better to learn it now than after the task is built.

BOr argue it

Show the problem is just hard

For some problems the difficulty is a property of the problem itself rather than something you have to measure.

  • The best method anyone has still reaches only a small fraction of what is possible.
  • A known limit puts the ceiling out of reach.
  • The problem has been worked on for years with slow, incremental progress.

Numbers and citations make this convincing.

If you have neither, propose anyway. The difficulty question then gets settled during the build.

Quality

What a good task has

All of the followingseven things
  • A problem drawn from real work. Something a researcher or engineer in your field would actually be paid to improve, not a textbook exercise.
  • A metric computed programmatically, with a stated direction. It need not be a single raw number — a mean over instances, several components combined, or one quantity gated on another are all normal. It only has to reduce to one number by a rule you define, with no human in the loop.
  • A metric with no shortcut. Imagine a submission that does no real work: it always predicts the same value, returns nothing, or copies the average of the labels. Run that through your metric — it should land at the bottom. If a lazy answer scores well, an agent will find that loophole long before it finds the science.
  • Visible data the agent can measure itself on. Every task ships a self-check the agent runs to score its current method; that measurement is what it steers by. It has to be quick enough for the agent to get through it at least twice in a twelve-hour run. Where you can, give it a train/validation split so it can estimate out-of-sample performance honestly.
  • Hidden data that cannot be looked up. Never in the agent's container, used once by the grader. The visible set and the sealed set are not required to share a distribution — the sealed set may be larger, or cover cases the visible data does not. What is required is being able to say why.
  • A starting method that runs. It defines the baseline anchor, so it must produce valid output before anyone touches it, and be plain but competent — not a crippled strawman, not something already near the ceiling. Submitted untouched, it must score at the floor.
  • A reference solution you can run. Not "the paper reports 0.87" — a solution you have executed and whose number you reproduced on your data. If you cannot reproduce it, say so: the task can still be accepted, it just ships without that anchor rather than with a fabricated one.
How it works

Propose your task

Before writing anything, get the idea reviewed. Building a task is a real investment, and the most common way that investment is wasted is a task that turns out to saturate.

  1. Read the guidelines above10 min

    Most declined proposals are declined because the task saturates. The sections above are what a reviewer will be checking.

  2. Send the formnothing built yet

    The problem, the metric, the visible and hidden data, the anchors and whether you can reproduce the reference number, and why the task will not saturate. If you have already tried a little of it, there is a field for that and it carries weight.

  3. We respond1–2 days

    Accepted, accepted with changes, or a specific reason it will not work as proposed.

  4. Revise and resubmitif needed

    Proposals that do not work as submitted can usually be re-cut rather than abandoned — different data, a different metric, a weaker starting method.

What you write on the form is what the build is based on, so the clearer it is, the less of the next stage is spent re-deriving decisions you have already made.

After acceptance

What happens next

You get the authoring kit

  • Access to the task authoring repository.
  • The directory layout and the task.toml reference.
  • A complete example task to work from.
  • Two skills that check your work before you submit it.

Then it goes through review

  • Automated checks on structure, format and sealing.
  • A human review against the full checklist.
  • A run by frontier agents that places its difficulty.
  • Tasks that need changes are iterated on, not discarded — nearly every task in the benchmark went through at least one round.
Credit

Authorship

Task authors are credited in the task metadata, on the task's page on this site, and in the benchmark release. If you contribute several tasks, or take on review work for other people's tasks, we would like to talk about a larger role.

Merged tasks are released publicly under MIT. Please only contribute data and code you have the right to release.

Resources

Where to look

The 88 tasks Every task has a page with its metric, its anchors and the trajectories of nine models. The 35 public tasks Complete with their grading containers — the fastest way to see what a finished task looks like. Evaluation infrastructure How a run is executed, and the network policy that seals it. Harbor task format The framework underneath, and the reference for the task directory itself.
Next step

Send the proposal

If you already have a task in mind, bring one sentence about the problem and one about the metric.

Open the proposal form Ask on Discord

Questions and feedback: Discord, the WeChat group, or contact@rsi-exam.ai.