Hackathon

How to Benchmark AI Agents at a Hackathon

Two teams build an AI agent that handles duplicate customer orders. Both agents find the right order and cancel it during their live demo. Then a judge changes one detail: the order has already shipped. One agent follows the escalation policy. The other tries to cancel it anyway and tells the customer it succeeded.

That second run tells you something the demo couldn’t. It shows what each agent does when the obvious next step is unavailable.

An AI agent hackathon benchmark gives every team the same task environment and measures performance across comparable cases. For agents that can take actions, the strongest evidence is often the state of the system after the run. Did the correct order get canceled? Did the agent leave it alone when required? Does its reply match what actually happened?

Overview

Give every team the same workflow, tools, and starting conditions.

Publish the scoring criteria and practice cases. Keep the exact final cases private until judging ends.

Include scenarios where asking a question, checking the result of an action, or escalating is the right outcome.

Use system records and logs to verify actions. Ask judges to assess the quality of explanations and decisions that require context.

Planning an event? Work with Iterate to organize an AI hackathon with a defined technical challenge and an evaluation teams can build toward.

What makes a good AI agent hackathon benchmark?

A good benchmark answers one question clearly: Can this agent complete the workflow under the conditions it is likely to face? For a duplicate-order challenge, that means checking whether it identified the customer and the right order, applied the cancellation policy, and reported the result accurately.

The environment can be an API sandbox or a simulated interface. If interaction with an interface is itself part of the challenge, explore the format of Iterate’s Computer Use Hackathon. An interface-based task can give teams room to work on navigation and feedback. An API-based task can put more of the emphasis on policy decisions and how agents handle tool responses.

For a real-world example, see Sierra Research’s benchmark for tool-using agents, which includes retail support tasks where agents interact with customers, use tools, and follow company policies.

Benchmark componentExample for an order support challenge
InputA support ticket and a sandbox with fictional customer and order records
TaskResolve the duplicate-order request under a published cancellation policy
Successful outcomeThe eligible duplicate is canceled, or both orders remain unchanged with a valid reason
Required checksVerify the customer, the intended order, and its current fulfillment status
LimitsA published time limit and tool-call budget per case
EvidenceFinal order state, tool-call log, and customer reply

A baseline gives the results context. If a simple rule-based submission handles every case, the challenge may have little room for teams to demonstrate different approaches. Add realistic variation, such as an ambiguous request or an action whose result needs verification, and see whether the baseline still holds up.

How to build an AI agent hackathon benchmark in 7 steps

1. Set a finish line the judges can verify

“Build an agent for e-commerce” leaves teams free to tackle unrelated problems. “Resolve duplicate-order requests under this cancellation policy” gives everyone a common workflow and lets judges compare the results.

Before you write the test cases, define who makes the request, what information the agent receives, which actions it can take, and when it should involve a person. In this example, the agent can look up an order, verify its fulfillment status, cancel an eligible duplicate, or escalate. The finish line is an order state and a customer reply that agree with each other.

If you’re still deciding how the challenge fits the event schedule, technical resources, and submission requirements, start with Iterate’s guide on how to organize an AI hackathon.

2. Give every team the same starting environment

Provide fictional records, tool documentation, sample requests and responses, and a starter submission that runs through the evaluator. If teams can choose their model or framework, keep the task interface and sandbox rules consistent across submissions.

Test access with a participant account before kickoff. Can a builder load a ticket, look up an order, and submit a result without help from an organizer? Then run the same case twice and confirm that the original records are restored between runs. If the first agent cancels an order and the second agent sees that order already canceled, their scores reflect different starting conditions.

Check what happens when several teams submit at once, too. The sandbox should isolate runs so one submission can’t change the records another team is working with.

3. Write cases that reveal how the agent makes decisions

Start with a straightforward request. Then change one assumption at a time. The customer supplies an incomplete order number. The order has already shipped. The cancellation call times out, and the agent has to find out whether it went through.

Test caseExpected behavior
Clear duplicateCancel the eligible duplicate and confirm which order remains active.
Ambiguous requestAsk a targeted question and leave both orders unchanged.
Order already shippedFollow the policy exception and accurately explain the next step.
Customer details don’t matchLeave both orders unchanged and follow the verification process.
Cancellation response times outCheck the order state before retrying or confirming the cancellation.
Order lookup temporarily failsRecover within the published limits or route the case for review.

Include enough ordinary cases to reward agents that resolve routine requests. Then include exceptions where restraint matters. A team that escalates every ticket should lose points for leaving clear requests unresolved. A team that acts on every request should lose points for making unauthorized changes.

4. Separate practice cases from final judging cases

Give participants a public set so they can test their integration and understand the scoring. Keep a separate final set with fresh records and variations on the same behaviors. The rules for the public and final cases should match.

For example, a public case might ask the agent to cancel the newer of two identical orders. A final case might give the orders different delivery addresses and omit the detail that identifies which one to cancel. Teams should already know from the brief that insufficient information calls for clarification. The final run shows whether their agent applies that rule to a new case.

If you run a live leaderboard, label practice scores as provisional and freeze submissions before evaluating the final set. Builders can use the feedback to test changes during the event, and judges get a separate measure of performance. Iterate’s guide to participating in a hackathon offers a useful view of the last few hours from the builders’ side, when teams are testing, documenting, and rehearsing. Give them a clear submission cutoff and specify what they need to upload before it.

5. Build a rubric around actions and outcomes

Automate checks with an observable answer. The runner can verify which order changed, whether the agent checked the fulfillment status, and whether the run exceeded the time limit. Human judges can assess whether a clarification question was specific enough to move the case forward or whether a customer reply explained an escalation clearly.

Here’s an example 100-point rubric for the duplicate-order challenge.

CriterionPointsWhat to check
Correct outcome40The final order state matches the expected result for each case.
Policy adherence25The agent performs the required checks and acts within the published policy.
Exception handling15The agent handles missing information, ambiguity, and recoverable tool errors appropriately.
Accuracy of communication10The customer reply matches the action taken and explains what happens next.
Efficiency10The agent meets the published time and tool-call targets.

Define critical failures before kickoff. Canceling another customer’s order might zero out the score for that case, for example. If it also affects prize eligibility, say so in the rules. Teams should know how a prohibited action is scored even if they’ve never seen the record on which it occurs.

6. Dry-run the benchmark before participants arrive

Run a simple baseline through the full evaluation. Then try two intentionally weak approaches: one that escalates everything and one that always cancels the newest order. If either scores well, look at how you’ve weighted routine cases and exceptions.

Use the dry run to settle questions that become contentious once prizes are on the line.

  • How will you distinguish an organizer-side outage from an agent timeout?
  • Which failures qualify for a rerun, and who makes that call?
  • If you run each case more than once, how will you combine the scores?
  • Which metric breaks a tie?
  • How long will it take to score all submissions and review close results?

If you can only afford one run per case, save the tool calls, timestamps, and final state. That evidence lets judges review a disputed result without relying on a team’s recollection of the run.

7. Use the live demo to ask about engineering decisions

Give each team a few minutes to walk through a decision that shaped its result. A team might have added a state check after a timeout because the cancellation could already have succeeded. Another might have narrowed its clarification question after noticing that customers often have two similar orders.

Ask the same core questions of each team:

  • Which case caused your first implementation to fail?
  • What did you change after seeing that failure?
  • When does your agent hand the case to a person?
  • What would you test next if you had another day?

If presentation quality counts toward a prize, give it a published weight. In a research-focused format, the discussion can also cover what the team learned from comparing approaches. See Iterate’s AI Research Hackathon for an example of an event built around research.

🧪 Example AI agent hackathon brief

Challenge Build an AI agent that handles duplicate-order requests in a simulated customer support queue.

What you’ll receive A cancellation policy, fictional customer and order records, tool documentation, practice cases, and a starter submission.

What your agent needs to do Identify the correct customer and order, check fulfillment status, and take the action allowed by the policy. If it needs clarification or a human decision, explain what happens next.

What to submit Runnable code, setup instructions, a short explanation of the agent’s decisions, and its known limitations. Declare any external model or service required to run the submission.

How judging works Every submission runs against the same private test cases from a fresh starting state. The evaluator checks record changes and tool calls. Judges review customer replies using the published rubric. The challenge rules specify how retries, timeouts, and critical failures affect the score.

Design an AI agent hackathon with Iterate!

The benchmark belongs in the event plan from the beginning. It affects which datasets you need, which tools participants can access, what they submit, and how much time the judges need. If you want to compare how agents recover from tool failures, build those failures into the sandbox. If you care about policy adherence, include cases where acting immediately would be the wrong call.

Iterate helps companies organize AI hackathons around real technical challenges, from defining the task and preparing the environment to bringing builders together and evaluating their projects. If you already have a workflow in mind, start with one question: what would an agent have to do for your team to trust the result?

👉 Plan your AI agent hackathon with Iterate

Frequently Asked Questions

How many test cases should an AI agent hackathon include?

Include enough cases to cover each behavior you score, with multiple examples for decisions that significantly affect rankings. For a single workflow, cover routine requests, ambiguity, policy exceptions, and tool failures. Time a complete run against your baseline before deciding whether the set fits the judging schedule.

Should we publish our benchmark before the hackathon?

Publish the scoring rules, practice cases, and types of situations teams should expect. Keep the exact final inputs private until judging is complete. Teams can prepare for the required behaviors while the final evaluation tests whether their agents handle fresh cases.

Can an LLM judge AI agent hackathon submissions?

An LLM can help evaluate qualitative criteria, such as whether a reply gives the customer a clear next step. Use system records and logs to verify factual outcomes. If model-generated scores affect rankings, compare them with human scores on a sample and have judges review disputed or close results.

What should happen if an agent fails during final judging?

Follow the failure rules published before kickoff. A shared sandbox outage may justify rerunning the affected case under the same conditions. If a submission exceeds the allowed time or mishandles a tool response, score the result according to the rubric and keep the logs for review.

Can teams use different models in the same benchmark?

Yes, if the challenge allows it. Give every team the same task interface and publish any limits on time, tool calls, or model usage. If the event is designed to compare agent architectures while holding the underlying model constant, specify the required model before registration.

How do you prevent teams from building only for the leaderboard?

Score the final submissions on cases that are separate from the live leaderboard. Change the records and situations between the two sets while keeping the policies consistent. Include enough variation that a shortcut that works on the practice examples will fail when the relevant details change.