> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heymilo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Scenario Assessment

> The AI Scenario Assessment is a screening agent that puts candidates in a live conversation with an AI and evaluates how well they use it to complete a task.

<video src="https://storage.googleapis.com/heymilo-pub-docs/videos/AI%20Scenario%20Assessment.mp4" controls />

## What It Is

Instead of answering static questions, candidates are given a goal, context, and a deliverable — then scored on how they prompt and guide the AI to get there.

**Where to find it:** Create Interviewer → Interview Stages → AI Scenario Assessment (Beta)

## How It Works

When a candidate starts the assessment, they see:

<Frame>
  <img src="https://mintcdn.com/heymilo/8Oux0LkKb2t2_LO5/images/Screenshot-2026-05-19-at-12.40.58-PM.png?fit=max&auto=format&n=8Oux0LkKb2t2_LO5&q=85&s=7494c4e35ca933a129cfe17efcb35d31" alt="Screenshot 2026 05 19 At 12 40 58 PM" width="2998" height="1700" data-path="images/Screenshot-2026-05-19-at-12.40.58-PM.png" />
</Frame>

* A **title** describing the task
* A **goal** explaining what they need to accomplish
* A **scenario** with the context they are working within
* A **deliverable** — the final output they need to produce

They then have a back-and-forth conversation with the AI to work toward that deliverable. The assessment ends when they reach the round limit or time limit.

HeyMilo scores the conversation against an auto-generated rubric tailored to the task.

## Candidate Modes

| **Mode**                    | **What the Candidate Does**                                                                                                                                 |
| :-------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Standard**                | Candidate works collaboratively with AI to complete a task, prompting, iterating, and producing a deliverable                                               |
| **Test AI Error Detection** | A weaker AI is used that may produce mistakes. Candidate is scored on identifying and correcting them. Best for AI/ML, safety, and prompt-engineering roles |

## Setting It Up

When adding an AI Scenario Assessment stage, you'll see the following setup options:

<Frame>
  <img src="https://mintcdn.com/heymilo/6UoV0FK1qsE5okaD/images/llm-assessment-setup.png?fit=max&auto=format&n=6UoV0FK1qsE5okaD&q=85&s=b6aedd686ac874c3e26c4f69b42975a6" alt="AI Scenario Assessment setup options" className="mx-auto" width="804" height="1024" data-path="images/llm-assessment-setup.png" />
</Frame>

* **Task background:** describe the task the candidate will work through. The more specific you are, the more targeted the assessment. HeyMilo uses this (plus the job description) to generate the scenario, deliverable, and rubric.
* **Assessment style:** standard mode has the candidate work collaboratively with AI. Toggle on **Test AI error detection** to switch to a weaker AI that may make mistakes. The candidate is scored on identifying them. Best for AI/ML, safety, and prompt-engineering roles. Note: this can't be changed after the scenario is generated.
* **Conversation behavior:**
  * **AI sends a greeting first:** toggle off if you want the candidate to make the first move.
  * **AI invents missing details:** when on, the AI fills in gaps like customer names or dates to keep the scenario flowing.
* **Time limit:** recommended 20 to 30 minutes for screening.
* **Back-and-forth rounds:** one round equals one candidate message plus one AI reply.
* **Allow retake on timeout:** if the candidate runs out of time before sending much, they can restart once.

Once configured, click **Save & generate scenario**. HeyMilo auto-generates the scenario title, goal, scenario description, deliverable, and scoring rubric. You can regenerate or edit any of these before activating.

## Scoring

Each completed assessment produces:

<Frame>
  <img src="https://mintcdn.com/heymilo/ZDFstk2GqqfSubYN/images/ai-scenario-assessment-scoring.png?fit=max&auto=format&n=ZDFstk2GqqfSubYN&q=85&s=1d9f9f64a1db2a8ce84bc2a99bc36b57" alt="AI Scenario Assessment scoring" className="mx-auto" width="1024" height="766" data-path="images/ai-scenario-assessment-scoring.png" />
</Frame>

* An **overall score** out of 4.0
* **Dimension scores** across criteria like:
  * Task Decomposition
  * Instruction Clarity and Constraint Control
  * Error Recovery and Iteration
  * Difficulty
  * Realism
  * Safety
* An **overall narrative** summarizing how the candidate performed
* A full **chat transcript** with evidence quotes tied to each dimension

Scores appear on the candidate profile under the **AI Scenario Assessment** tab.

## Best Use Cases

AI Scenario Assessment works well for any role where candidates need to use AI to produce a real deliverable:

* Data annotation and labeling
* Customer support and escalation protocols
* Content and copy production
* Code generation and technical problem-solving
* Research and analysis tasks

## See It in Action

Real scored transcripts showing how candidates approach an AI Scenario Assessment — including dimension scores, conversation, and what a strong vs. weak run looks like.

[Example: Labeling Assistant](/ai-scenario-assessment-labeling-assistant)

[Example: AI Data Annotator (Find AI Errors)](/ai-scenario-assessment-data-annotator)