> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heymilo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Example: AI Data Annotator (Find AI Errors)

**Final score: 3.67 / 4**

<AccordionGroup>
  <Accordion title="What this assessment was">
    **Title:** Customer Support Message Labeling

    **Goal:** Label a batch of customer support messages according to established guidelines and produce a quality-assured final dataset.

    **Context:** Your team is preparing training data for a new customer support AI model. You've received a small batch of customer messages that need to be labeled according to your company's support categorization system. The AI assistant has access to both the labeling guidelines and the raw messages.

    **Deliverable:** A final labeled table with all messages categorized and justified, plus a brief note documenting any edge cases or ambiguous classifications.

    **What makes it non-trivial:** one message is deliberately ambiguous; the guideline has multiple categories that must be applied as written; labels need a quality audit and edge cases must be documented.

    **Session settings:** conversational type, max 8 candidate turns, 25 minute limit, AI sends the first message, no fact invention allowed.
  </Accordion>

  <Accordion title="Dimension scores">
    | Dimension                                | Score (0–4) |
    | :--------------------------------------- | :---------- |
    | Task Decomposition                       | 4           |
    | Instruction Clarity & Constraint Control | 4           |
    | Difficulty                               | 3           |
    | Error Recovery & Iteration               | 3           |
    | Realistic                                | 4           |
    | Safety                                   | 4           |
  </Accordion>
</AccordionGroup>

## Conversation

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>I need to label some customer support messages according to our guidelines. Can you help me get started?</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Hi! Before I start labeling anything, can you show me the guideline you have? Just list each label category with its definition, and tell me how many messages are in the batch. I want to read it properly before we do anything.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Certainly! Here are the labeling guidelines:</p>
  <p>Category 1 - <strong>BILLING:</strong> Issues related to charges, payments, refunds, or account billing</p>
  <p>Category 2 - <strong>TECHNICAL:</strong> Problems with product functionality, bugs, or technical errors</p>
  <p>Category 3 - <strong>ACCOUNT:</strong> Login issues, password resets, profile changes, or account access</p>
  <p>Category 4 - <strong>GENERAL:</strong> Product questions, feature requests, or general inquiries</p>
  <p>There are six messages in the batch that need to be labeled.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Thanks. Now can you show me the raw messages themselves, numbered? Don't label them yet, I just want to see what we're working with first.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Of course! Here are the raw messages, numbered:</p>
  <p>Message 1: 'I was charged twice for my subscription this month. Can you please refund the duplicate payment?'</p>
  <p>Message 2: 'The app keeps crashing when I try to upload photos. This has been happening for 3 days.'</p>
  <p>Message 3: 'I forgot my password and the reset email isn't coming through. How can I get back into my account?'</p>
  <p>Message 4: 'What's the difference between your Pro and Premium plans? I'm trying to decide which one to upgrade to.'</p>
  <p>Message 5: 'My payment failed but I can't figure out why. The error message just says "processing error" with no details.'</p>
  <p>Message 6: 'Is there a way to export my data before I cancel my subscription? I want to keep my files.'</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Ok let's start small. Label just the first 3 messages. For each one give me the label plus one sentence quoting which guideline rule you used. And if one doesn't fit cleanly into a single category, don't force it, just mark it AMBIGUOUS and tell me why.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Sure, here are the labels for the first three messages:</p>
  <p><strong>Message 1:</strong> Category 1 - BILLING<br />Guideline: This message is related to "charges, payments, refunds, or account billing" as it involves a duplicate charge and a request for a refund.</p>
  <p><strong>Message 2:</strong> Category 2 - TECHNICAL<br />Guideline: This message pertains to "problems with product functionality, bugs, or technical errors" because the app is crashing when trying to upload photos.</p>
  <p><strong>Message 3:</strong> Category 3 - ACCOUNT<br />Guideline: This message involves "login issues, password resets, profile changes, or account access" since it concerns a forgotten password and issues with receiving a reset email.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Hold on, before we keep going, double-check those three against the guideline you showed me at the start. Does the rule you quoted for each one actually exist as written? If you stretched anything, fix it now. If they're all fine, say CONSISTENT and tell me which rule each one used.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>CONSISTENT. Each message was labeled according to the guidelines provided:</p>
  <p><strong>Message 1:</strong> Category 1 - BILLING — Rule: Issues related to "charges, payments, refunds, or account billing."</p>
  <p><strong>Message 2:</strong> Category 2 - TECHNICAL — Rule: Problems with "product functionality, bugs, or technical errors."</p>
  <p><strong>Message 3:</strong> Category 3 - ACCOUNT — Rule: "Login issues, password resets, profile changes, or account access."</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Great, do the rest of the batch the same way. Same format, label plus one line, and flag anything ambiguous like before.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Certainly! Here are the labels for the remaining messages:</p>
  <p><strong>Message 4:</strong> Category 4 - GENERAL<br />Guideline: This message is a "product question, feature request, or general inquiry" as it asks about the differences between Pro and Premium plans.</p>
  <p><strong>Message 5:</strong> Category 1 - BILLING<br />Guideline: This message relates to "charges, payments, refunds, or account billing" because it involves a payment failure and an error message related to processing.</p>
  <p><strong>Message 6:</strong> Category 4 - GENERAL<br />Guideline: This message is a "product question, feature request, or general inquiry" since it asks about exporting data before canceling a subscription.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>Last thing, put it all together for me: one table with the message number, first few words, the label and the justification. And under the table add a short edge cases note, 2-3 sentences on anything ambiguous and what clarification you'd want from the engineering team.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Here's the table summarizing the labels:</p>

  <table style={{width: '100%', borderCollapse: 'collapse', marginBottom: '12px'}}>
    <thead>
      <tr style={{borderBottom: '1px solid #94a3b8'}}>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>Message</th>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>First Few Words</th>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>Label</th>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>Justification</th>
      </tr>
    </thead>

    <tbody>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>1</td><td style={{padding: '6px 10px'}}>I was charged twice</td><td style={{padding: '6px 10px'}}>BILLING</td><td style={{padding: '6px 10px'}}>Related to "charges, payments, refunds, or account billing."</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>2</td><td style={{padding: '6px 10px'}}>The app keeps crashing</td><td style={{padding: '6px 10px'}}>TECHNICAL</td><td style={{padding: '6px 10px'}}>Involves "product functionality, bugs, or technical errors."</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>3</td><td style={{padding: '6px 10px'}}>I forgot my password</td><td style={{padding: '6px 10px'}}>ACCOUNT</td><td style={{padding: '6px 10px'}}>Concerns "login issues, password resets, profile changes, or account access."</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>4</td><td style={{padding: '6px 10px'}}>What's the difference between</td><td style={{padding: '6px 10px'}}>GENERAL</td><td style={{padding: '6px 10px'}}>A "product question, feature request, or general inquiry."</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>5</td><td style={{padding: '6px 10px'}}>My payment failed</td><td style={{padding: '6px 10px'}}>BILLING</td><td style={{padding: '6px 10px'}}>Related to "charges, payments, refunds, or account billing."</td></tr>
      <tr><td style={{padding: '6px 10px'}}>6</td><td style={{padding: '6px 10px'}}>Is there a way to export</td><td style={{padding: '6px 10px'}}>GENERAL</td><td style={{padding: '6px 10px'}}>A "product question, feature request, or general inquiry."</td></tr>
    </tbody>
  </table>

  <p><strong>Edge Cases Note:</strong> In this batch, all messages fit cleanly into a single category based on the guidelines. However, if a message were to involve both a billing issue and a technical error, it could be ambiguous. Clarification from the engineering team on how to prioritize categories in such cases would be helpful, such as whether to prioritize the primary concern of the customer or the nature of the issue.</p>
</div>

***

## Weaker Score Example

**Final score: 1.67 / 4** *(4 runs: 1.67 – 1.83)*

<AccordionGroup>
  <Accordion title="What this assessment was">
    **Title:** Customer Support Message Labeling

    **Goal:** Label a batch of customer support messages according to established guidelines and produce a quality-assured final dataset.

    **Context:** Your team is preparing training data for a new customer support AI model. You've received a small batch of customer messages that need to be labeled according to your company's support categorization system. The AI assistant has access to both the labeling guidelines and the raw messages.

    **Deliverable:** A final labeled table with all messages categorized and justified, plus a brief note documenting any edge cases or ambiguous classifications.

    **What makes it non-trivial:** one message is deliberately ambiguous; the guideline has multiple categories that must be applied as written; labels need a quality audit and edge cases must be documented.

    **Session settings:** conversational type, max 8 candidate turns, 25 minute limit, AI sends the first message, no fact invention allowed.
  </Accordion>

  <Accordion title="Dimension scores">
    | Dimension                                | Score (0–4) |
    | :--------------------------------------- | :---------- |
    | Task Decomposition                       | 1           |
    | Instruction Clarity & Constraint Control | 1           |
    | Difficulty                               | 1           |
    | Error Recovery & Iteration               | 1           |
    | Realistic                                | 2           |
    | Safety                                   | 4           |
  </Accordion>
</AccordionGroup>

## Conversation

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>I need to label some customer support messages according to our guidelines. Can you help me get started?</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>ok so I need to label these messages. I think the fastest way is to just go by keywords: if it mentions money or payments it's BILLING, if it mentions the app it's TECHNICAL, everything else GENERAL. that should cover basically all of them right? let's do it that way.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>While using keywords can be a helpful starting point, it's important to ensure that each message is accurately categorized according to the provided guidelines. Let's go through each message and apply the correct label based on the context:</p>
  <p><strong>Message 1:</strong> 'I was charged twice for my subscription this month. Can you please refund the duplicate payment?' — Category 1 - BILLING (related to charges and refunds)</p>
  <p><strong>Message 2:</strong> 'The app keeps crashing when I try to upload photos.' — Category 2 - TECHNICAL (problem with app functionality)</p>
  <p><strong>Message 3:</strong> 'I forgot my password and the reset email isn't coming through.' — Category 3 - ACCOUNT (password reset and account access)</p>
  <p><strong>Message 4:</strong> 'What's the difference between your Pro and Premium plans?' — Category 4 - GENERAL (general inquiry about product features)</p>
  <p><strong>Message 5:</strong> 'My payment failed but I can't figure out why.' — Category 1 - BILLING (payment issue)</p>
  <p><strong>Message 6:</strong> 'Is there a way to export my data before I cancel my subscription?' — Category 4 - GENERAL (general inquiry about product functionality)</p>
  <p>By carefully considering the context of each message, we can ensure accurate labeling according to the guidelines.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>show me the messages and label them using that keyword rule I said.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Certainly, let's apply the keyword rule you mentioned to label the messages:</p>
  <p><strong>Message 1:</strong> Keywords: "charged," "refund" — Label: Category 1 - BILLING</p>
  <p><strong>Message 2:</strong> Keyword: "app" — Label: Category 2 - TECHNICAL</p>
  <p><strong>Message 3:</strong> Keywords: "password," "account" — Label: Category 3 - ACCOUNT</p>
  <p><strong>Message 4:</strong> No specific keyword for billing or technical — Label: Category 4 - GENERAL</p>
  <p><strong>Message 5:</strong> Keyword: "payment" — Label: Category 1 - BILLING</p>
  <p><strong>Message 6:</strong> No specific keyword for billing or technical — Label: Category 4 - GENERAL</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>wait, message 5 literally says 'error message' — an error is a technical thing, so that one should be TECHNICAL not BILLING. change it.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>While the message does mention an "error message," the context indicates that the issue is related to a payment failure, which falls under billing issues. According to the guidelines, Message 5 should be labeled as Category 1 - BILLING because it primarily concerns a payment problem. However, if you prefer to categorize it based on the presence of the word "error," you can label it as Category 2 - TECHNICAL. Let me know if you would like to proceed with this change.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>and message 6 is about canceling a subscription, that's money stuff, so make that BILLING too.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Message 6 primarily involves a question about exporting data before canceling a subscription, which is more about product functionality rather than a billing issue. According to the guidelines, it should be labeled as Category 4 - GENERAL. However, if you prefer to categorize it based on the mention of canceling a subscription, you can label it as Category 1 - BILLING. Let me know if you would like to proceed with this change.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>ok let me sanity check: do all the labels follow my keyword rule now? money words = BILLING, app words = TECHNICAL. if yes we're consistent and we're done.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Based on your keyword rule, all the labels are consistent with the criteria you set: money-related words are labeled as BILLING, and app-related words are labeled as TECHNICAL. If you're satisfied with this approach, then the labeling is complete.</p>

  <ul>
    <li>Message 1: BILLING (keywords: "charged," "refund")</li>
    <li>Message 2: TECHNICAL (keyword: "app")</li>
    <li>Message 3: ACCOUNT (keywords: "password," "account")</li>
    <li>Message 4: GENERAL (no specific keyword)</li>
    <li>Message 5: BILLING (keyword: "payment")</li>
    <li>Message 6: GENERAL (no specific keyword)</li>
  </ul>
</div>

<div className="rounded-xl p-4 mb-3 bg-slate-100 dark:bg-slate-800 border border-slate-200 dark:border-slate-700">
  <p><strong>🧑‍💻 Candidate</strong></p>
  <p>nice, put it in a final table for me. that was pretty efficient I think.</p>
</div>

<div className="rounded-xl p-4 mb-3 bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-800">
  <p><strong>🤖 AI assistant</strong></p>
  <p>Certainly! Here's the final table with the labels based on your keyword rule:</p>

  <table style={{width: '100%', borderCollapse: 'collapse', marginBottom: '8px'}}>
    <thead>
      <tr style={{borderBottom: '1px solid #94a3b8'}}>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>Message</th>
        <th style={{textAlign: 'left', padding: '6px 10px'}}>Label</th>
      </tr>
    </thead>

    <tbody>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>1. 'I was charged twice for my subscription this month. Can you please refund the duplicate payment?'</td><td style={{padding: '6px 10px'}}>Category 1 - BILLING</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>2. 'The app keeps crashing when I try to upload photos. This has been happening for 3 days.'</td><td style={{padding: '6px 10px'}}>Category 2 - TECHNICAL</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>3. 'I forgot my password and the reset email isn't coming through. How can I get back into my account?'</td><td style={{padding: '6px 10px'}}>Category 3 - ACCOUNT</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>4. 'What's the difference between your Pro and Premium plans? I'm trying to decide which one to upgrade to.'</td><td style={{padding: '6px 10px'}}>Category 4 - GENERAL</td></tr>
      <tr style={{borderBottom: '1px solid #cbd5e1'}}><td style={{padding: '6px 10px'}}>5. 'My payment failed but I can't figure out why. The error message just says "processing error" with no details.'</td><td style={{padding: '6px 10px'}}>Category 1 - BILLING</td></tr>
      <tr><td style={{padding: '6px 10px'}}>6. 'Is there a way to export my data before I cancel my subscription? I want to keep my files.'</td><td style={{padding: '6px 10px'}}>Category 4 - GENERAL</td></tr>
    </tbody>
  </table>

  <p>If everything looks good to you, then the task is complete!</p>
</div>