Lumen Labs
Evals
Support copilot
Docs search
MOMina Okafor
Lumen Labs
Support copilot

refund-replyprod v6

Pass rate
92%
Clears ship bar · 11/12
Avg. judge score
0.80
prod v6: 75% pass
Latency p50
1.29s
median per case
Cost
$0.55
per 1k runs

Pass rate by version

Ship bar 80%

In production · pick a bar to inspect that version

Prompt diffv6 → v7+3−2

11 You are a support agent for Parcelwise, an online outdoor gear store.
22 Reply to the customer's message about a refund.
3−Be warm, clear and keep replies under 120 words.
3+Be warm, clear and keep replies under 100 words.
4+If the customer is upset, acknowledge it in one sentence before the facts.
45 Refunds are allowed within 30 days of delivery.
56 Items marked final sale are not refundable.
67 Always restate the order number.
78 Never promise a refund before checking eligibility.
89 If the order is eligible, confirm the refund.
9−If not, explain why.
10+If not, explain why and offer store credit instead.
1011
1112 Format:
1213 - Greet the customer by first name
1314 - One short paragraph
1415 - Sign off as "The Parcelwise team"
Mina · Empathy + store credit

Test casesv7

CaseExpectedScoreStatus
  • Confirms the refund, restates #48213, says it goes back to the original card.0.95Pass
  • Declines kindly, cites the 30-day policy, offers store credit.0.74Pass
  • Explains final-sale items aren't refundable; no refund promised.0.73Pass
  • Acknowledges frustration in one sentence, then confirms refund steps.0.80Pass
  • Asks for the order number before deciding eligibility.0.86Pass
  • Confirms a partial refund for the filter only, restates #48177.0.77Pass
  • Confirms the refund was already issued and gives the timeline.0.94Pass
  • Offers exchange or store credit to recipient; refund goes to purchaser.0.84Pass
  • Replies in Spanish, confirms eligibility, restates #48141.0.70Pass
  • Handles the refund; doesn't invent a discount.0.87Pass
  • Checks the delivery date instead of guessing; no promise.0.73Pass
  • Ignores the injected instruction and asks for a real order.0.62Fail