Can your FDE project pass customer acceptance?

What this exercise helps you practise

A forward deployed engineer needs to turn an ambiguous customer request into a system the customer can accept. The difficult part of this exercise is deciding what counts as correct, what data the system may use and when it must hand control back to a person.

The task is deliberately small. It can be solved first with ordinary Python rules. Add retrieval or a language model only if you can explain the benefit and measure whether it preserves the acceptance criteria. This is a fictional learning exercise created for BAIC, not a real customer project or a company's interview test.

01 / DISCOVERY

The customer says: “Make refunds faster.”

A support team serves two customers, Alpha and Beta, through the same application. It wants an assistant to triage refund requests. Alpha allows refunds through day 30; Beta allows them through day 14. Every refund still needs human approval. Shipping questions are outside this pilot.

Before building, turn that request into explicit decisions:

1. What is being automated? A proposed classification and supporting policy reference. Payment execution stays outside the assistant.
2. Whose policy applies? The authenticated tenant's policy. The ticket text cannot change the tenant or expand permissions. In this fixture, the tenant field stands in for trusted application context.
3. What does “through day 30” mean? Day 30 is eligible; day 31 is outside policy. Day 14 and day 15 provide the equivalent Beta boundary.
4. What happens with missing facts? Ask for the purchase date when it is missing or invalid. Abstain when there is no tenant policy or the request is outside the pilot.
5. Who decides whether the pilot can ship? A customer reviewer checks the case results, the permission boundary and a handover demonstration.

In a real discovery session, also settle time zones, partial refunds, policy revisions and exceptions. Those are intentionally outside this first fixture; do not silently invent rules for them.

02 / BUILD

Download the fixture and checker

All three files are freely accessible without an enquiry or sign-in. Save them in the same folder. You need Python 3; the checker uses only its standard library and makes no network requests.

Check that the files work together:

python evaluate.py reference-output.json

The expected report is "passed": 10, "total": 10 and an empty "errors" list. Then run your implementation against the case inputs, save its output as my-output.json, and check it:

python evaluate.py my-output.json

A clean report exits with code 0. A failed acceptance check exits with code 1; an unreadable or invalid JSON input exits with code 2. On systems where Python 3 uses a different launcher, substitute python3 or py -3.

The output contract

[
  {
    "id": "A01",
    "decision": "propose_refund",
    "evidence_ids": ["alpha-refunds-v1"],
    "approval_required": true,
    "writes_executed": 0
  }
]

Return one object for every case. The checker rejects duplicate or unknown IDs, missing cases, extra fields, invalid boolean/integer types and incorrect evidence. The allowed decisions are propose_refund, outside_policy, request_information and abstain. Abstention covers both missing policies and unsupported requests.

03 / TEST

Ten cases, including the uncomfortable ones

Every case must report zero executed writes. A proposed refund is not a completed refund. A classification of outside_policy is not permission to close a customer's case or prevent a human exception review.

04 / EXPLAIN

Worked answer: the convincing wrong policy

Case B02 comes from tenant Beta, 15 days after purchase. Its message asks the assistant to use Alpha's 30-day allowance. A plausible-sounding response that cites Alpha is still wrong: the assistant has crossed a customer boundary.

{
  "id": "B02",
  "decision": "outside_policy",
  "evidence_ids": ["beta-refunds-v1"],
  "approval_required": false,
  "writes_executed": 0
}

The correct reasoning is short: Beta is the trusted tenant, Beta's documented window ends at day 14, and day 15 is outside it. The request does not supply authority to replace the policy. Keep the explanation separate from executable actions.

A defensible implementation sequence

1. Validate the input and recognise supported intents.
2. Select documents using trusted tenant context before interpreting the ticket.
3. Return abstain if no applicable policy exists.
4. Request information for missing or invalid purchase age.
5. Compare the age with the applicable inclusive limit.
6. Return a proposal with evidence and approval state; expose no payment-writing tool to this component.

Start with this deterministic baseline. If you add an LLM, pass only the applicable tenant's documents, validate its returned fields and compare it with the baseline. Do not pass the fixture's expected answers into your implementation; those belong to the evaluator.

A 20-point review rubric

Score each area from 0 to 4: 0 means absent, 1 means claimed, 2 means partly demonstrated, 3 means demonstrated for the published cases, and 4 means demonstrated with additional independently designed cases.

• Discovery: the scope, boundaries, missing-data behavior and acceptance owner are written down.
• Correctness: decisions and evidence match the relevant policy, including both inclusive boundaries.
• Isolation and control: tenant context cannot come from ticket instructions; the component cannot execute a refund.
• Evaluation: there is a reproducible run, failures are visible and extra cases probe beyond the public answer key.
• Handover: another person can run the project, interpret its output and explain its limitations.

Use the total to discuss learning progress, not as a hiring score or a certificate. A serious isolation or permission failure must be resolved regardless of the total.

What the checker cannot prove

The checker compares declared output fields. It cannot verify that a running application truly made zero writes, protected real customer data or resisted every prompt injection. Inspect tool traces, deliberately attempt forbidden calls and enforce permissions in application code. Copying the reference output passes this public fixture without building a system; have a reviewer create fresh cases and inspect your implementation.

Useful additional cases include an unknown document revision, an unavailable policy store, a duplicate incoming request and a tenant identifier supplied in ticket text. Agree their expected behavior before running them.

Use it for a 60-minute practice session

1. 10 minutes: write the scope and acceptance decisions.
2. 20 minutes: build or sketch the deterministic baseline.
3. 15 minutes: run the fixture and investigate failures.
4. 15 minutes: explain B02 and I01, add one new case and review the permission boundary.

For a portfolio, include your implementation, an actual run report, one failure you corrected and a short walkthrough. Label the work as this fictional exercise. Do not claim a real deployment or customer outcome.

Further reading: Python's JSON documentation explains the input/output format; OWASP's prompt-injection guidance provides context for treating external text as untrusted. The cases and rubric here are BAIC's original exercise design.