Use an AI agent pilot checklist to define one repeatable workflow, the information it needs, the actions it may take, and the evidence that will support a scale, revise, or stop decision. Agree those boundaries before choosing tools. This guide offers ten practical planning steps, a clearly hypothetical example, and questions to discuss with the people who do the work. Suggested durations and thresholds are planning choices, not promised results. If you need help checking data access or comparing implementation options, start with the scope of the AI consulting work rather than a broad automation promise.
Why scoping matters more than tooling
An AI agent can be configured to read a ticket, query a system, draft a reply, or update a record. Define exactly which of those actions your pilot needs. That action-taking is what creates value and what creates risk. A vague pilot ("let's see what AI can do for support") produces demos that impress in a meeting and collapse on real edge cases, because nobody agreed what "working" means.
A scoped pilot, by contrast, is a decision-making instrument. It exists to produce one of three answers by a known date: scale it, fix one specific thing and retest, or stop. If your pilot plan can't produce those three answers, it isn't a pilot — it's an open-ended research project, and small teams rarely have the slack to fund one.
The 10-point AI agent pilot checklist
1. Name one workflow, not one department
Write the workflow as a single sentence with a trigger and an outcome: "When a customer emails about a delivery delay, produce a drafted response with the current shipment status attached." Avoid "improve customer service" or "automate finance". If you can't write the trigger and the outcome in one sentence, the scope is still too wide.
Decision criterion: choose enough recurring work to evaluate within your time box. For example, 20–30 items a week could be a useful planning starting point, not a required minimum or an industry benchmark. For high-volume work, start with a filtered subset.
2. Check the task has a judgeable output
Some outputs are easy to grade (was the invoice coded to the right account? yes/no). Some are not (was this strategy memo good?). Pilots on ungradable tasks drift into opinion. Pick something where a person on your team can look at the output and say "correct" or "not correct" in under a minute.
3. Confirm the data the agent needs actually exists and is reachable
List every system the agent must read from or write to, and for each one note: is there an API, who owns access, and how fresh is the data? Unclear access can delay the timeline. If order status lives only in a desktop ERP screen with no integration path, your pilot is really an integration project wearing a costume. Teams in that position often get more value from fixing the data layer first — the same groundwork that underpins any custom software build.
4. Define the human checkpoint
Decide where a person sits in the loop: reviews every output, reviews a sample, or reviews only flagged exceptions. Start at "reviews every output" for anything that touches a customer, money, or a legal record. You can loosen it later with evidence; tightening it after an incident is much more expensive.
5. Write down the baseline before you build anything
You cannot show improvement against a number you never measured. Spend the first week measuring the current state: average handling time, error rate, backlog age, cost per transaction — whatever your success metric will be. Measure it manually if you have to. A baseline captured after the agent is live is always contested.
6. Pick one primary metric and two guardrail metrics
Primary metric: the thing you're trying to move (e.g., median time from ticket arrival to first response). Guardrail metrics: the things you must not break (e.g., customer satisfaction, escalation rate, rework rate). Check the guardrails even when the primary metric improves.
7. Set a time box and a cost ceiling
Four to eight weeks is a reasonable range for a first agent pilot in a small team — that is a planning heuristic, not a benchmark. Attach a ceiling for build effort and for running costs (model/API usage, any licences). Agree what happens when you hit the ceiling: you stop and review, you don't silently extend.
8. Write the exit criteria as numbers
Fill in the blanks before you start: "We scale if the agent handles ≥ ___% of cases without human edits and the escalation rate stays below ___. We stop if ___ after week 6." Teams that skip this end up negotiating success after the fact, which usually means the loudest voice decides.
9. Decide what the agent is not allowed to do
List prohibited actions explicitly: no refunds above a threshold, no sending external email without approval, no deleting records, no writing to the production ledger. Also decide how the agent should behave when it's uncertain — escalate, ask a clarifying question, or return "unknown". Define an explicit response to missing or conflicting information instead of relying on a plausible-looking answer.
10. Name an owner and a weekly review slot
One person owns the pilot and has 30 minutes a week with the people doing the work to review failure cases. Not a steering committee. For example, a review might reveal recurring failures caused by one malformed data field. Read the actual examples before deciding whether the model needs to change.
Illustrative example (hypothetical, not a customer result)
Imagine a 14-person wholesale distributor. Inbound email volume is roughly 120 messages a day, and about 35 of those are "where is my order?" Staff currently check the ERP, copy a tracking link, and reply — an estimated 4 minutes each.
Their scoped pilot: an agent that classifies inbound email, and for order-status messages only, looks up the order in the ERP and drafts a reply for a human to approve in the shared inbox. Baseline measured in week one. Primary metric: median first-response time. Guardrails: incorrect-order-referenced rate, and customer complaint count. Exit criteria: scale if ≥70% of drafts are sent with no edits and incorrect-order rate is under 2%; stop if either condition fails by week six. Prohibited actions: no sending without approval, no touching pricing or credit terms.
Note that this scope excludes returns, credit queries, and anything with an attachment. That exclusion is the point. Those categories can become pilot two — if pilot one earns the right.
Common mistakes small teams make
Scoping to impress rather than to learn. The most impressive demo is usually the hardest workflow. Pick the boring one with volume.
Treating accuracy as binary. Agents fail in grades: slightly wrong tone, right answer with a hallucinated detail, correct but 40 seconds too slow. Build a simple rubric — correct / correct-with-minor-edit / wrong / refused — and count each bucket.
Forgetting the cost of review. If a human spends three minutes verifying what the agent did in a task that used to take four, you've achieved almost nothing. Measure end-to-end human time, not agent time.
Skipping the "what if it's wrong in production" conversation. Decide now who notices, how they notice, and how a bad action gets reversed.
Letting the pilot become permanent infrastructure by accident. Even an experimental pilot needs shared ownership, scoped credentials, and an exit plan. A personal key and an undocumented spreadsheet make handoff fragile; plan the handoff before starting.
What a good pilot produces, even when it fails
A properly scoped pilot that ends in "stop" is still a good investment, because it returns three assets: a measured baseline you didn't have before, a documented map of which systems can and can't be reached programmatically, and a labelled set of real cases showing exactly where automation breaks. That material makes the next attempt cheaper. An unscoped pilot that ends in "it was interesting" returns nothing.
Frequently Asked Questions
Should we build the pilot ourselves or use an off-the-shelf agent product?
For a first pilot, prefer whatever gets you to real cases fastest — often an existing platform with a connector to your systems. Build custom when the workflow depends on logic or data that no vendor can reach, or when the vendor's pricing model doesn't survive your volume. Decide this after step 3 of the checklist, because data reachability usually settles the question for you.
How many people do we need on an AI agent pilot?
Typically three roles, not three full-time people: an owner who makes decisions, someone technical who can wire up access and iterate on prompts and tools, and at least one practitioner who does the work today and grades outputs. The practitioner is the role teams most often omit and most regret omitting.
What if our data is messy — should we clean it first?
Run a small sample test before committing to a cleanup project. Agents tolerate some inconsistency in free text but are brittle with inconsistent identifiers, duplicate records, and missing keys. If your failure analysis in week two points repeatedly at data quality, that's your real first project, and it will pay back beyond the agent.
How do we handle security and sensitive data in a pilot?
Scope the data the agent can see as narrowly as the task allows, confirm your vendor's data-retention and training settings in writing, and keep a log of every action the agent takes. Involve whoever handles compliance in your organisation before the pilot starts, not at the review.
When is a team not ready for an agent pilot at all?
If nobody can describe the target workflow step by step, if the required systems have no programmatic access and no budget to add it, or if no one has 30 minutes a week to review failures — wait. Fix those first; the pilot will be cheaper and more honest afterwards.
Next step
Take your top three candidate workflows and run each through points 1, 2, and 3 of the checklist — one sentence, judgeable output, reachable data. Usually one survives cleanly, and that's your pilot. If none survive, you've learned something more valuable than a demo would have told you.
If you'd like a second opinion on which workflow to pilot first, or help mapping what your current systems can realistically expose to an agent, the team at software4 can work through the checklist with you and your practitioners.