How long should a pilot run?

Long enough to meet the awkward cases and short enough that stopping is still a decision rather than a disruption, which for most teams is weeks rather than quarters. The end date belongs in the plan at the start, because a pilot that drifts becomes a production system nobody agreed to own or support.

People who represent the work rather than people who wanted the tool. A group of volunteers produces a flattering result that does not generalise. Including somebody sceptical and somebody who deals with the messy end is what makes the finding worth acting on across a wider group.

What is the commonest way a pilot misleads?

Handling exceptions by hand to keep the trial clean. It feels like good project management and it removes the information you most needed, because those cases are what decide whether the thing works at volume. Collect them rather than absorbing them, even where that makes the headline result look worse.

What should we measure?

Output quality and review burden together. A tool producing good drafts that each need heavy checking has not saved anything, and that effect is invisible if only the outputs are read. Measuring the checking is what separates a pilot that tested value from one that tested capability.

Glossary

Pilot project

A pilot is a limited but genuine deployment: real users doing real work with real consequences, run at a size small enough to stop and large enough to learn from.

In plain terms

A handful of people use the thing for their actual job, with actual customers or actual documents, and somebody watches what happens. The distinguishing feature is consequence: if it produces something wrong, that matters to somebody. That is exactly why it teaches you things a demonstration cannot, and exactly why it needs to be bounded before it starts rather than allowed to expand because it seemed to be going well.

Why it matters

Because it is the only stage that tests the parts nobody can anticipate: whether people actually use it, whether the awkward cases are common, whether the output survives contact with somebody who did not build it. Those questions cannot be answered by a demonstration, and they are the questions that decide whether a rollout succeeds. A pilot that skips them and simply shows the tool working has repeated the proof of concept with more people watching.

How it works

Choose participants who represent the work rather than the enthusiasm. A pilot staffed by volunteers who wanted the tool tells you it works for people who wanted it, which was never in doubt. Including somebody sceptical, and somebody who handles the messy end of the process, is what makes the result generalise beyond the room it was run in.

Bound it in time and scope before it starts, because a pilot without an end date becomes an unowned production system by default. People begin relying on it, nobody has agreed to support it, and stopping it becomes a disruption rather than a decision. The bound is what preserves the option to say no.

Decide the success criteria in advance and write them down. What proportion of output is usable, how much review it needs, what would count as failure: agreeing these afterwards means agreeing them in the light of the result, which is not agreeing them at all. This is the same discipline the use case demanded, applied at the point where it is about to be tested.

Watch the review burden as closely as the output quality, because that is where pilots quietly fail. A tool producing good drafts that each take fifteen minutes to check has not saved anything, and the effect is invisible unless somebody is measuring the checking rather than only reading the results.

Collect the awkward cases rather than routing around them. The instinct during a pilot is to handle exceptions manually and keep the trial clean, which produces a flattering result and loses the most valuable information in the exercise. Those cases are the ones that decide whether this works at scale, and they belong in the record.

What a pilot is allowed to leave out

A pilot sits deliberately in the middle, and the useful discipline is being explicit about which allowances it is still taking. It has real users and real work, so it tests adoption and quality in a way a demonstration cannot. It usually still limits who participates, how much volume flows through, and what happens when something goes wrong, and each of those limits is a question deferred rather than answered. Writing them down at the start is what makes the eventual rollout estimate honest, because the remaining distance to the right-hand end is exactly the list of allowances still outstanding. The failure this picture is drawn against is a team reading a successful pilot as though it sat at 0.95, which is how a six-week trial turns into a rollout plan with no allowance for the volume, the exceptions or the support nobody has yet provided.

Seen in the wild

Give a small team an assistant for their real drafting work for six weeks, with somebody recording how much editing each output needed.
ChatGPT
Point internal search at one department's genuine questions, including the ones people currently give up on.
Glean
Run an automation over a real but bounded slice of incoming records, so the volume is honest and the blast radius is not.
n8n

Common misconceptions

People assume

The pilot went well, so we can roll it out.

In fact

It depends what was tested. A pilot with willing participants, curated inputs and exceptions handled by hand has demonstrated the good path. Rollout meets the other paths at volume, so the question worth asking is not whether it went well but which parts of the real work it actually touched.

People assume

A longer pilot gives a better answer.

In fact

Past a point it mostly delays the decision and quietly builds dependence. What improves the answer is including harder cases and less enthusiastic people, not running the same narrow test for another two months while the tool becomes something people rely on without anybody having decided that.

Telling them apart

Pilot vs Proof of concept

Pilot

Real users, real work, real consequences. Bounded in time and scope.

Proof of concept

Whether the idea can work at all. Shortcuts allowed, nobody depending on it.

Ask whether anybody is affected if it produces something wrong. If yes, it is a pilot.

Questions

How long should a pilot run?: Long enough to meet the awkward cases and short enough that stopping is still a decision rather than a disruption, which for most teams is weeks rather than quarters. The end date belongs in the plan at the start, because a pilot that drifts becomes a production system nobody agreed to own or support.
Who should be in it?: People who represent the work rather than people who wanted the tool. A group of volunteers produces a flattering result that does not generalise. Including somebody sceptical and somebody who deals with the messy end is what makes the finding worth acting on across a wider group.
What is the commonest way a pilot misleads?: Handling exceptions by hand to keep the trial clean. It feels like good project management and it removes the information you most needed, because those cases are what decide whether the thing works at volume. Collect them rather than absorbing them, even where that makes the headline result look worse.
What should we measure?: Output quality and review burden together. A tool producing good drafts that each need heavy checking has not saved anything, and that effect is invisible if only the outputs are read. Measuring the checking is what separates a pilot that tested value from one that tested capability.

Key takeaways

A pilot has real users, real work and real consequences; that is what distinguishes it.
Bound it in time before it starts, or it becomes an unowned production system.
Staff it with people who represent the work, not people who wanted the tool.
Measure the review burden, not only the output quality.
Collect the awkward cases instead of handling them by hand to keep the trial clean.