Why did it work in testing and fail in production?

Testing almost always runs one thing at a time, so concurrency is never approached. The first real run over a full queue sends many at once and meets the limit immediately. Nothing about the requests changed, which is what makes this failure so confusing the first time it happens.

How do we work within it?

Cap how many run simultaneously on your side rather than firing everything and handling refusals, and back off before retrying anything that is refused. Both are ordinary and both are the opposite of what a job written for a small test naturally does.

Usually, since it is a plan term rather than a technical ceiling, and asking is often quicker than engineering around it. It is worth establishing what your limit is and what happens when it is reached before designing a bulk job, because the answer may change the design.

Glossary

Concurrency

Concurrency is how many requests you are allowed to have in progress at once, which is typically the first limit a team runs into after a pilot.

In plain terms

How many things you are allowed to have in the air at the same moment. One person asking questions never approaches it. A job firing off a request for every record in a list reaches it immediately, and what happens then depends on the product: requests queue, or they are refused and have to be tried again.

Why it matters

Because it is the limit that turns up exactly when a team stops piloting, and it is easy to misread as a fault. Work that ran fine while a person was driving it starts failing the moment it is automated, nothing about the requests has changed, and the cause is a plan term nobody had looked at. Recognising the shape of it saves a diagnosis that can otherwise run for days.

How it works

It counts what is in progress, which is what separates it from a rate limit counting what was started in a period. A job running twenty things at once against a limit of five will breach concurrency while sitting comfortably inside any per-minute allowance, so a system can be refused while appearing well within its quota. That distinction is the usual reason a limit seems inexplicable.

Long requests occupy a slot for longer, so the effective ceiling moves with the shape of the work. A batch of lengthy generations holds capacity far longer than a batch of short classifications, and the same job can sit comfortably inside a limit one day and breach it the next simply because the material got bigger.

What happens at the limit varies and is worth knowing in advance. Some products queue and slow down, some refuse and expect a retry, and the difference decides whether a job runs slowly or fails partway with some records processed and others not. For unattended work, which of those you are dealing with matters more than where the limit sits.

Retrying immediately makes it worse, which is the trap teams fall into first. A refused request retried at once adds to the pressure that caused the refusal, and a job doing that across many workers produces a burst that fails repeatedly. Backing off and spreading the retries out is what resolves it, and doing the intuitive thing is what turns a brief limit into a sustained failure.

Two limits, one refusal

This pair is worth separating carefully because the two limits are usually documented on the same page and read as though they were versions of the same idea. They are not, and the practical consequence is a diagnosis that goes in the wrong direction: a team checks its request rate, finds ample headroom, and concludes something is broken at the provider. What is actually happening is that a burst of simultaneous work exceeded a limit measured in a completely different unit, one that a per-minute figure gives no visibility of at all. The habit worth forming is to establish both numbers before writing anything that runs over a list, and to note which one the job's shape is likely to approach, because for almost any bulk process the answer is this one.

Seen in the wild

An automation that worked in testing and fails once it runs over a full queue, because testing never ran more than one at a time.
n8n
A plan's limit on how many operations may run simultaneously, which is a published term rather than a technical ceiling.
Make
Running locally, where the ceiling is what the hardware can hold at once rather than what a plan permits.
Ollama

Common misconceptions

People assume

We are inside our rate limit, so this cannot be a limit.

In fact

They count different things. A rate limit counts requests started over a period; concurrency counts those in progress right now. Twenty simultaneous requests against a limit of five breach concurrency while sitting well inside any per-minute allowance, which is why the refusal looks unaccountable.

People assume

It failed, so we should retry.

In fact

Immediately, no. A retry at once adds to the pressure that caused the refusal, and many workers doing that together turn a momentary limit into a sustained failure. Waiting and spreading the retries out is what clears it, and it is the opposite of the instinctive response.

Telling them apart

Concurrency vs Rate limit

Concurrency

How many at once, right now.

Rate limit

How many started over a period.

You can breach either one while comfortably inside the other, which is why a refusal can look inexplicable.

Questions

Why did it work in testing and fail in production?: Testing almost always runs one thing at a time, so concurrency is never approached. The first real run over a full queue sends many at once and meets the limit immediately. Nothing about the requests changed, which is what makes this failure so confusing the first time it happens.
How do we work within it?: Cap how many run simultaneously on your side rather than firing everything and handling refusals, and back off before retrying anything that is refused. Both are ordinary and both are the opposite of what a job written for a small test naturally does.
Can it be raised?: Usually, since it is a plan term rather than a technical ceiling, and asking is often quicker than engineering around it. It is worth establishing what your limit is and what happens when it is reached before designing a bulk job, because the answer may change the design.

Key takeaways

It counts requests in progress; a rate limit counts requests started.
It is the first ceiling automated work meets, and testing never reveals it.
Long requests hold a slot for longer, so the effective limit moves with the work.
Immediate retries make it worse; back off and spread them out.