Rate limits cap how many requests or tokens you may use per period.
Providers enforce them to keep capacity fair across customers.
Exceeding one returns a 429 response.
Retry with exponential backoff and jitter rather than immediately.
You should apply your own limits to your users too, protecting cost and capacity.
Rate limits cap how much you may use in a period.
Providers enforce them to keep capacity fair and to protect their systems.
They are usually expressed in requests per minute and tokens per minute.
Tokens per minute is the one that bites first for large prompts.
Exceeding a limit returns a 429 response.
Retrying immediately makes it worse, since your retries are also requests.
Exponential backoff waits longer after each failure.
Jitter adds randomness so many clients do not retry in unison.
Without jitter, a burst of retries arrives together and fails together.
Cap the total retries and fail cleanly rather than retrying forever.
Respect the retry-after header when one is provided, since it beats guessing.
Queue work rather than firing it all at once, especially in batch jobs.
Apply your own limits to your own users, or one caller can consume the whole quota.
That protects both cost and availability for everyone else.
Monitor how close you run to the limit, because growth arrives without warning.
A request takes one token and is allowed if one is available. The bucket refills continuously, so idle time accumulates into permission to burst.
Those two numbers are the whole policy, and they are independent: a bucket of 100 refilling at 10 per second allows a burst of 100 and then 10 per second forever.
A fixed window counter cannot express that, and it also allows twice the limit across a window boundary, which is why the bucket is the standard.
Model APIs limit requests per minute (RPM) and tokens per minute (TPM).
Token limits are usually the binding one, and they are charged on input plus expected output, which means the limiter must estimate the completion before it runs.
Setting a large maximum output therefore reduces throughput even when the responses are short, because the reservation is taken up front.
Responses carry X-RateLimit-Remaining, X-RateLimit-Reset and Retry-After headers.
The server publishes its own state, and a client that reads those headers can slow down before it is refused rather than after.
A 429 means back off, not retry at once.
A retry that ignores the reset time makes the overload worse and can extend the penalty.
Little's law says the wait is the queue length over the arrival rate, so a queue absorbs bursts by converting rejections into latency.
If the sustained arrival rate exceeds the refill rate, the queue grows without bound and every request eventually times out.
A bounded queue with a rejection at the tail is therefore correct, and an unbounded one is a way of failing later and worse.
Priorities help when they reflect a real difference. An interactive request and a batch job in the same queue means the user waits behind work that nobody is watching.
More keys do not mean more capacity.
Splitting traffic across keys usually shares one account-level limit, so it adds complexity and no throughput.
Higher limits come from the provider, and the code's job is to use what exists without wasting it.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.