You have limits from the provider and limits you impose on users. Stay under the provider’s token-per-minute ceiling and cap per-user usage so one caller can’t drain your quota.
Why: the provider limits YOU (requests and tokens per minute), and you must limit your USERS so a single caller cannot exhaust your quota or run up the bill. When: enforce both — provider-facing concurrency control and user-facing per-user caps. Where: hitting the provider limit returns 429s to everyone; a per-user cap contains the damage to the abuser.
PROVIDER -> YOU requests/min, tokens/min ceilings
exceed -> 429 for everyone
YOU -> YOUR USERS per-user request/token caps
one abuser can't drain the shared quota
Enforce BOTH. They protect against different failures.Why: a per-user token budget in a shared store (Redis) stops one account from consuming everyone’s capacity and gives you a lever for tiered plans. When: check and decrement the budget before the call, reject over-limit requests with a clear 429. Where: key by user and a time window so the budget refills automatically.
import time, redis
r = redis.Redis()
def check_budget(user_id: str, tokens: int, limit_per_min=50_000) -> bool:
key = f"budget:{user_id}:{int(time.time() // 60)}" # per-minute window
used = r.incrby(key, tokens)
r.expire(key, 120)
return used <= limit_per_min
if not check_budget(user_id, estimated_tokens):
raise Exception("Rate limit exceeded — try again shortly.") # return HTTP 429Why: firing unlimited concurrent calls guarantees 429s under load — a concurrency limit or token-bucket smooths your request rate to what the provider allows. When: bound in-flight requests to the model API, especially in batch jobs and agents that fan out. Where: combine this with the retry/backoff layer so the rare 429 that slips through still recovers.
import asyncio, anthropic
client = anthropic.AsyncAnthropic()
gate = asyncio.Semaphore(5) # at most 5 concurrent calls to the provider
async def bounded_call(messages):
async with gate: # blocks the 6th until a slot frees up
return await client.messages.create(
model="claude-opus-4-8", max_tokens=512, messages=messages)
# Fan out a batch WITHOUT blowing past the provider's rate limit.
async def run_batch(batch):
return await asyncio.gather(*(bounded_call(m) for m in batch))