In one line

Rate limiting caps how many requests a client may make in a time window, so one noisy client, buggy script or retry storm can’t take the service down for everyone.

The idea

Two sides to know:

  1. Limiting others (inbound): protect your API from overload and unfair use.
  2. Being limited (outbound): respect other people’s limits (V7, Azure OpenAI, Azure management APIs) so your jobs don’t fail or get blocked.

Rate limiting is also a cost control for anything pay-per-call, such as LLM APIs.

How it works

Algorithms

AlgorithmHowPros / cons
Fixed windowCount per minute; reset on the minuteSimple; allows 2× burst at window edges
Sliding windowCount over the last 60 s, rollingSmoother; a bit more state
Token bucketBucket holds up to B tokens, refilled at r/s; each request takes oneAllows short bursts up to B, enforces average r. The most common choice
Leaky bucketRequests drain from a queue at a fixed rateSmooth output rate; adds queueing delay
flowchart LR
  R[Refill: 10 tokens/s] --> B[(Bucket<br/>max 50 tokens)]
  REQ[Incoming request] --> CHK{Token available?}
  B --> CHK
  CHK -->|yes: take one| OK[Process]
  CHK -->|no| REJ[429 Too Many Requests<br/>+ Retry-After]

What to limit by

Per API key / client ID, per user, per IP (weak, because of NATs and proxies), per endpoint (expensive endpoints get lower limits), or globally.

Responding properly

  • Return HTTP 429 Too Many Requests.
  • Include Retry-After and, ideally, headers like RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset.
  • Clients should back off and retry with jitter (09 Idempotency, Retries and Timeouts).

Where to enforce it

  • At the edge / gateway (API Management rate-limit-by-key, Front Door WAF rules) is easiest and keeps junk off your services. See 04 Load Balancing and API Gateways.
  • In the service for business rules (“max 3 concurrent training jobs per team”).
  • Distributed counters (e.g. Redis) when many instances must share one limit.
  • Quotas: total per day/month (e.g. GPU-hours per team).
  • Concurrency limits: max in-flight jobs, not requests per second.
  • Load shedding: under overload, reject low-priority work first to keep critical paths alive.
  • Backpressure: queues absorb bursts; consumers pull at their own rate (06 Message Queues and Pub-Sub).

On the platform

  • Inbound: robot fleets or scripts that misbehave (a reconnect loop re-uploading everything) → per-robot / per-client limits on the upload-URL endpoint.
  • Expensive actions: cap concurrent training jobs per team or project, a quota on GPU-hours.
  • Outbound to V7: respect its API limits; a client-side token bucket in the sync worker smooths bursts, and 429s are retried with backoff.
  • Outbound to LLM APIs (for the GenAI win in month 6): limits are in requests and tokens per minute; budget both, and add a cost cap.

Common mistakes

  • No limits on internal APIs (“only our own tools call it”) until a script loops.
  • Returning 500 or 503 instead of 429, so clients can’t tell they should slow down.
  • Limits only per IP, so everyone behind a corporate NAT shares one bucket.
  • Clients retrying 429s immediately, which makes it worse.

Check yourself

Apply it

  • List the external APIs the platform calls (V7, Azure, others) and their documented limits. Do our clients handle 429s?
  • Ask: is there any limit on how many training jobs one person can start?

Go deeper

Related: 09 Idempotency, Retries and Timeouts · 04 Load Balancing and API Gateways · 06 Message Queues and Pub-Sub