In one line

A cache stores the result of expensive work so the next request is cheap. The benefit is speed and lower load; the price is stale data and the problem of knowing when to throw copies away.

The idea

Caches exist at every level:

flowchart LR
  U[User / client cache] --> CDN[CDN / edge]
  CDN --> GW[Gateway response cache]
  GW --> APP[App in-memory cache]
  APP --> R[(Distributed cache<br/>Redis)]
  R --> DB[(Database)]

Cache when data is read often, changes rarely, and is expensive to compute or fetch. Measure with the hit ratio: hits ÷ total lookups.

How it works

Read strategies

Cache-aside (lazy loading): the most common:

sequenceDiagram
  participant App
  participant Cache
  participant DB
  App->>Cache: get(dataset:42)
  alt hit
    Cache-->>App: value
  else miss
    Cache-->>App: null
    App->>DB: query
    DB-->>App: value
    App->>Cache: set(dataset:42, value, ttl=300s)
  end

Read-through: the cache library itself loads from the DB on a miss. Same idea, less app code.

Write strategies

StrategyHowTrade-off
Write-throughWrite to cache and DB togetherCache always fresh; writes slower
Write-aroundWrite to DB only; cache fills on next readGood when written data isn’t read soon
Write-behind (write-back)Write to cache, flush to DB laterFast writes; risk of data loss. Rarely right for important data

Invalidation

“There are only two hard things in Computer Science: cache invalidation and naming things.” (Phil Karlton)

Options, from simplest:

  1. TTL (time to live): entries expire after N seconds. Simple; accept staleness up to N.
  2. Explicit delete on write: when a dataset changes, delete dataset:42. Next read reloads.
  3. Versioned keys: dataset:42:v7. A new version means a new key; old ones expire naturally. Great for immutable data.
  4. Event-driven invalidation: publish DatasetUpdated events; caches subscribe and evict.

Eviction

When the cache is full: LRU (least recently used) is the usual default; LFU (least frequently used) for skewed popularity.

Cache stampede

A popular key expires → hundreds of requests miss at once → all hit the DB. Fixes: a lock so only one request recomputes, slight random jitter on TTLs, or refresh-ahead before expiry.

On the platform

  • Immutable things cache best. A versioned dataset manifest or a registered model artefact never changes, so cache it forever by version key.
  • Dashboard aggregates (images per site per day, annotation progress) → cache with a TTL of a few minutes; nobody needs real-time.
  • Thumbnails → generate once, store in Blob, serve via CDN.
  • Model weights on inference pods → cache on local disk at startup instead of downloading per request.
  • Don’t cache job status that users are actively polling unless the TTL is very short.

Trade-offs

  • Every cache is a second copy, so you’ve chosen eventual consistency for that data.
  • Another component to run, monitor and secure (Azure Cache for Redis).
  • Caches can hide performance problems until they go cold (after a deploy or restart).

Common mistakes

  • Caching without a TTL or invalidation plan.
  • Caching per-user or permission-sensitive data under a shared key, which leaks data across users.
  • In-memory caches on many pods, each with different stale values.
  • Treating the cache as the source of truth.

Check yourself

Apply it

  • Identify the slowest frequent read on the platform (often a dashboard or dataset query). Would a 5-minute TTL be acceptable to its users?
  • Check whether dataset versions and model artefacts are immutable. If yes, they’re free wins for caching.

Go deeper

Related: 08 CAP Theorem and Consistency · 10 Blob Storage Patterns · 06 Stateless vs Stateful Services