Caching: Make Stale Data a Decision

The source of truth owns the value. A cache stores a copy that is cheaper to read. Every useful cache contract answers four questions: what exact request maps to one key, when the copy becomes unacceptable, who removes or replaces it, and what happens when the cache is unavailable. Speed is the easy part. The contract is the engineering.

request → key → cached value
              ↘ miss → source of truth → fill cache → response

Useful latency sketch:
in-process memory     microseconds
same-region Redis     often under a few milliseconds
database query        depends on indexes, load, and network
remote API            usually the slowest and least predictable

Do not cache because a function looks expensive. Measure repeated reads, source latency, value size, update frequency, and the cost of being stale. A five millisecond indexed query called twice a minute does not need Redis. Adding a network cache can make it slower and harder to reason about.

Browser caches avoid a trip across the internet. A content delivery network avoids reaching your origin. A reverse proxy avoids application work. A shared cache such as Redis avoids database or API work across many app instances. An in-process cache avoids even the Redis call, but each process owns a different copy.

browser → CDN → reverse proxy → application memory → Redis → database

Question at each layer:
What work is skipped?
Who shares this copy?
How is it invalidated?
Can private data leak between users?
What is the failure behavior?

Use the lowest layer that removes the expensive work without multiplying copies you cannot invalidate. Product catalog metadata may fit a shared cache. A parsed configuration object may fit process memory. User-specific HTML should not enter a public CDN cache. Stacking every cache layer is not sophistication; it creates several places where yesterday's value can survive.

Two requests may share a cached value only if every input that can change the answer is represented in the key. Tenant, user permissions, locale, currency, feature flags, query filters, and schema version are common missing inputs. Normalize equivalent inputs so reordered query parameters do not create needless duplicates.

# Safer, inspectable key
product:v3:tenant_42:sku_9001:locale_en-GB:currency_GBP

# Dangerous: tenant and permission scope are absent
product:sku_9001

# Version the serialized shape during deployment
user-card:v7:{tenantId}:{userId}

Key versioning is the cleanest invalidation for format changes. Deploy readers that understand version seven, write version seven, and let version six expire. Do not deserialize old bytes into a new object shape and hope optional fields cover it. Keep keys readable until cardinality or key length proves hashing is necessary.

A time to live limits how long a copy may remain without another decision. It does not guarantee refresh at the deadline. Expiry usually removes or ignores the value; the next request pays for a miss. Choose the lifetime from tolerated staleness and update frequency, not from a tidy number such as one hour.

hard TTL:  10 minutes   # value cannot be served after this
soft TTL:   8 minutes   # refresh in background after this
negative TTL: 15 seconds # brief cache for “not found”

stored value = {
  data,
  refreshedAt,
  softExpiresAt,
  hardExpiresAt
}

Negative caching can protect a database when bots request nonexistent identifiers, but keep its lifetime short. A newly created object must not remain invisible behind a cached not-found result. Add random TTL jitter so millions of entries written by one batch do not expire in the same second.

Cache-aside is the practical default. Reads check cache, then load and fill on a miss. Writes commit to the database, then delete the related key. Deleting is usually safer than updating because the next read rebuilds from the authoritative state. The dangerous ordering is deleting before the database commit.

# Read path
value = cache.get(key)
if value is None:
    value = database.load(id)
    cache.set(key, serialize(value), ttl=600)
return value

# Write path
database.transaction(() => updateProduct(product))
cache.delete(productKey(product.id))  # only after commit

Here is the race: request A deletes the cache, request B misses and reads the old database row, request B fills the old value, then request A commits the new row. The stale copy now survives for a full TTL. Commit first, then delete. For stronger guarantees, write an outbox event in the same database transaction and let a worker invalidate from that durable event.

When a popular key expires, many workers can miss together and all rebuild the same value. That is a cache stampede. It often appears as a database spike exactly when the cache was meant to protect the database. Collapse concurrent work with single-flight locking, refresh popular values before hard expiry, and add jitter across keys.

value = cache.get(key)
if value is fresh: return value

if lock.tryAcquire("refresh:" + key, ttl=10s):
    try:
        value = source.load()
        cache.set(key, value, ttl=random(540s, 660s))
        return value
    finally:
        lock.release()

return staleValueIfSafe()  # or wait briefly, with a deadline

The lock needs an expiry so a crashed refresher cannot block the key forever. The source call needs a timeout shorter than the lock lifetime. Release should verify lock ownership, usually with a random token and an atomic script. For safe content, stale-while-revalidate gives users a quick old answer while one worker refreshes.

Use HTTP caching before inventing client-side storage. Freshness directives tell browsers and shared caches whether they may reuse a response. Validators let a stale cache ask whether its copy still matches. An ETag identifies a representation; Last-Modified uses time and is less precise.

# Public fingerprinted asset
Cache-Control: public, max-age=31536000, immutable

# Private account page; browser may store, CDN may not
Cache-Control: private, max-age=60
ETag: "account-42-v19"
Vary: Accept-Encoding

# Revalidation
If-None-Match: "account-42-v19"
HTTP/1.1 304 Not Modified

Do not attach a year-long lifetime to app.js unless its URL changes when its bytes change. Use a content fingerprint such as app.a81c9f.js. Use no-store for responses that must not be retained. no-cache does not mean do not store; it means revalidate before reuse.

Redis supports strings, hashes, sets, sorted sets, streams, and more. The command must match the stored type. Use expirations on cache entries, cap memory deliberately, choose an eviction policy that matches the workload, and measure serialized size. A cache with no memory budget becomes an outage scheduled by traffic growth.

# Inspect before changing
TYPE user:42
TTL user:42
MEMORY USAGE user:42

# Atomic write with expiry
SET product:42 '{"name":"Mug"}' EX 600

# Typical raw failure
WRONGTYPE Operation against a key holding the wrong kind of value

Never use Redis availability as a precondition for ordinary reads unless the product truly requires it. Set a short cache timeout, fall back to the source with bounded concurrency, and avoid retry storms. If falling back would crush the database, reject or degrade some work instead of pretending the cache is optional.

A hit ratio alone can lie. Ten million hits on cheap objects can hide one expensive key missing repeatedly. Measure request rate, hits, misses, load latency, errors, evictions, memory, value size, and age at serve. Break down only by bounded dimensions such as cache name and result, never by raw key or user identifier.

cache_requests_total{cache="product",result="hit"}
cache_requests_total{cache="product",result="miss"}
cache_load_duration_seconds{cache="product"}
cache_value_age_seconds{cache="product"}
cache_evictions_total{cache="shared_redis"}

# Failure drill
1. Add latency to cache calls
2. Make cache unavailable
3. Start with an empty cache
4. Expire the hottest keys together
5. Confirm source limits and degraded behavior

Test cold starts and cache outages on purpose. A fallback that has never met production-scale traffic is a theory. Set explicit connection and command timeouts. Keep cache health out of the application's basic readiness check if a transient cache failure would otherwise restart every app instance and amplify the incident.

Write this information beside the code that creates the cache, not in a diagram nobody updates. The person debugging stale data at two in the morning needs the exact key and ownership rules.

DESIGN
Source of truth:     database table / API / computed result
Skipped work:        exact query or computation
Key:                 every input that changes the answer
Value:               schema version and maximum size
Freshness:           soft TTL, hard TTL, allowed stale age
Invalidation:        actor, event, ordering, retry behavior
Failure mode:        bypass, stale serve, degrade, or reject
Capacity:            memory limit and eviction policy

DEBUG STALE DATA
1. Capture request identity, response, and timestamp
2. Identify browser, CDN, proxy, process, and shared caches
3. Inspect key inputs, stored value, TTL, and value age
4. Compare directly with the source of truth
5. Trace the write commit and invalidation event
6. Purge only the proven layer and key
7. Fix key, ordering, or freshness contract

DEBUG LOAD SPIKE
1. Graph hits, misses, evictions, and load latency
2. Find hot keys and synchronized expiry
3. Bound fill concurrency and source connections
4. Add single-flight refresh and TTL jitter
5. Exercise empty-cache recovery before the next incident

The opinion to keep is this: stale data must be an explicit product choice. Name its maximum age, make the cache key reviewable, invalidate after commit, and rehearse the miss path. If those decisions are absent, remove the cache until the bottleneck is worth the risk.