[{"content":"Imagine that you have just changed a product\u0026rsquo;s price. One request is still reading the old price from the database; another has already deleted the old cache entry. Which request writes the cache last? Now imagine thousands of visitors opening the same page just as its cache key expires. How many database queries will you get?\nThose are the questions I find more interesting than another SET/GET tutorial. This article starts with a small product catalog and follows three problems: stale cache fills, a database overwhelmed by cache misses, and rate limits that behave differently from what their configuration suggests. It ends with eight failures worth simulating before Redis sits on an important request path.\nThe examples use Redis 7.x, Python 3.10+, and redis-py. Numbers in the simulations explain the mechanism; they are not measurements from a real service. If Redis itself is new to you, read the fundamentals first. The persistence guide covers RDB and AOF in depth.\n1. Start with a Product Page Suppose a product page reads its name and price from PostgreSQL. Many people view the page, but prices change only occasionally. Asking PostgreSQL the same question for every view wastes work. With cache-aside, the application asks Redis first:\nflowchart LR U[User] --\u0026gt; API[Application API] API --\u0026gt;|GET product:42| R[(Redis)] R --\u0026gt;|cache hit| API R -.-\u0026gt;|cache miss| API API --\u0026gt;|SELECT| DB[(PostgreSQL: source of truth)] DB --\u0026gt; API API --\u0026gt;|SET with TTL| R On a miss, the application reads PostgreSQL and stores a temporary copy in Redis. When a price changes, it commits the new price to PostgreSQL and then deletes the cache key. Only products that people actually view enter the cache. Redis cache-aside guidance.\nThree different reasons a key disappears Mechanism Who triggers it? What it means for product:42 TTL expiration Time passes Redis treats the cached value as expired after, say, 60 seconds. Invalidation The application notices a change The price changes, so the application calls DEL rather than waiting for TTL. Eviction Redis needs memory Redis may remove the key before its TTL ends. A 60-second TTL sets an upper bound on the life of that cache entry if nobody overwrites or removes it first. It does not promise that the price stays accurate for 60 seconds. Redis keeps expiration against real time, so time continues while the server is down. Eviction can remove a key even sooner when RAM is tight. Key expiration, key eviction.\n1 2 3 GET product:42 # read cache SET product:42 \u0026#39;{\u0026#34;name\u0026#34;:\u0026#34;Blue shirt\u0026#34;}\u0026#39; EX 60 TTL product:42 # seconds left In this example, PostgreSQL owns the official price and Redis makes common reads cheaper. If a cache key disappears, the app can fetch the price again. Ask the same question for every other Redis key in your system: where would we rebuild this value from?\nRedis also offers different data types for different operations. A small cached JSON document may be a String; fields that change independently may fit a Hash; unique IDs fit a Set; ranks fit a Sorted Set; work requiring consumer acknowledgment may fit a Stream. The type helps with the operation, but it does not erase limits on RAM, durability, or traffic. Redis data types.\n2. How an Old Read Can Put an Old Price Back in Cache The usual read and write flow On the first read, GET finds no key. The application reads PostgreSQL and uses SET ... EX 60. Later reads use Redis. For an update, commit PostgreSQL first and then DEL the cache. If you delete the cache before the database commit, another request may miss, read the old database value, and fill the cache with it again. Unfortunately, committing first still leaves another race.\nThere are two smaller details worth fixing before we get to that race:\nGET returns nil both when a product has not been cached and when that product does not exist. Repeated requests for nonexistent IDs can repeatedly hit the database. A short-lived marker such as __NOT_FOUND__ avoids that; delete it if the product is later created. Do not use an empty string as the marker if empty is a valid value. Redis missing-record guidance. A cache key must include every input that changes the answer. If price varies by tenant or currency, product:42 is too broad. catalog:tenant:7:product:42:currency:VND:v1 is safer, though more variants cost more RAM. The v1 suffix makes it easier to change the cached format later. Redis cache-aside guidance. 1 2 3 GET catalog:product:404 # nil SELECT ... WHERE id = 404 # no row SET catalog:product:404 __NOT_FOUND__ EX 10 A useful race described by Meta In \u0026ldquo;Cache made consistent\u0026rdquo;, Meta\u0026rsquo;s engineers describe a slow cache fill: a read obtains an old value, an update happens, and then that delayed read writes the old value into cache. Their article is about TAO/Memcache, not a Redis incident. The ordering problem also applies to this Redis cache-aside design.\nSay the old price is 100 and the new price is 120:\nTime Request A: read Request B: update t1 Redis miss; start database read t2 Read 100, then pause before SET t3 Commit 120 to PostgreSQL t4 DEL product:42 t5 SET product:42 = 100 EX 60 PostgreSQL now says 120, while Redis says 100. The DEL was executed, yet the late reader put the old price back. TTL limits the mistake only if the new cache write actually includes a TTL. If request A forgets EX, the wrong price can remain much longer. A lagging PostgreSQL read replica can introduce a related problem: it may return 100 after invalidation has finished.\nLet a version number reject late fills Add product:{42}:version to Redis. Before reading PostgreSQL, request A remembers the current version. After the read, A fills the cache only if that version has not changed. When B commits the new price, it increments the version and deletes the cache within one Redis Lua script. Other Redis commands cannot run between the two script operations. Redis Lua scripting.\nsequenceDiagram participant A as Reader A participant R as Redis participant D as PostgreSQL participant B as Writer B A-\u0026gt;\u0026gt;R: GET cache (miss), GET version (= 0) A-\u0026gt;\u0026gt;D: SELECT price (= 100) B-\u0026gt;\u0026gt;D: UPDATE price = 120, COMMIT B-\u0026gt;\u0026gt;R: script increments version and deletes cache A-\u0026gt;\u0026gt;R: script: fill only if version is still 0 R--\u0026gt;\u0026gt;A: Reject: version is now 1 A-\u0026gt;\u0026gt;D: Read the new price (= 120) This blocks one specific race: A cannot fill Redis with a value it fetched before B changed the version. It does not make PostgreSQL and Redis one transaction. Another request can still see the old cache between the database commit and Redis invalidation. A lagging database replica can also return old data under the new version. At checkout, where the latest price may be mandatory, read the authoritative source or use a stronger application-level consistency design.\nPython lab: force the bad ordering We can make the race visible by pausing A just after it reads 100, letting B change the price, and then resuming A. You need Redis and Python. With Docker, run docker run --rm -p 127.0.0.1:6379:6379 redis:7.4 in one terminal. In another, run python -m pip install redis, save the code below as demo_cache.py, and run python demo_cache.py. SQLite stands in for PostgreSQL. The script deletes two exercise keys under demo:product:{42} and writes demo_cache.sqlite3 in the current directory; use a Redis instance dedicated to the lab.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 import json import sqlite3 import threading import redis DB = \u0026#34;demo_cache.sqlite3\u0026#34; CACHE = \u0026#34;demo:product:{42}:value\u0026#34; VERSION = \u0026#34;demo:product:{42}:version\u0026#34; r = redis.Redis(host=\u0026#34;localhost\u0026#34;, port=6379, decode_responses=True) FILL_IF_CURRENT = \u0026#34;\u0026#34;\u0026#34; local current = redis.call(\u0026#39;GET\u0026#39;, KEYS[2]) or \u0026#39;0\u0026#39; if current ~= ARGV[1] then return 0 end redis.call(\u0026#39;SET\u0026#39;, KEYS[1], ARGV[2], \u0026#39;EX\u0026#39;, ARGV[3]) return 1 \u0026#34;\u0026#34;\u0026#34; INVALIDATE = \u0026#34;\u0026#34;\u0026#34; redis.call(\u0026#39;INCR\u0026#39;, KEYS[2]) redis.call(\u0026#39;DEL\u0026#39;, KEYS[1]) return 1 \u0026#34;\u0026#34;\u0026#34; def db_read(): conn = sqlite3.connect(DB) try: return conn.execute(\u0026#34;SELECT price FROM product WHERE id = 42\u0026#34;).fetchone()[0] finally: conn.close() def read_product(paused=None, resume=None): while True: cached = r.get(CACHE) if cached is not None: return json.loads(cached) version = r.get(VERSION) or \u0026#34;0\u0026#34; price = db_read() if paused is not None: # force the race ordering in this lab paused.set() if not resume.wait(timeout=5): raise TimeoutError(\u0026#34;writer did not finish\u0026#34;) paused = None stored = r.eval( FILL_IF_CURRENT, 2, CACHE, VERSION, version, json.dumps({\u0026#34;price\u0026#34;: price}), 60 ) if stored == 1: return {\u0026#34;price\u0026#34;: price} print(\u0026#34;Version changed while reading; retrying\u0026#34;) def update_price(price): conn = sqlite3.connect(DB) try: conn.execute(\u0026#34;UPDATE product SET price = ? WHERE id = 42\u0026#34;, (price,)) conn.commit() finally: conn.close() r.eval(INVALIDATE, 2, CACHE, VERSION) if __name__ == \u0026#34;__main__\u0026#34;: conn = sqlite3.connect(DB) try: conn.execute(\u0026#34;CREATE TABLE IF NOT EXISTS product (id INTEGER PRIMARY KEY, price INTEGER)\u0026#34;) conn.execute(\u0026#34;INSERT OR REPLACE INTO product (id, price) VALUES (42, 100)\u0026#34;) conn.commit() finally: conn.close() r.delete(CACHE, VERSION) paused, resume = threading.Event(), threading.Event() reader = threading.Thread( target=lambda: print(\u0026#34;Reader received:\u0026#34;, read_product(paused, resume)) ) reader.start() if not paused.wait(timeout=5): raise TimeoutError(\u0026#34;reader did not reach the pause point\u0026#34;) update_price(120) resume.set() reader.join() print(\u0026#34;Final cache:\u0026#34;, r.get(CACHE)) You should see Version changed while reading; retrying, followed by {'price': 120}. Both Redis keys contain {42} so they share a hash slot if you run the example on Redis Cluster; a multi-key script requires its keys to be in the same slot. Redis Cluster hash tags.\nRead the code in three steps:\nread_product() obtains the version before SQLite returns a price. If the version changes while it reads, FILL_IF_CURRENT returns 0, and the reader starts again. update_price() commits SQLite and calls INVALIDATE. INCR version and DEL cache run together inside Redis, with no other Redis command between them. Two threading.Event objects pause A after it reads 100. B changes the price before A resumes. Without those deliberate pauses, the race depends on timing and can be hard to observe. For a useful experiment, replace FILL_IF_CURRENT with redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3]); return 1, then rerun the script. The reader can return 100 even though SQLite holds 120. Restore the version check and compare the results.\nWhere the version approach ends There is still a gap between committing SQLite and invalidating Redis. If the application stops in that gap, stale cache may survive until its TTL ends. A Redis outage can also make invalidation fail. For data that needs stricter freshness, an outbox or CDC pipeline can record the database change and retry invalidation. Critical paths may read directly from the authoritative database.\nThe version key itself matters. If Redis evicts it while an old reader is working, the version may appear to return to 0, and that reader may incorrectly fill the cache. An allkeys-lru cache instance is not automatically a safe place for a long-lived correctness token. A stricter design can obtain the version from the database or a durable change log and use Redis only as a fast copy. That adds I/O and complexity, and it should be tested with the actual Cluster and failover setup.\nThe product question is simpler to state: may the catalog show an old price for one second, one minute, or not at all at checkout? The same product can use cache on a listing page and an authoritative read at order confirmation.\n3. The Cache Is Correct, but the Database Still Falls Over A high hit rate can hide a dangerous peak Imagine a popular product\u0026rsquo;s cache key expiring just as 2,000 requests arrive. All 2,000 miss Redis and run the same SELECT on PostgreSQL. One database lookup has become 2,000 lookups. Several keys with similar TTLs can expire together and produce the same effect.\n1 2 Before expiration: 2,000 requests -\u0026gt; cache hit -\u0026gt; almost no database reads After expiration: 2,000 requests -\u0026gt; cache miss -\u0026gt; up to ~2,000 database reads The daily cache hit rate could still be 99%. The remaining 1% might cluster into a few seconds after a deployment, a failover, or the expiration of a very popular key. The database has to survive the short peak, not the daily average. Slow queries make requests wait longer; more requests accumulate, time out, and may retry, extending the peak.\nFour ways to reduce that peak Choose based on how stale the data may be:\nAdd jitter to TTLs. Let keys live 60–75 seconds rather than exactly 60. This spreads expirations across many keys. It does not solve 2,000 requests racing on one key. Coalesce identical requests. Within one application process, let one request load a missing key while others wait for its result. This is often called single-flight. With 20 application instances, you may still get 20 database queries. A short Redis lock such as SET lock:key token NX PX 2000 can coordinate instances; callers that miss the lock can wait briefly and retry the cache, or receive stale data if the product permits. Redis distributed locks. Serve stale data while refreshing. One worker refreshes the value in the background while readers temporarily receive the old version. This is stale-while-revalidate. It helps with load but may be wrong for account balances or final checkout prices. Protect the database directly. Cap concurrent cache-miss queries, set timeouts, and choose a controlled degraded response. A lock can expire before a slow database query finishes, so it is not proof that only one worker queried the database. Situation First idea to try What to measure Many keys expire together TTL jitter Expirations and misses per second One very popular key Single-flight, then perhaps a lock across instances Database queries when that key expires Data may be a few seconds old Serve stale while refreshing Age of data actually returned to users Database is near its limit Concurrency cap and timeouts Active connections and database p99 A lock must belong to its owner Worker A acquires a lock, but its database read takes longer than the lock\u0026rsquo;s TTL. The lock expires, and B acquires a new one. If A then calls DEL lock:key, it deletes B\u0026rsquo;s lock. Give each owner a random token and remove the lock only when the token still matches. The compare-and-delete step must run as one Lua script:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import secrets token = secrets.token_hex(16) acquired = r.set(\u0026#34;demo:lock:product:42\u0026#34;, token, nx=True, px=2000) RELEASE_IF_OWNER = \u0026#34;\u0026#34;\u0026#34; if redis.call(\u0026#39;GET\u0026#39;, KEYS[1]) == ARGV[1] then return redis.call(\u0026#39;DEL\u0026#39;, KEYS[1]) end return 0 \u0026#34;\u0026#34;\u0026#34; if acquired: try: # Only this worker loads the cache; see the version check in section 2. pass finally: r.eval(RELEASE_IF_OWNER, 1, \u0026#34;demo:lock:product:42\u0026#34;, token) This snippet shows safe release, not a complete cache implementation. A worker that fails to acquire the lock still needs a policy: wait briefly and retry GET, return an allowed stale value, or return an error the caller can retry. Do not wait forever. Choose a lock TTL based on the slower end of database read times, with margin. A process pause or network delay can still outlast that TTL. The lock reduces duplicate cache fills; by itself it does not guarantee an external task runs exactly once.\nFour cases worth testing A popular key is manually deleted. This can produce the same database surge as expiration. Test it under simulated traffic, not only while the service is idle. Clients repeatedly request nonexistent IDs. Without a short \u0026ldquo;not found\u0026rdquo; cache entry, every request hits the database. One value grows huge. A large JSON reply costs network time even when every lookup is a hit. Measure MEMORY USAGE key and response size. Redis MEMORY USAGE. Errors cause synchronized retries. If Redis slows down and every application retries immediately while PostgreSQL is also slow, the system amplifies its own load. Cap retries and add backoff. Measure misses for individual popular keys and the resulting database queries. A single daily hit-rate number will not reveal the peak.\n4. Rate Limits: How Redis Can Protect an API When one customer calls your API too often, a per-second limit sounds like the obvious fix. But what if each call runs for 30 seconds? Even a modest arrival rate can occupy every worker.\nIn \u0026ldquo;Scaling your API with rate limiters\u0026rdquo;, Stripe describes four layers: requests over time, requests currently in flight, load across the fleet, and load on a worker group. They used Redis for a token-bucket rate limiter. Before enforcing it, they measured which requests would have been denied in a monitoring-only mode. They also planned a quick way to disable it and decided what to do when Redis failed.\nLayer Stripe describes What it controls Example Request rate Too many calls in a period A broken client loops on the API. Concurrent requests Too much work in flight for one customer Twenty reports each take 30 seconds. Fleet load Resource pressure across the service Defer reports to preserve payments. Worker load A worker group is already busy Reject less important work before the queue grows. Requests arriving per second and requests still running are different numbers. One token bucket cannot solve every overload problem.\nChoose the shape of the limit Algorithm How to picture it Useful property Trap Fixed window Count calls in each fixed interval Simple and compact Calls near the end of one interval and start of the next can form a burst. Sliding window Count calls in a window that moves with time Smoother limit More timestamps or an approximation are needed. Token bucket Tokens refill a bucket with fixed capacity Allows a short burst while controlling long-term rate Updates must be atomic; losing the key refills the bucket. A fixed-window limit of ten requests a minute may allow ten at 12:00:59 and another ten at 12:01:00: twenty requests in two seconds. A bucket that holds ten tokens and refills five per second permits an initial burst of ten, then makes later requests wait for new tokens. Redis rate-limiting guidance.\nBuild one token bucket In our lab, every user has a bucket of at most ten tokens. Each request consumes one; Redis adds five tokens per second, up to the maximum. Ten immediate calls can pass when the bucket is full. Over a longer period, the rate approaches five calls per second. These are example numbers, not Stripe\u0026rsquo;s configuration.\nflowchart LR Q[Request from user 42] --\u0026gt; S[Lua script in Redis] S --\u0026gt; T{At least one token?} T --\u0026gt;|Yes: subtract one| API[Allow request] T --\u0026gt;|No| E[HTTP 429 and Retry-After] Why use Lua? If the application runs HGET, calculates, and then runs HSET, two simultaneous requests can both see the last token and both pass. A script runs the read, decision, and write as one Redis operation. A pipeline reduces network trips but cannot make a decision that depends on a preceding reply atomic. Redis pipelining and scripting.\nThe script updates the bucket with:\n1 new_tokens = min(capacity, old_tokens + elapsed_time * refill_rate) If new_tokens \u0026gt;= 1, it subtracts one and allows the request. Otherwise, it reports how many milliseconds the caller should wait. capacity and refill_rate must be positive server-side configuration values; do not accept a caller-supplied limit.\nPython lab: let Redis make the decision Save the code below as demo_limit.py. It uses the same local Redis and redis Python package as the cache lab:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 import redis r = redis.Redis(host=\u0026#34;localhost\u0026#34;, port=6379, decode_responses=True) TOKEN_BUCKET = \u0026#34;\u0026#34;\u0026#34; local key = KEYS[1] local capacity = tonumber(ARGV[1]) local refill = tonumber(ARGV[2]) -- tokens per second local clock = redis.call(\u0026#39;TIME\u0026#39;) local now = tonumber(clock[1]) + tonumber(clock[2]) / 1000000 local previous = redis.call(\u0026#39;HMGET\u0026#39;, key, \u0026#39;tokens\u0026#39;, \u0026#39;updated_at\u0026#39;) local tokens = tonumber(previous[1]) or capacity local last = tonumber(previous[2]) or now tokens = math.min(capacity, tokens + math.max(0, now - last) * refill) local allowed = 0 if tokens \u0026gt;= 1 then allowed = 1 tokens = tokens - 1 end redis.call(\u0026#39;HSET\u0026#39;, key, \u0026#39;tokens\u0026#39;, tokens, \u0026#39;updated_at\u0026#39;, now) redis.call(\u0026#39;PEXPIRE\u0026#39;, key, math.ceil(capacity / refill * 1000)) local retry_ms = 0 if allowed == 0 then retry_ms = math.ceil((1 - tokens) / refill * 1000) end return {allowed, math.floor(tokens), retry_ms} \u0026#34;\u0026#34;\u0026#34; bucket = r.register_script(TOKEN_BUCKET) key = \u0026#34;demo:limit:user:42\u0026#34; r.delete(key) for n in range(12): allowed, remaining, retry_ms = bucket(keys=[key], args=[10, 5]) print(n + 1, \u0026#34;ALLOW\u0026#34; if allowed else \u0026#34;429\u0026#34;, remaining, retry_ms) Run python demo_limit.py. The first calls pass; after tokens run out, the output shows 429. Time still passes during the loop, so the exact number allowed can vary slightly by machine. The script uses Redis\u0026rsquo;s clock so application servers do not depend on their own possibly skewed clocks. PEXPIRE removes an inactive user\u0026rsquo;s bucket after enough time has passed for it to refill; the next call starts with a full bucket. An API can turn retry_ms into a Retry-After header, rounding a denied request up to at least one second.\nThree code details matter:\nTIME runs in Redis. Two application servers with clocks a few seconds apart still calculate refills from the same clock. HMGET fetches the remaining tokens and the last update time. A missing key means a full bucket. That is a product decision: after failover loses a bucket, the user gets another initial burst. PEXPIRE cleans up inactive users\u0026rsquo; state. For an active user, each call refreshes the TTL because the bucket is still needed. In a real API, 429 is an HTTP status; the lab only prints it. Middleware also needs an error body, Retry-After, metrics by user or endpoint, and care not to log sensitive identities. Start in monitoring-only mode before enforcing the limit, as Stripe describes.\nWhen the limiter itself fails Test these cases before relying on the limiter:\nIdentity is ambiguous. Limiting by IP can group many people behind one NAT. Limiting by account requires identifying the account first. A large customer may need a different quota. The bucket disappears. Failover or data loss can create a fresh, full bucket. Do not make a Redis bucket the only protection for a financial or security-critical constraint. Redis is unavailable. Fail open allows traffic so the API remains available, but risks overload. Fail closed protects a resource, but can reject legitimate users. Stripe chose fail open for their limiter; your choice may differ by endpoint. Requests run a long time. A token bucket limits arrivals, not the number of calls currently running. Add a concurrency limit where necessary. The script gets expensive. Lua executes atomically from other clients\u0026rsquo; perspective, so a long script delays other Redis commands. Keep the script short and avoid loops over large data. Redis Lua scripting. Your policy can vary by API. A public read endpoint may temporarily allow traffic with a local per-instance cap when Redis fails. Login or transaction endpoints may need another protection layer in the gateway or application. If a quota is contractual or tied to money, include failover and multi-region behavior in the design.\n5. The Operational Questions That Follow Memory: a good hit rate can be expensive Redis keeps active data in RAM. maxmemory sets a limit, and an eviction policy determines what happens as Redis reaches it. For a rebuildable cache, test allkeys-lru or allkeys-lfu. With noeviction, some writes fail when memory is exhausted. Test with realistic numbers of keys and realistic value sizes. A million tiny keys still have metadata overhead; one enormous value is expensive to send over the network. Redis eviction, MEMORY USAGE.\nAn evicted product-cache key can be reloaded from PostgreSQL. An evicted version key from section 2 can undermine the stale-fill protection. Do not mix disposable cache entries and important coordination state under a broad eviction policy without a plan. Separate Redis instances or keep important state in a durable source where necessary.\nDurability: restart changes different keys differently Redis can persist data with RDB, AOF, or both. RDB captures a point-in-time snapshot, so changes after the last snapshot may be lost. AOF records changes; appendfsync everysec usually trades some performance for a potential loss window of roughly a second in certain failures. Define RPO (how much data may be lost) and RTO (how long recovery may take) for each workload. A persistence setting does not replace backups and restore practice. The RDB/AOF guide walks through the mechanisms. Redis persistence.\nIf the product cache disappears, PostgreSQL can refill it, but the database may face a surge. If a token bucket disappears, a user may receive an extra burst. If Redis holds the only copy of business data, the consequences are much more serious. Evaluate keys by their role rather than treating everything in Redis as equally disposable or equally durable.\nLatency: is Redis slow, or is the path to Redis slow? Even if Redis processes a command quickly, the application still pays for network travel, connection-pool waits, JSON encoding, and the reply size. MGET or a pipeline can reduce network waits for independent commands. When the next action depends on a read result, consider a Lua script or transaction instead. Keep pipeline batches reasonable; Redis buffers replies until clients read them. On large systems, use SCAN rather than KEYS *, and remember that a scan is not a fixed snapshot while keys change. Redis pipelining, latency guidance, SCAN.\nCompare p99 latency at the application with command time in Redis. If Redis is fast but the app is slow, investigate network trips, connection waits, and JSON decoding. If Redis is slow too, investigate commands that traverse large collections, long scripts, big values, persistence work, and memory pressure.\nCluster: keys used together need the same slot The cache lab uses demo:product:{42}:value and demo:product:{42}:version. They share {42}, so Redis Cluster places them in the same hash slot. Without the braces, the keys could land in different slots and the multi-key script could fail. Put the hash tag around the unit of data you must update together. If every product uses {product}, all products collapse into one slot and defeat sharding. Redis Cluster specification.\nObserve what users and operators actually need to know Question Useful signals Sign worth investigating Is cache reducing database work? Hits and misses by API/key; database reads per second Misses jump after a deployment or key expiration. Is Redis running out of RAM? used_memory, maxmemory, evicted_keys, sample key sizes Important keys disappear or writes fail. Are users seeing latency? Application p50/p95/p99, Redis latency, SLOWLOG p99 grows while the average looks fine. Is the limiter making good decisions? Allows and denies by API/customer; monitoring-only results One group is blocked unexpectedly or overload is not limited. Can the database survive Redis errors? Redis timeouts, database connections, database p99 A Redis problem cascades into database overload. INFO stats exposes keyspace_hits, keyspace_misses, and evicted_keys, but they describe the entire Redis instance. They cannot tell you which misses came from product cache versus the rate limiter. Add application metrics by feature. Redis INFO.\n6. Eight Failures You Are Unlikely to See in a Tiny Demo These failures become more likely with several workers, a failover, or a key that quietly grows for months. For each one, ask: what happened, why was it easy to miss, and what should we test?\n6.1. SET silently removes the old TTL One service creates product:42 with SET product:42 ... EX 60. Another overwrites the same key with SET product:42 ... and assumes the old 60-second TTL remains. It does not: a plain SET removes the previous expiration unless the new command sets a TTL or uses KEEPTTL. If a later invalidation is missed, an outdated cache entry can live indefinitely. Redis SET.\n1 2 3 4 SET demo:ttl old EX 60 TTL demo:ttl # positive SET demo:ttl new TTL demo:ttl # -1: the key exists without an expiration Check: make every cache-writing path set a TTL in the write itself, or intentionally use KEEPTTL when retaining the old deadline is correct. Sample cache keys and look for TTL = -1. This is also why the section 2 lab puts SET ... EX in the script that checks the version.\n6.2. An expired event is not a business clock Suppose an order reserves stock for 15 minutes, and your app waits for Redis to publish an expired key event before releasing that stock. Redis emits the event when it actually deletes the key, which need not happen at the exact instant its TTL reaches zero. If the subscriber is disconnected, Pub/Sub will not replay the missed notification. In a Cluster, keyspace notifications are local to each node. Redis keyspace notifications.\nBetter approach: TTL can clean up temporary Redis state. If releasing stock must happen, store the deadline in a durable source and have a worker periodically find overdue reservations. Make the worker safe to run twice for the same order. The Redis notification can wake it early, but periodic reconciliation catches missed events.\n6.3. Two workers can both believe they hold a lock Worker A acquires SET lock:order:42 tokenA NX PX 5000. The Redis primary dies before that key reaches a replica. The replica is promoted, and worker B acquires the same lock while A is still working. The same overlap can happen without failover: A pauses longer than the lock TTL, B acquires the expired lock, and A resumes. A single-primary lock backed by asynchronous replication does not provide absolute mutual exclusion through these events. Redis distributed locks, Martin Kleppmann on distributed locking.\n1 2 3 A gets lock -\u0026gt; A pauses / primary dies -\u0026gt; lock expires or is absent on replica -\u0026gt; B gets lock and writes A resumes -\u0026gt; A also writes after losing ownership The random token and safe-release script from section 3 prevent A from deleting B\u0026rsquo;s lock. They do not prevent A from writing to an external database after its lease expires. If the lock only reduces duplicate cache loads, an occasional extra database read may be acceptable. If it protects a correctness-critical write, the receiver needs to reject old workers. One approach is a monotonically increasing fencing token issued by a sufficiently reliable source; the receiver accepts only newer tokens. A random ownership token has no ordering information.\n6.4. WAIT does not turn Redis into a strongly consistent database WAIT 1 1000 waits up to 1,000 ms for at least one replica to acknowledge earlier writes from the same connection. Your app must inspect the returned replica count. WAIT can reduce the chance of losing a recent write on failover, but it does not guarantee that the replica later promoted is the one that received that write. SET lock followed by WAIT therefore does not fully fix section 6.3. Redis WAIT.\nTest: in an isolated environment, keep writing IDs, stop the primary unexpectedly, then compare acknowledged IDs with data on the new primary. One clean run does not prove all future failovers are safe. It shows the behavior of your setup under that particular failure.\n6.5. A Stream can redeliver work that already succeeded Worker A reads an entry with XREADGROUP and successfully charges an external payment service. A dies before XACK. Redis still lists the entry as pending in the Pending Entries List (PEL). Worker B uses XAUTOCLAIM, receives the entry, and may charge the customer again. Acknowledging before charging is also dangerous: if A dies after XACK, nobody may perform the charge. The external effect and the Redis acknowledgment are not one transaction. Redis Streams, XPENDING.\n1 2 XREADGROUP -\u0026gt; external payment succeeds -\u0026gt; worker dies -\u0026gt; no XACK XAUTOCLAIM -\u0026gt; another worker receives it -\u0026gt; possible second payment Handle duplicates deliberately. Give the task a stable event_id or order_id; the payment service should reject an ID it has already processed. Acknowledge only after confirming the business result. Track XPENDING, entry age, and redelivery count. Repeatedly failing work needs a separate handling path.\nTake care when trimming the Stream. In Redis 7.x, XTRIM MAXLEN can remove an entry\u0026rsquo;s payload while the PEL still refers to it. Redis 8.2 added KEEPREF, DELREF, and ACKED options for more control. Choose retention to accommodate slow workers and verify behavior on the Redis version you actually run. Redis XREADGROUP, XTRIM.\n6.6. MULTI/EXEC does not roll back every execution error If you are used to SQL transactions, you may expect the first command to be undone whenever the second fails. Redis does not do that for an error that happens during execution. An earlier command can succeed, the next can fail, and later queued commands can still run. Redis does discard a transaction for certain errors detected while commands are being queued before EXEC. Redis transactions.\n1 2 3 4 MULTI SET demo:tx:key abc INCR demo:tx:key # valid syntax, but \u0026#34;abc\u0026#34; is not an integer EXEC # SET succeeded; INCR errors; the key is still abc Check: read the result of each command returned by EXEC and test type errors. If another system is involved, design how you will correct a partial business operation. Lua scripts also prevent other Redis commands from interleaving during execution, but that does not make them SQL transactions with automatic rollback for every error.\n6.7. One hot key can overload one Cluster node Redis Cluster maps keys to hash slots and distributes slots among nodes. One key belongs to one slot. If most traffic reads or updates flash-sale:stock:42, its node can saturate while the others are idle. Adding nodes does not split one popular key into pieces. A broad hash tag such as {catalog} can accidentally push a whole group of keys into the same slot. Redis Cluster scaling, Cluster specification.\nMeasure traffic by key and by node. For mostly read-only data that tolerates a little lag, local application cache, a CDN, or replica reads may help. Splitting an atomic counter into several keys changes its meaning: you need a new aggregation and correctness design, not only new key names.\n6.8. Deleting a huge key can make Redis pause DEL on a structure with a very large number of elements can spend time freeing memory as part of the command path, delaying other work. UNLINK removes the key from the keyspace first and frees the heavier memory work on another thread. The cleanup still happens; it is moved away from the immediate reply path. Redis UNLINK.\nCheck: use MEMORY USAGE, inspect collection sizes, and compare SLOWLOG with p99 latency near cleanup time. If the same key repeatedly becomes huge, investigate splitting the data or setting retention earlier rather than only changing DEL to UNLINK.\n7. Exercises and a Pre-Launch Checklist Try these exercises on a dedicated test instance:\nReproduce stale cache. Run demo_cache.py, remove the version check as described in section 2, and compare Redis with SQLite. Restore the check and see which step changes the outcome. Observe the limiter. Run demo_limit.py with [10, 5], then with [3, 1]. Add different pauses between calls. Why does a call pass after waiting? Start with an empty cache. Under simulated traffic, delete a popular key or restart with no cache. Measure database queries per second and p99 latency. Add request coalescing or stale-while-revalidate and measure again. Before a real rollout, write down answers to these questions:\nQuestion Decision to make What if Redis is unavailable? Read the database, return permitted stale data, or fail the request? Set a timeout and limit database fallback load. How long may old data appear? Choose TTL from product requirements. Important writes may need reliable invalidation or an authoritative read. What if RAM fills? Set maxmemory and eviction by data type; keep correctness-critical keys away from broad cache eviction. Redis eviction. Must data survive restart? Choose RDB/AOF from allowed data loss, keep backups, and practice restore. Redis persistence. Which commands might cause latency? Avoid KEYS * on large systems; use SCAN, check command complexity, value size, and SLOWLOG. Redis latency guidance. Too many network trips? Use MGET/MSET or reasonable pipeline batches for independent commands. Redis pipelining. What should we monitor? Cache misses by feature/key, p95/p99, evictions, memory, connections, database load, limiter denials, and Redis errors. Two concrete answers matter more than a generic checklist: \u0026ldquo;If Redis has not replied in 500 ms, what does this request do?\u0026rdquo; and \u0026ldquo;If the cache shows an old price, what happens to the customer?\u0026rdquo; Without agreed answers, timeout, fallback, and TTL settings are guesses.\n8. What I Would Take Away GET and SET are the easy part. The hard parts are the ordering between Redis and the database, the peak when many readers miss together, and the effects of Redis losing state or changing primaries. Meta\u0026rsquo;s cache example makes a stale fill easy to see. Stripe\u0026rsquo;s rate-limiter story shows why limits need monitoring and a failure policy before enforcement. The eight cases above show how a command that seems fine in isolation can fail once time, retries, or failover are involved.\nRun the Python labs, change one step at a time, and compare the observed behavior with the guarantees your product actually needs.\nFurther reading Meta Engineering, Cache made consistent: the delayed cache-fill race. Meta\u0026rsquo;s system is not Redis, but the ordering lesson applies. Stripe Engineering, Scaling your API with rate limiters: four protection layers, Redis token buckets, and a careful rollout. Martin Kleppmann, How to do distributed locking: why expired leases and paused workers matter, and when fencing tokens help. Redis Lua scripting, Cluster hash tags, eviction, persistence, and latency guidance. ","permalink":"https://xops.io.vn/posts/redis-production-pitfalls/","summary":"\u003cp\u003eImagine that you have just changed a product\u0026rsquo;s price. One request is still reading the old price from the database; another has already deleted the old cache entry. Which request writes the cache last? Now imagine thousands of visitors opening the same page just as its cache key expires. How many database queries will you get?\u003c/p\u003e\n\u003cp\u003eThose are the questions I find more interesting than another \u003ccode\u003eSET\u003c/code\u003e/\u003ccode\u003eGET\u003c/code\u003e tutorial. This article starts with a small product catalog and follows three problems: \u003cstrong\u003estale cache fills\u003c/strong\u003e, \u003cstrong\u003ea database overwhelmed by cache misses\u003c/strong\u003e, and \u003cstrong\u003erate limits that behave differently from what their configuration suggests\u003c/strong\u003e. It ends with eight failures worth simulating before Redis sits on an important request path.\u003c/p\u003e","title":"Redis in Production: Why a Fast Cache Can Still Return Stale Data"},{"content":"Someone registers for your event. The application increments a counter in Redis, receives a successful reply, and confirms the registration. One second later, the machine loses power. Will that increment still be there when Redis starts again?\nThe answer depends on what Redis wrote to disk, when it wrote it, and which files it loads during recovery. Those three questions make RDB and AOF much easier to understand than memorizing \u0026ldquo;snapshot\u0026rdquo; and \u0026ldquo;log\u0026rdquo; as definitions.\nThis article uses Redis Open Source 7.x; the labs use Redis 7.4 with Docker. Redis 7 matters here because its AOF is split across multiple files. If keys and TTLs are new to you, start with the Redis fundamentals article. The production pitfalls article continues with cache consistency and failover.\n1. First, Define What \u0026ldquo;Durable\u0026rdquo; Means for Your App Not all failures have the same effect:\nFailure What can happen Why it matters Redis process exits but the machine stays up Redis RAM is gone; the OS may still have buffered writes Data already handed to the OS may eventually reach disk. The machine loses power RAM and OS page cache may be lost write() and fsync() are different steps. A disk or volume is lost Local persistence files are gone A file on the same disk is not an off-machine backup. An operator runs DEL or FLUSHALL by mistake Valid delete commands can be persisted and replicated A replica is not a historical backup. Two goals help turn \u0026ldquo;we need durability\u0026rdquo; into a decision:\nRPO, Recovery Point Objective: how much acknowledged data can we afford to lose? RTO, Recovery Time Objective: how long can the service take to recover? A cache may tolerate losing everything, yet still need a fast recovery so the database is not overwhelmed by a cold start. A payment ledger may not tolerate losing even one confirmed transaction. The two workloads need different designs even if both currently use Redis.\nAn OK response is not a disk receipt When SET returns OK, Redis has processed the command under its current configuration. With RDB, the latest snapshot may predate that command. With AOF using appendfsync everysec, the change may not yet have been synchronized to the storage device. Do not read OK as \u0026ldquo;this survives an immediate power failure.\u0026rdquo; Redis persistence documentation.\nflowchart LR C[Client: SET k v] --\u0026gt; M[Redis changes data in RAM] M --\u0026gt;|RDB| S[Snapshot at a point in time] M --\u0026gt;|AOF| B[AOF buffer] B --\u0026gt; W[write to OS page cache] W --\u0026gt; F[fsync to storage device] The diagram simplifies some implementation details, but the timing is the important part: a command can be processed before its bytes have reached durable storage. The appendfsync setting changes how wide that gap can be.\n2. RDB: A Photograph of the Dataset When Redis makes a snapshot RDB is a binary snapshot of the Redis dataset at a point in time. You can request one with BGSAVE, or configure a rule such as save 60 1000. That rule allows a snapshot after at least 60 seconds and at least 1,000 changes. It is a trigger threshold, not a promise that a new file appears every 60 seconds.\nRedis also has SAVE, but it blocks command processing while writing the snapshot. For a running application, BGSAVE is generally the appropriate command to discuss. Redis persistence, BGSAVE.\n1 2 3 4 5 6 save 60 1000 # In redis-cli: BGSAVE INFO persistence LASTSAVE One easy mistake is treating a BGSAVE reply as proof that the file is complete. The reply says Redis accepted the request. BGSAVE SCHEDULE can even defer the work when an AOF rewrite is in progress. Check rdb_bgsave_in_progress, rdb_last_bgsave_status, and whether LASTSAVE changed before relying on the new snapshot. BGSAVE, INFO, LASTSAVE.\nHow fork and copy-on-write keep the picture stable When BGSAVE starts, Redis creates a child process using fork(). The child sees the dataset as it existed at that instant and writes the RDB file. The parent continues serving the application.\nInitially the two processes share physical memory pages. When the parent changes a page, the operating system copies that page so the child can keep seeing the old contents. This is copy-on-write, or COW: a page is copied when a write makes a copy necessary. The child writes a temporary RDB file and replaces the old snapshot after the new file is complete. Redis persistence, Redis FAQ on fork and COW.\nsequenceDiagram participant P as Redis parent participant C as BGSAVE child participant D as Disk P-\u0026gt;\u0026gt;C: fork at 10:00 C-\u0026gt;\u0026gt;D: write the 10:00 dataset P-\u0026gt;\u0026gt;P: accept a new SET at 10:00:30 Note over P,C: Changed memory pages are copied on write C-\u0026gt;\u0026gt;D: finish temporary file and replace old snapshot Suppose the snapshot starts at 10:00 and finishes at 10:01. A registration at 10:00:30 is not in that snapshot: the child sees the 10:00 state. If Redis fails before another snapshot and there is no AOF, that registration can be lost.\nBackground work still has a cost BGSAVE runs in the background, but fork() itself can pause Redis, sometimes noticeably with a large dataset. The child reads memory and writes disk. If the parent changes many pages during the snapshot, COW can substantially increase RAM use. Near the machine\u0026rsquo;s memory limit, fork may fail or latency may rise. Redis persistence, latency guidance.\nAround snapshot time, watch free RAM, rdb_last_cow_size, current_cow_size, save duration, disk throughput, and p99 latency. A dump.rdb file on disk might be old if recent snapshot attempts failed; the important question is when the last successful snapshot completed. Redis INFO.\n3. AOF: Record the Changes, Then Mind the fsync Gap AOF stands for Append Only File. It records operations that change data, using the Redis protocol. On startup, Redis can replay those operations to rebuild the dataset. If a counter is incremented a million times, the log can grow even though the current state is only one number. An AOF rewrite represents the current state more compactly. Redis persistence.\n1 2 appendonly yes appendfsync everysec The three appendfsync choices Setting When Redis requests fsync Practical tradeoff always With newly appended writes, before acknowledging the corresponding batch Stronger protection against an OS or power failure, with more I/O and latency. everysec Approximately once per second Often a useful balance; roughly a second of changes may be at risk in a catastrophic failure. no Redis does not actively call fsync; the OS decides Less Redis-directed synchronization, but the loss window depends on the OS. Redis documentation recommends everysec for many workloads. \u0026ldquo;About a second\u0026rdquo; describes the tradeoff; it is not a hard guarantee for every failure. Even always does not protect against a destroyed disk, an operational mistake, or a failover to a replica that has not received a write. Redis persistence.\nWhat is the difference between write() and fsync()? After write(), bytes are commonly in the operating system\u0026rsquo;s page cache. If only the Redis process dies, the OS may still write them to disk. If the whole machine loses power, that page cache can disappear. fsync() asks the OS to synchronize the data to the storage device. The final durability still depends on the filesystem and hardware. This is also why an AOF benchmark should report p99 latency, not just commands per second.\nA TTL does not pause while Redis is down Redis stores expiration against real time. If a session has five seconds left, Redis is down for ten seconds, and then it restarts, the session has already expired. When testing recovery, first use a key without a TTL to see whether the data was restored. Test an expiring key separately so its expected expiration does not look like data loss. Key expiration.\n4. Why Redis 7 Uses Multiple AOF Files Rewrite is more than deleting repeated lines Imagine incrementing a counter a million times. Retaining every increment takes space, even though Redis currently holds only a single number. BGREWRITEAOF creates a shorter representation of the current dataset. It is not simply a text editor removing duplicate lines from the old log. Redis keeps accepting writes while the rewrite runs. Redis persistence, BGREWRITEAOF.\nBe careful with pre-7 explanations Older articles often describe a secondary buffer that holds new writes while a child process creates a replacement AOF. That describes the approach before Redis 7. Since Redis 7, AOF consists of multiple files. This is an important version distinction when reading the article by Chanh Le: its secondary-buffer explanation is useful historical context, while a Redis 7.4 backup must follow the newer file layout. Redis multi-part AOF documentation.\nflowchart TD M[Active manifest] --\u0026gt; B[Base AOF: dataset at rewrite] M --\u0026gt; I1[Incremental AOF: later writes] I1 --\u0026gt; R[BGREWRITEAOF starts] R --\u0026gt; C[Child creates a new base] R --\u0026gt; I2[Parent opens a new incremental file and keeps writing] C --\u0026gt; N[New manifest points to new base and incremental files] I2 --\u0026gt; N N --\u0026gt; S[New manifest becomes active] The manifest is the list of files Redis needs to reconstruct the dataset. A base file contains the starting state and may use RDB or AOF format. Incremental files contain subsequent changes. During rewrite, the child creates a new base while the parent keeps recording new writes. When the rewrite completes, Redis switches the manifest to the new set. If the rewrite fails, the still-valid files and current incremental writes remain the recovery path. Redis persistence, Redis 7+.\nThis matters for backups. Copying one *.base.rdb file is not a complete AOF backup. Keep the manifest and every file it references together. Copying the directory mid-rewrite without a consistency procedure can leave you with files from different moments. The Redis documentation describes pausing rewrite, checking aof_rewrite_in_progress, copying the directory, and re-enabling rewrite. Managed services may have their own backup procedure. Redis backup guidance.\nWhat \u0026ldquo;hybrid AOF\u0026rdquo; means With aof-use-rdb-preamble yes, the rewritten AOF base can be in binary RDB format while the incremental files continue recording later changes. This can make loading the base faster than replaying a long command history. \u0026ldquo;Hybrid\u0026rdquo; describes the format inside the AOF set; it does not automatically create a separate, independent RDB backup. Redis persistence.\n5. What Redis Loads After a Restart If both RDB and AOF are enabled, Redis prefers AOF at startup because it usually includes newer changes. Seeing a valid dump.rdb on disk does not prove that Redis will load it. Check CONFIG GET appendonly, AOF status, and the startup log to see which files Redis actually read. Redis persistence.\nflowchart TD START[Redis starts] --\u0026gt; A{AOF enabled?} A --\u0026gt;|Yes| AOF[Read manifest and AOF files] A --\u0026gt;|No| RDB[Read RDB snapshot, if present] AOF --\u0026gt; RAM[Rebuild dataset in RAM] RDB --\u0026gt; RAM A cut-off tail is different from corruption in the middle After an abrupt stop, the final AOF command may be only partly written. Depending on configuration, Redis can discard the incomplete tail and continue loading; inspect the log to confirm what it did. Damage in the middle of a file is more serious. Truncating from that point could discard many later changes. Make a copy of the original file and use redis-check-aof to inspect it before considering --fix. A successful startup after repair is not proof that all business data survived. Redis AOF corruption guidance.\nPING can succeed while persistence is broken Redis can reply PONG even when saving has failed. Full disks, incorrect permissions, a failed fork, and rewrite errors all need alerts. Watch rdb_last_bgsave_status, aof_last_write_status, aof_last_bgrewrite_status, aof_delayed_fsync, free disk space, and logs. Depending on configuration, Redis can refuse writes after an RDB save failure. A health check that only sends PING cannot tell you whether data is being persisted. Redis INFO, persistence.\n6. Choose Persistence by Data, Not by Slogan Setup Suitable when What to verify or accept No disk persistence Cache is rebuildable and the database can handle a cold start Restart means an empty cache; many requests may hit the database together. RDB only Losing changes since the last snapshot is acceptable Loss depends on the last successful snapshot. AOF only You want a smaller potential loss window Manage multi-part AOF, disk space, rewrite, and load time. RDB + AOF You want newer AOF data and an independent snapshot for backup Extra I/O and disk use; Redis prefers AOF at startup. Does a replica replace a backup? No. Replication is usually asynchronous, so recent writes can be missing when a replica is promoted. A mistaken delete can also be replicated. A backup is a separate copy that you can inspect and restore after the current files or an entire cluster are lost. Redis persistence, WAIT and its limits.\nFor a cache or session, start with: \u0026ldquo;What will users see if all of this state disappears?\u0026rdquo; For a confirmed financial operation, agree on RPO and RTO with the product owners and practice both failover and restore. A single appendfsync always setting cannot replace that exercise.\nA GitHub case study: moving durable data elsewhere In \u0026ldquo;Moving persistent data out of Redis\u0026rdquo;, GitHub\u0026rsquo;s engineers separated data they could rebuild from data that existed only in Redis. They moved the latter to MySQL, which they already knew how to operate, and disabled persistence for the Redis data that remained. This reduced operational complexity and disk I/O in their environment. It was a decision for GitHub\u0026rsquo;s workload at that time, not a rule that Redis persistence is always wrong.\nThe interesting part is how they approached the move. They measured writes and found that activity feeds produced much of the load. Instead of copying each event into every user\u0026rsquo;s feed, they changed the data model to store activity by organization and filter at read time. They migrated feed types in stages, used a period of dual writes, and watched capacity and replication delay. Sometimes the better persistence decision starts with where the authoritative data lives and how it is modeled, not with changing appendfsync. GitHub Engineering.\n7. Two Docker Labs You Can Inspect The following PowerShell commands use Docker on your own machine and create named volumes dedicated to this exercise. Run each command after the previous one finishes.\nLab A: make and reload an RDB snapshot 1 2 3 4 5 docker volume create redis-rdb-lab-vol docker run -d --name redis-rdb-lab -v redis-rdb-lab-vol:/data redis:7.4 redis-server --save 60 1 --appendonly no docker exec redis-rdb-lab redis-cli SET lab:rdb hello docker exec redis-rdb-lab redis-cli BGSAVE docker exec redis-rdb-lab redis-cli INFO persistence In the INFO persistence output, wait for rdb_bgsave_in_progress:0 and check rdb_last_bgsave_status:ok. Now stop and remove the container, then recreate it with the same volume:\n1 2 3 4 docker stop redis-rdb-lab docker rm redis-rdb-lab docker run -d --name redis-rdb-lab -v redis-rdb-lab-vol:/data redis:7.4 redis-server --save 60 1 --appendonly no docker exec redis-rdb-lab redis-cli GET lab:rdb GET should return hello. This demonstrates loading a completed snapshot after a normal stop. It does not simulate a power failure or measure RDB\u0026rsquo;s RPO; we deliberately waited for BGSAVE before stopping Redis.\nLab B: inspect the AOF file set 1 2 3 4 5 6 7 docker volume create redis-aof-lab-vol docker run -d --name redis-aof-lab -v redis-aof-lab-vol:/data redis:7.4 redis-server --appendonly yes --appendfsync everysec docker exec redis-aof-lab redis-cli SET lab:aof hello docker exec redis-aof-lab redis-cli INCR lab:aof:counter docker exec redis-aof-lab redis-cli BGREWRITEAOF docker exec redis-aof-lab redis-cli INFO persistence docker exec redis-aof-lab ls -l /data/appendonlydir Wait for aof_rewrite_in_progress:0 and check aof_last_bgrewrite_status:ok. You will typically find base, incremental, and manifest files in the directory. Names and counts can vary with rewrite timing; the manifest tells you which set is active. The image may also retain default RDB save rules. This lab observes AOF while it is enabled, and Redis will prefer AOF at startup.\n1 2 3 4 docker stop redis-aof-lab docker rm redis-aof-lab docker run -d --name redis-aof-lab -v redis-aof-lab-vol:/data redis:7.4 redis-server --appendonly yes --appendfsync everysec docker exec redis-aof-lab redis-cli GET lab:aof GET should again return hello. If you later want to experiment with AOF repair, copy the volume first; editing the only recovery copy tells you little about safe recovery.\nWhat should be on a dashboard? Area Example signals Question to answer RDB rdb_last_bgsave_status, rdb_last_save_time, rdb_last_cow_size Did the latest snapshot finish, and how much RAM did COW use? AOF aof_last_write_status, aof_rewrite_in_progress, aof_delayed_fsync, aof_current_size Are writes and rewrites healthy? Machine Free disk, available RAM, p99 latency, restart/load time Could this machine recover within our target? Backup Last successful copy, checksum, last restore exercise Can we actually use the backup? INFO fields can differ between Redis versions. Check the fields on the version you run rather than assuming every field exists everywhere. Redis INFO.\n8. Six Easy Misreadings of Persistence \u0026ldquo;AOF means zero data loss.\u0026rdquo; everysec has a synchronization gap. Disk failure and mistaken deletes need separate recovery plans. \u0026quot;BGSAVE returned, so the new snapshot is ready.\u0026quot; Wait for completion and check its status. \u0026ldquo;Redis 7 has one appendonly.aof file.\u0026rdquo; Look for base, incremental files, and the manifest. \u0026ldquo;We have a replica, so we have a backup.\u0026rdquo; A replica can copy a mistaken delete. Keep a separate copy and test restoring it. \u0026ldquo;Background save uses no extra memory.\u0026rdquo; Fork can pause Redis, and writes during a snapshot can grow COW memory. \u0026ldquo;Redis will load dump.rdb because it exists.\u0026rdquo; When AOF is enabled, Redis prefers AOF. 9. Closing Thought RDB captures the dataset at a moment. AOF records changes needed to rebuild the dataset. RDB\u0026rsquo;s snapshot can miss later writes; AOF\u0026rsquo;s durability depends on when those changes reach disk. Since Redis 7, the AOF files work as a set managed by a manifest. When deciding what to enable, name the failure you are preparing for, the amount of data you can lose, and the file or backup from which you will recover.\nFurther reading Chanh Le, Redis Persistence: RDB and AOF: another explanation of RDB and AOF; note its pre-Redis-7 rewrite description. GitHub Engineering, Moving persistent data out of Redis: the data-model and persistence case study. Redis persistence, BGSAVE, BGREWRITEAOF, INFO, and latency guidance. ","permalink":"https://xops.io.vn/posts/redis-persistence-rdb-aof/","summary":"\u003cp\u003eSomeone registers for your event. The application increments a counter in Redis, receives a successful reply, and confirms the registration. One second later, the machine loses power. Will that increment still be there when Redis starts again?\u003c/p\u003e\n\u003cp\u003eThe answer depends on \u003cstrong\u003ewhat Redis wrote to disk, when it wrote it, and which files it loads during recovery\u003c/strong\u003e. Those three questions make RDB and AOF much easier to understand than memorizing \u0026ldquo;snapshot\u0026rdquo; and \u0026ldquo;log\u0026rdquo; as definitions.\u003c/p\u003e","title":"Redis Persistence Explained: What RDB and AOF Actually Save"},{"content":"The first two Redis commands take about a minute to learn: SET name Alice, then GET name. The harder questions arrive when you put Redis behind a real application. How should you store a list of attendees? When does a key disappear? What happens when two people try to reserve the last ticket at the same time?\nWe\u0026rsquo;ll answer those questions with one running example: an online event platform that tracks attendees, page views, scores, sessions, and activity. The examples use Redis Open Source 7.x and Python 3.10+. Every key beginning with lab: belongs to the exercises.\nThis is the first article in a series. Next, we\u0026rsquo;ll examine RDB and AOF persistence, then look at the failures that appear under real traffic.\n1. What Redis Is, and Why It Usually Feels Fast A store of data structures in memory Think of Redis as a place where the working data lives in RAM. Your application sends commands over the network to read or change that data. Redis understands more than strings: a Set can answer whether Alice has registered, and a Sorted Set can return the top ten scores without sending the entire leaderboard to your application. Redis data types.\nflowchart LR U[User] --\u0026gt; A[Application] A --\u0026gt;|commands and replies| R[(Redis: working data in RAM)] A --\u0026gt;|orders and payments| D[(System-of-record database)] R -.-\u0026gt;|RDB or AOF, if enabled| F[(Disk)] Redis often responds quickly, but the time a user sees also includes network travel, serialization, the size of the reply, and application code. A benchmark from another machine does not promise the same latency in your system. Redis latency guidance.\nRedis can be the primary database for some workloads, provided you understand their durability and recovery requirements. In our event app, orders and payments live in the business database. Redis helps with cache, sessions, leaderboards, and short-lived coordination. If Redis restarts and loses a key, we should know whether the app can rebuild it and what the user will notice.\nFast does not mean every command is cheap Compare GET on a small value with SMEMBERS on a Set containing millions of people. The second command must produce a very large response. A long-running Lua script can also make other commands wait. \u0026ldquo;It is in RAM, so it is always fast\u0026rdquo; is an unsafe assumption.\nWhen choosing a command, read its complexity in the command documentation and measure latency from the application. p95 and p99 reveal the slowest requests that an average can hide. Redis also uses supporting threads for some I/O and background work, so the phrase \u0026ldquo;Redis is single-threaded\u0026rdquo; needs context. Redis latency guidance.\nBefore adding Redis to a feature, ask three questions:\nQuestion In our event app Can we rebuild this data? A leaderboard derived from an event log can be rebuilt. A payment record needs a trusted source. How fresh must it be? A score may lag by a few seconds. The number of tickets left at checkout may not. What happens if Redis is unavailable? Read the database, return slightly old data, or reject the request in a controlled way? These answers tell you what Redis is responsible for before you start tuning speed.\n2. Run Redis and Follow the Life of a Key If you have Docker, start a local instance:\n1 docker run --rm --name redis-learning -p 127.0.0.1:6379:6379 redis:7.4 Open redis-cli from another terminal:\n1 docker exec -it redis-learning redis-cli The port is bound to your own machine. Press Ctrl+C in the first terminal to stop Redis. This container has no persistent volume; use it to learn commands, then use the persistence lab when you want to test recovery. If port 6379 is already in use, choose another host port and update the Python example below.\nCreate, inspect, and remove a key 1 2 3 4 5 6 SET lab:event:42:title \u0026#34;Redis Workshop\u0026#34; GET lab:event:42:title TYPE lab:event:42:title EXISTS lab:event:42:title DEL lab:event:42:title EXISTS lab:event:42:title A key name should tell you which app it belongs to and what it contains, for example app:env:event:42:title. Clear names help debugging, although excessively long names consume RAM when you have millions of keys. The lab: prefix separates our exercise data from other data.\nOn a large instance, avoid KEYS * as a way to browse everything at once. Use SCAN to walk through the keyspace in batches. Its result is not a fixed snapshot if other clients are changing keys while you scan. Redis SCAN.\nTTL, expiration, and eviction are different events 1 2 3 SET lab:session:alice token-123 EX 30 TTL lab:session:alice # seconds remaining GET lab:session:alice # nil after the key expires TTL is the lifetime you set. Expiration is Redis treating the key as no longer valid once that lifetime ends. Eviction happens when Redis needs memory and removes keys according to its configured policy. A key can therefore be evicted before its TTL expires.\nTTL returns -2 when the key does not exist and -1 when it exists without an expiration. A plain SET that overwrites a key also removes its old TTL unless you provide EX, PX, or KEEPTTL. Key expiration, SET, eviction.\n1 2 3 SET lab:temp v1 EX 30 SET lab:temp v2 TTL lab:temp # -1: v2 no longer expires For sessions, caches, and short-lived locks, put the TTL in the same command that creates the key when possible. SET followed by EXPIRE leaves a gap: if your process stops between the two commands, the key has no expiration.\n3. Choose a Data Type for the Question You Need to Answer It is tempting to store everything as a JSON string. That can be fine for a small value, but changing one field means rewriting the entire JSON. Asking for the top scores means moving data back into your application. Redis data types let you perform many of those operations directly. Compare Redis data types.\nApplication question Type to consider Starting commands Store a token, flag, or counter? String SET, GET, INCR Read or change one property of an event? Hash HSET, HGET, HGETALL Has this user registered? Set SADD, SISMEMBER, SCARD Who is at the top of the leaderboard? Sorted Set ZADD, ZINCRBY, ZREVRANGE Need a simple ordered queue? List LPUSH, RPOP, or a blocking command Need acknowledgments and redelivery? Stream XADD, XREADGROUP, XACK String: values and atomic counters 1 2 3 4 SET lab:event:42:views 0 INCR lab:event:42:views INCR lab:event:42:views GET lab:event:42:views # \u0026#34;2\u0026#34; INCR performs the increase as one Redis command. If two people view the page at once, Redis records both increases. If the app instead does GET, adds one in its own code, then calls SET, both requests can read the old number and overwrite each other\u0026rsquo;s result. Use INCR for a Redis counter. Its atomicity does not automatically synchronize that counter with another database. Redis INCR.\nHash: an entity with fields 1 2 3 4 HSET lab:event:42 title \u0026#34;Redis Workshop\u0026#34; capacity 100 status open HGET lab:event:42 capacity HINCRBY lab:event:42 capacity -1 HGETALL lab:event:42 A Hash lets us change capacity without rewriting the whole event. But HINCRBY capacity -1 can take ticket stock below zero. For real tickets, the check and decrease must happen together, and the result must agree with the system of record. We will revisit that kind of concurrency problem in the production article. Also avoid putting every event into one enormous Hash: a huge key is harder to read, delete, or move.\nSet: unique membership 1 2 3 4 SADD lab:event:42:attendees alice bob SADD lab:event:42:attendees alice # returns 0: already present SISMEMBER lab:event:42:attendees alice SCARD lab:event:42:attendees A Set answers \u0026ldquo;has Alice registered?\u0026rdquo; nicely and removes duplicates for you. It does not keep registration order, and Alice cannot appear twice. If the set grows large, SMEMBERS returns a large response; you may only need a membership check or a partial scan. Decide when old events and their attendee sets should be removed.\nSorted Set: scores and ranks 1 2 3 4 ZADD lab:event:42:score 15 alice 10 bob ZINCRBY lab:event:42:score 7 bob ZREVRANGE lab:event:42:score 0 9 WITHSCORES ZREVRANK lab:event:42:score bob A Sorted Set holds each member once and orders members by score. It is useful for leaderboards, priority lists, and some rate-limiting algorithms. Define what a tied score means for your product; do not assume insertion order decides the tie. Redis Sorted Sets.\nList, Pub/Sub, and Stream solve different delivery problems A List can hold simple ordered work, but your application must design retries and acknowledgments. Pub/Sub behaves more like a live broadcast: subscribers that are connected receive the message; a disconnected subscriber misses it. A Stream retains entries and can track work delivered to a consumer group but not yet acknowledged with XACK. A worker may receive the same entry again, so its processing must tolerate duplicates. Choose based on delivery requirements, not only on which API is shorter. Redis Pub/Sub, Streams.\n1 2 XADD lab:event:42:activity * user alice action joined XRANGE lab:event:42:activity - + COUNT 10 XADD does not set a retention limit automatically. If you keep a Stream for a long time, decide when to trim it and what happens to slow consumers. Trimming too early can remove an entry\u0026rsquo;s payload while it is still pending. Redis XTRIM.\n4. What \u0026ldquo;Atomic\u0026rdquo; Means When Requests Run Together INCR is one command, and Redis handles that command as a unit. GET followed by SET is two commands: another client can change the key in between. Likewise, writing paid in Redis and updating an order in PostgreSQL does not create a transaction across both systems.\nPipeline saves network trips, not race conditions Suppose you need to write 1,000 independent keys. Sending a command and waiting for a reply 1,000 times costs many network round trips. A pipeline sends multiple commands before collecting the replies. That usually helps throughput, but it does not make a GET → calculate → SET sequence atomic. The calculation still depends on a reply. Keep batches reasonably sized because Redis must buffer replies before your client reads them. Redis pipelining.\nTransactions, WATCH, and Lua have different jobs Tool Reach for it when Easy mistake MULTI/EXEC You need Redis to execute a group of commands without another client\u0026rsquo;s command in between An error during execution does not roll back commands already run. WATCH + MULTI/EXEC You read first and want to write only if a watched key has not changed Competing clients can force retries. Lua script You need to read, decide, and write in one call to Redis A long script blocks other work; it is not a transaction with an external database. For example, a plain DECR on ticket stock can go below zero. A short Lua script could check that stock is above zero and then decrease it in one execution. Payment and the business database still live outside Redis, however. You still need rules for reservations, timeouts, and refunds. Redis transactions, Lua scripting.\n5. Four Common Uses and Their Limits Sessions. SET session:\u0026lt;random-id\u0026gt; \u0026lt;user-id\u0026gt; EX 1800 gives a session a 30-minute lifetime. Generate an unpredictable ID; keep sensitive information out of key names and logs. Delete the key on logout. If Redis restarts without that session, the user may have to log in again. Some products accept this; others need more durable session storage.\nCache-aside. The application checks Redis first. On a miss it reads the database and fills Redis:\nsequenceDiagram participant U as User participant A as Application participant R as Redis participant D as Database U-\u0026gt;\u0026gt;A: View event 42 A-\u0026gt;\u0026gt;R: GET cache:event:42 alt cache hit R--\u0026gt;\u0026gt;A: Cached event else cache miss A-\u0026gt;\u0026gt;D: SELECT event 42 D--\u0026gt;\u0026gt;A: Event A-\u0026gt;\u0026gt;R: SET cache:event:42 ... EX 60 end A--\u0026gt;\u0026gt;U: Response This reduces database reads, but two problems remain: an update can race with a cache fill, and many requests can miss the same popular key together. The production article works through both. Redis cache-aside.\nLeaderboards. ZINCRBY adds points and ZREVRANGE reads the top N. A recreational leaderboard that can be rebuilt from event history fits Redis well. If points determine a cash prize, keep an auditable source of changes; a Sorted Set in RAM should not be the only record.\nNotifications and work. Pub/Sub can tell online viewers that Alice joined. A Stream is a better candidate when a worker must send an email even after restarting. XACK tells Redis that the worker acknowledged an entry; it does not prove that an external email was sent exactly once. Redis Pub/Sub, Streams.\n6. Memory, Cluster, Security, and Observability Decide what happens when RAM fills up Redis uses RAM, so maxmemory and its eviction policy are product decisions. For a rebuildable cache, allkeys-lru or allkeys-lfu may be worth testing. With noeviction, some writes fail when Redis reaches the limit. Be careful if the same instance also holds sessions or important coordination keys: losing one of those can change application behavior. Monitor used_memory, evicted_keys, and the load on your database after a cold cache starts filling again. Redis eviction, INFO.\nReplicas, Cluster, and persistence answer different questions Mechanism Main purpose What it does not guarantee Replica Another copy for reads and failover The newest write may not have reached the promoted replica. Cluster Distribute keys across machines One extremely popular key still belongs to one slot; multi-key commands have slot constraints. RDB/AOF Reload data from disk after restart A separate backup or the survival of every acknowledged write. Redis Cluster assigns keys to hash slots. Keys used together by a multi-key command or script generally need the same slot. Hash tags can do that: event:{42}:stock and event:{42}:version share {42}. Do not tag every event {event}, or they will all land in one slot. Redis Cluster specification.\nA useful GitLab incident: long TTLs and eviction spikes In GitLab\u0026rsquo;s analysis of Redis latency spikes, slow periods coincided with bursts of eviction. Evicting many keys consumed CPU; waiting requests added memory pressure and made the episode worse. The team followed metrics from the client through CPU and RAM, then reduced the default cache TTL from two weeks to eight hours. In their workload, RAM usage fell without a significant drop in cache hit rate.\nThe lesson is to look beyond a pleasing hit-rate number. Rarely read keys still consume RAM, and continuous eviction can make users wait. Measure hit rate alongside p99 latency, eviction rate, and memory. Eight hours was GitLab\u0026rsquo;s result for that workload, not a universal Redis TTL.\nMeasure and protect the system people actually use Measure p95/p99 from the application, connection and timeout counts, memory, evictions, keyspace_hits/keyspace_misses, and database load behind the cache. A good overall hit rate can hide one popular key expiring and overwhelming the database for a few seconds. Track the event cache separately from sessions where possible. Redis INFO.\nDo not expose Redis directly to the Internet. Limit network access to services that need it, use ACLs to restrict commands and keys, and keep credentials out of code and blog posts. The password-free local CLI in this article is only for an isolated learning environment. Redis security, ACLs.\n7. A Small Python Lab Keep the Redis container from section 2 running. Install the client with python -m pip install redis, save the following as redis_fundamentals_demo.py, and run python redis_fundamentals_demo.py. The script deletes four exercise keys under lab:event:{42}:, so use an instance dedicated to learning.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 import redis r = redis.Redis(host=\u0026#34;localhost\u0026#34;, port=6379, decode_responses=True) event = \u0026#34;lab:event:{42}\u0026#34; meta = f\u0026#34;{event}:meta\u0026#34; attendees = f\u0026#34;{event}:attendees\u0026#34; score = f\u0026#34;{event}:score\u0026#34; views = f\u0026#34;{event}:views\u0026#34; r.delete(meta, attendees, score, views) r.hset(meta, mapping={\u0026#34;title\u0026#34;: \u0026#34;Redis Workshop\u0026#34;, \u0026#34;capacity\u0026#34;: 100}) r.sadd(attendees, \u0026#34;alice\u0026#34;, \u0026#34;bob\u0026#34;) r.zadd(score, {\u0026#34;alice\u0026#34;: 10, \u0026#34;bob\u0026#34;: 15}) r.set(views, 0, ex=60) r.incr(views) r.zincrby(score, 8, \u0026#34;alice\u0026#34;) print(\u0026#34;Event:\u0026#34;, r.hgetall(meta)) print(\u0026#34;Is Alice registered?\u0026#34;, bool(r.sismember(attendees, \u0026#34;alice\u0026#34;))) print(\u0026#34;Attendee count:\u0026#34;, r.scard(attendees)) print(\u0026#34;Top scores:\u0026#34;, r.zrevrange(score, 0, 1, withscores=True)) print(\u0026#34;Views:\u0026#34;, r.get(views), \u0026#34;TTL:\u0026#34;, r.ttl(views)) You should see one view. Alice has 18 points and ranks above Bob, who has 15. The views key has a positive TTL; the other three keys do not. That difference is deliberate: even keys belonging to the same event have separate lifetimes.\nTry changing the example:\nRun SET lab:event:{42}:views 99 and then TTL lab:event:{42}:views. Why is the result -1? Add alice to the attendee Set again. Does SCARD grow? What if the product needs to record two separate registrations? Would you accept losing page views after a restart? What about the attendee list? Answer from product requirements, not from a preference for Redis. Ask ZREVRANGE for the top ten. Does the application need to download and sort the entire Sorted Set? 8. Where to Go Next When I decide whether to use Redis for a feature, I walk through four questions: how will the app read and change this data, how long should a key live, what if several requests touch it together, and what happens when Redis is unavailable? Those answers lead to the data type, TTL, and recovery plan.\nThe next article explains how RDB and AOF keep data across restarts. The final article examines cache races, traffic spikes, rate limits, and less obvious production failures.\nFurther reading Chanh Le, Learning Redis Fundamentals: another introduction to the core concepts. GitLab Engineering, How we diagnosed and resolved Redis latency spikes: the eviction and TTL case study. Redis data types, key expiration, transactions, and Cluster specification. ","permalink":"https://xops.io.vn/posts/redis-fundamentals/","summary":"\u003cp\u003eThe first two Redis commands take about a minute to learn: \u003ccode\u003eSET name Alice\u003c/code\u003e, then \u003ccode\u003eGET name\u003c/code\u003e. The harder questions arrive when you put Redis behind a real application. How should you store a list of attendees? When does a key disappear? What happens when two people try to reserve the last ticket at the same time?\u003c/p\u003e\n\u003cp\u003eWe\u0026rsquo;ll answer those questions with one running example: an \u003cstrong\u003eonline event platform\u003c/strong\u003e that tracks attendees, page views, scores, sessions, and activity. The examples use Redis Open Source 7.x and Python 3.10+. Every key beginning with \u003ccode\u003elab:\u003c/code\u003e belongs to the exercises.\u003c/p\u003e","title":"Learning Redis from the Ground Up: Keys, Data Types, and the Decisions That Matter"},{"content":"Introduction OOMKill is a common problem when running workloads on Kubernetes.\nLinux OOM Killer Linux may terminate processes when the system runs out of available memory.\nKubernetes Memory Limits For example:\n1 2 3 4 5 resources: requests: memory: 256Mi limits: memory: 512Mi ","permalink":"https://xops.io.vn/posts/kubernetes-oomkill/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eOOMKill is a common problem when running workloads on Kubernetes.\u003c/p\u003e\n\u003ch2 id=\"linux-oom-killer\"\u003eLinux OOM Killer\u003c/h2\u003e\n\u003cp\u003eLinux may terminate processes when the system runs out of available memory.\u003c/p\u003e\n\u003ch2 id=\"kubernetes-memory-limits\"\u003eKubernetes Memory Limits\u003c/h2\u003e\n\u003cp\u003eFor example:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cdiv class=\"chroma\"\u003e\n\u003ctable class=\"lntable\"\u003e\u003ctr\u003e\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode\u003e\u003cspan class=\"lnt\"\u003e1\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e2\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e3\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e4\n\u003c/span\u003e\u003cspan class=\"lnt\"\u003e5\n\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\n\u003ctd class=\"lntd\"\u003e\n\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eresources\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erequests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ememory\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e256Mi\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003elimits\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ememory\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e512Mi\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/div\u003e\n\u003c/div\u003e","title":"Understanding OOMKill in Kubernetes"},{"content":"Hi, I\u0026rsquo;m Andrew Nguyen 👋\nI\u0026rsquo;m curious about cloud engineering, DevOps, and the infrastructure that keeps everything running. I like figuring out how things work, especially when they don\u0026rsquo;t work the first time.\nAround here, I share notes and practical lessons on:\nAWS Kubernetes Linux Terraform Docker CI/CD System Design I started this blog because the best lessons often begin with \u0026ldquo;Why is this broken?\u0026rdquo; and end with far too many open browser tabs. Writing helps me remember the answer; sharing it might save you a few tabs (and a little coffee).\nIf something here helps you solve a problem or sparks a new idea, I\u0026rsquo;m glad you stopped by!\n","permalink":"https://xops.io.vn/about/","summary":"\u003cp\u003eHi, I\u0026rsquo;m Andrew Nguyen 👋\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m curious about cloud engineering, DevOps, and the infrastructure that keeps everything running. I like figuring out how things work, especially when they don\u0026rsquo;t work the first time.\u003c/p\u003e\n\u003cp\u003eAround here, I share notes and practical lessons on:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAWS\u003c/li\u003e\n\u003cli\u003eKubernetes\u003c/li\u003e\n\u003cli\u003eLinux\u003c/li\u003e\n\u003cli\u003eTerraform\u003c/li\u003e\n\u003cli\u003eDocker\u003c/li\u003e\n\u003cli\u003eCI/CD\u003c/li\u003e\n\u003cli\u003eSystem Design\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI started this blog because the best lessons often begin with \u0026ldquo;Why is this broken?\u0026rdquo; and end with far too many open browser tabs. Writing helps me remember the answer; sharing it might save you a few tabs (and a little coffee).\u003c/p\u003e","title":"About"}]