DISTRIBUTED SYSTEMS
Distributed Fault-Tolerant Task Queue
A Redis + Lua distributed task queue with at-least-once delivery, heartbeat lease recovery, an independent Janitor reaper, Dead Letter Queue routing, and a React observability dashboard.
Overview
A production-grade distributed task queue built from scratch on Redis with custom Lua scripts for atomic task acquisition, heartbeat-based leases, an independent Janitor microservice that re-queues crashed workers' tasks, configurable retries, Dead Letter Queue isolation for poison pills, and an Express + React observability stack. Stress-tested at 10,000 concurrent jobs with 0% task loss.
The Problem
Background job processing needs guarantees beyond list-popping. Workers crash, networks partition, and poison messages can block the queue forever. The system needed at-least-once delivery, automated recovery from worker failures, poison-pill isolation, and real-time visibility into queue health.
Architecture
Key Decisions
Lua scripts for atomic acquisition
Custom Redis Lua combines RPOP + ZADD into one atomic transaction, closing the microsecond gap where a crash could lose a task between pop and claim.
Independent Janitor (reaper)
A detached microservice sweeps queue:processing for zombie tasks whose heartbeat expired and safely re-queues them — recovery is not coupled to worker processes.
Heartbeat lease + Sorted Set state
Tasks move into a Sorted Set scored by timestamp. Workers emit heartbeats to extend the lease; expired leases become reclaimable by the Janitor.
Dead Letter Queue for poison pills
Tasks exceeding MAX_RETRIES are banished to a DLQ so infinite-loop failures never block the main processing queue. The dashboard supports manual reset/reprocess.
Failure Modes
Worker crash mid-task (SIGINT / OOM)
Heartbeat stops; Janitor detects the stale Sorted Set entry and re-queues the task for another worker.
Duplicate delivery
At-least-once semantics with idempotent handlers; atomic Lua acquisition prevents two workers claiming the same pending task.
Poison messages
Retry counter routes persistently failing tasks to the DLQ after MAX_RETRIES, keeping the hot path clear.
Queue saturation
React observability dashboard and Express APIs expose queue depth, throughput, DLQ size, and recovery controls.
Implementation
- Atomic task acquisition via Redis Lua (RPOP + ZADD)
- Heartbeat-based lease recovery with configurable TTL
- Independent Janitor process for zombie-task reaping
- Configurable retry limits with Dead Letter Queue routing
- PM2 cluster-mode workers for horizontal CPU scaling
- Express observability API + React real-time dashboard
- Unit + Redis integration tests (Vitest) and GitHub Actions CI
Performance
~15,800 tasks/sec
Ingestion
10,000 jobs / 631ms
Burst load
0%
Task loss
Lessons
- —Atomic Redis Lua is essential for correct distributed queue semantics — list-popping alone is not crash-safe.
- —Recovery belongs in a dedicated Janitor, not inside the workers that just failed.
- —Observability and DLQ tooling turn failure from silent data loss into an operable workflow.