REAL-TIME SYSTEMS
LogStream
An event-driven log-processing platform using FastAPI, Redis Streams and asynchronous workers for high-throughput log ingestion.
Overview
An event-driven log ingestion and processing platform that accepts high-throughput log streams via FastAPI, buffers them through Redis Streams with consumer groups, processes them in parallel workers, persists to PostgreSQL with batch writes, and streams real-time ingestion metrics via WebSockets.
The Problem
Centralized log ingestion at scale requires decoupling producers from consumers, handling burst traffic, deduplicating entries, and providing real-time visibility into ingestion health — all without losing data under load spikes.
Architecture
Key Decisions
Redis Streams over Pub/Sub
Redis Streams provide persistence, consumer groups, and acknowledgment — unlike Pub/Sub which drops messages if no consumer is listening.
SHA-256 deduplication
Content-hash based deduplication prevents duplicate log entries from being persisted, even under retry scenarios.
Batch PostgreSQL persistence
Workers batch-insert log entries to PostgreSQL rather than writing one row at a time, dramatically improving write throughput.
WebSocket metrics dashboard
Real-time ingestion rate, queue depth, and error counts streamed to a React dashboard via WebSockets.
Failure Modes
Worker crash during processing
Consumer group acknowledgment ensures unacknowledged messages are re-delivered to another consumer.
Duplicate log entries
SHA-256 content hashing deduplicates entries before PostgreSQL persistence.
Database write bottleneck
Batch persistence with configurable batch size prevents overwhelming PostgreSQL under burst load.
Ingestion spike
Redis Streams buffer absorbs burst traffic; workers scale horizontally via consumer groups.
Implementation
- FastAPI ingestion endpoints with async request handling
- Redis Streams with consumer groups for parallel processing
- SHA-256 based deduplication layer
- PostgreSQL batch persistence with connection pooling
- WebSocket server for real-time ingestion metrics
- React dashboard for live monitoring
- AWS deployment for production hosting
Lessons
- —Redis Streams are underused compared to Pub/Sub — the persistence and consumer group model is far more robust.
- —Deduplication at the ingestion layer saves significant downstream storage and query cost.
- —Real-time metrics dashboards make debugging production ingestion issues dramatically faster.