[ Our Projects ] · Backend Infrastructure
02 / 04
// Backend Infrastructure Project

Distributed
Task
Queue

↗ Live Demo ↗ GitHub

A production-deployed Distributed Task Queue in Go — asynchronous job processing with Redis-backed queuing, PostgreSQL persistence, worker heartbeats, dead-letter queue mechanics, and real-time WebSocket monitoring. Deployed on Railway with Neon and Redis Cloud.

// Language
Go
Gin HTTP Framework
// Queue Backend
Redis
Redis Cloud + Pub/Sub
// Persistence
PostgreSQL
Neon Serverless DB
// Deployment
Railway
API + Worker Services
// Overview
Overview
Built and deployed a distributed task queue system in Go — enabling asynchronous job submission, Redis-backed queuing, and PostgreSQL-persisted task state. Workers register, send heartbeats, and process tasks independently. Failed tasks graduate to a dead-letter queue with retry mechanics. A WebSocket-powered real-time dashboard surfaces live queue stats, worker health, and event streams — all Dockerized and deployed to Railway with cloud-native database and cache layers.
// Architecture
System Flow
Client
Dashboard (HTML/JS)
API Service
Gin HTTP Server
Persistence
PostgreSQL / Neon
Queue
Redis Cloud — task queue + pub/sub events
Worker Service
Go Worker Pool
Real-Time
WebSocket — GET /ws → dashboard events
// State Machine
Task Lifecycle
// State 01
Pending
Submitted to Redis queue, persisted in PostgreSQL
// State 02
Processing
Picked up by an active worker, status updated
// State 03
Completed
Successfully processed, record finalized
// State 04
Failed
Error encountered, retries incremented
// State 05
DLQ
Max retries exceeded, moved to dead-letter queue
// Specs
Specifications
Domain
Distributed Systems / Backend Infrastructure
Language
Go (Golang)
HTTP Layer
Gin Web Framework
Queue
Redis Cloud (List + Pub/Sub)
Database
PostgreSQL via Neon Serverless
Real-Time
WebSocket (GET /ws)
Deployment
Railway (API + Worker — separate services)
Containerization
Docker (Dockerfile + Dockerfile.worker)
// Features
Features
01
Redis-backed task queue with PostgreSQL persistence — every job survives restarts
02
Worker registration with periodic heartbeats — active and offline status tracked in real-time
03
Dead-letter queue for failed tasks — view and retry failed jobs from the dashboard
04
Redis Pub/Sub event pipeline — task lifecycle events streamed in real-time across services
05
WebSocket endpoint — live dashboard updates without page refresh
06
Full observability dashboard — live metrics, worker health, task history, queue stats, and DLQ view
07
Split Dockerized services — independent API and Worker containers deployed to Railway
// Roadmap
Next Up
01
Exponential Backoff Retry Strategy
Retry 1 → 5s · Retry 2 → 30s · Retry 3 → 2min · then DLQ
02
Idempotency Keys
POST /tasks with idempotency_key returns existing task if already submitted — prevents duplicate processing
03
Prometheus Metrics Endpoint
Expose queue depth, worker throughput, and failure rates as Prometheus-compatible metrics
04
Priority Queues
High / Medium / Low priority lanes — workers consume high-priority tasks first
05
GitHub Actions CI/CD + Multiple Worker Replicas
Automated build & deploy pipeline with horizontal worker scaling on Railway
// Tech Stack
Tech Stack
Go Gin Redis Redis Cloud PostgreSQL Neon WebSockets Docker Railway Pub/Sub Dead Letter Queue REST API
Interested
in DTQ?

Want to discuss the architecture, explore collaboration, or talk distributed systems? Let's get into the details of building reliable async infrastructure at scale.

Get In Touch →
// End of Case Study
DTQ — Distributed Task Queue in Go