Real-Time Logistics Ingestion Engine
Imagine trying to prevent train derailments before they happen using nothing but GPS coordinates and vibration sensors. That's RailPulse which is a real-time anomaly detection system that thinks in milliseconds.
10,000
events/sec processed
<50ms
end-to-end latency (p99)
🧠The Geospatial Magic
Here's the clever part: trains speeding through sharp curves = bad. But how do you detect "sharp curves" in real-time?
- Geofence correlation: Define danger zones using PostGIS polygons (e.g., "Curve_234" at lat/lon bounding box)
- Stream processing: Kafka Streams checks
if (GPS in danger_zone && speed > 60 km/h) → alert(CRITICAL) - Zero database lookups: Geofences loaded into memory—no I/O bottlenecks when processing 10k events/sec
🏗️ System Architecture (Lambda Pattern)
| 🔥 Hot Path | Real-time anomaly detection → PostgreSQL + WebSocket broadcast |
| ❄️ Cold Path | Batch archival (1000 records/min) → MinIO S3 for historical analysis |
🎯 Technical Wins
- SIMD acceleration: Custom AVX2 kernels for vibration threshold checks (8 sensors processed per CPU cycle)
- Zero data loss: Kafka's distributed commit log + S3 lakehouse = every event survives server crashes
- Live dashboard: React frontend with Leaflet maps—watch trains move in real-time with color-coded alerts (🟢/🟠/🔴)
🚂 Fun fact: The simulator injects realistic anomalies by modeling train physics—acceleration curves, braking distance, even weather friction coefficients.
Technologies: Spring Boot 3.2, Apache Kafka, PostgreSQL+PostGIS, MinIO, Python simulator
- StackC++, SIMD, OpenMP, MinIO
- Sourcehttps://github.com/Shorya-agarwal/RailPulse
