7,300+ lines of Go. Custom LSM-tree storage engine with compaction, Raft consensus with real RPC transport, gRPC streaming API, and Prometheus observability. No dependencies on Kafka or ZooKeeper.
It passes messages between different parts of an application. App A sends a message, App B receives it later. The messages are stored durably on disk and replicated across servers so nothing gets lost.
When you have multiple services that need to communicate. Instead of Service A calling Service B directly (and failing if B is down), A drops a message in the queue and moves on. B picks it up when it's ready. This decouples your services and makes them resilient to failures.
Kafka requires Java, ZooKeeper, and a lot of configuration. This is one Go binary with two command line flags. Same architecture (topics, partitions, consumer groups, LSM storage), built to understand the internals rather than depend on them.
It's a working distributed system with 45 passing tests, including multi-node cluster elections, network partition recovery, and compaction correctness under concurrent writes. It hasn't been battle-tested at Kafka's scale, but it runs and handles real workloads.
Everything from scratch: the LSM-tree storage engine with skip list memtable and multi-level compaction, the Raft consensus layer with real RPC transport and network partition handling, consumer groups with session timeout detection, Prometheus metrics, and gRPC streaming. No copy-paste from existing projects.
Producers send records. The broker stores them durably via an LSM-tree, replicates to followers through Raft consensus, and serves consumers at their own pace via gRPC streaming.
graph LR
P[Producer] --> B[Broker]
B --> S[LSM Storage]
B --> R[Raft Consensus]
R --> F1[Follower 1]
R --> F2[Follower 2]
S --> WAL[Write-Ahead Log]
S --> MT[MemTable]
MT --> SST[SSTables]
SST --> CMP[Compaction]
B --> C[Consumer Group]
B --> M[Prometheus /metrics]
When you write data, it goes to a write-ahead log first (so we don't lose it if the server crashes), then into a skip list memtable in memory (for fast reads). When the memtable fills up, a background worker flushes it to disk as a sorted SSTable file.
As L0 SSTables accumulate, a compaction worker kicks in: it reads all overlapping files, performs a k-way merge to deduplicate and sort entries, drops old tombstones, and writes a single merged file to L1. If L1 gets too large, compaction cascades down to L2 and beyond.
sequenceDiagram
Client->>Broker: write("order:123")
Broker->>WAL: append to log file
Broker->>MemTable: insert in skip list
Broker-->>Client: ok, offset=42
Note over MemTable: memtable full
MemTable->>L0 SSTable: flush sorted file to disk
Note over L0 SSTable: L0 threshold reached
L0 SSTable->>L1 SSTable: k-way merge compaction
If you run multiple servers, they form a Raft cluster. One node wins an election and becomes the leader. The leader replicates log entries to followers via AppendEntries RPCs. If the leader goes down, the remaining nodes detect the timeout, start a new election, and pick a new leader — all without losing committed data.
The transport layer is pluggable: a ChannelTransport for in-process testing (with a NetworkSimulator that can partition and heal nodes), and a gRPC transport for real multi-node deployments.
flowchart TD
A[Follower] -->|election timeout| B[Candidate]
B -->|wins majority vote| C[Leader]
B -->|higher term seen| A
C -->|sends heartbeats via transport| C
C -->|network partition| A
A -->|receives AppendEntries| A
Consumers can join a group to share the work of reading from a topic. Partitions are assigned using stable round-robin (consumers sorted by ID), so the same consumer always gets the same partitions. When a consumer joins, leaves, or misses its heartbeat (session timeout), the group automatically rebalances.
The server exposes a Prometheus-compatible /metrics endpoint with counters for messages produced/consumed, bytes throughput, error rates, active connections, average produce/consume latency, and per-partition consumer lag.
Skip list memtable, WAL durability, block-based SSTables with CRC32 checksums, L0-L1-L2 compaction with k-way merge.
Leader election, log replication, term management, pluggable transport layer, network partition recovery.
Stable round-robin assignment, auto-rebalance on join/leave, session timeout failure detection, offset tracking.
Throughput counters, latency tracking, connection gauges, partition lag, error rates. HTTP /metrics endpoint.
These are the kinds of problems a distributed message queue like Sentinel solves.
Decouple services so they communicate through events instead of direct calls. The order service publishes "order.created", and the payment, inventory, and notification services each consume it independently. If one is slow or down, the others keep running.
e.g. produce "order.created" → payment service, inventory service, email service all consume
Collect logs from hundreds of application instances into a single stream. Each instance produces log entries to a topic partitioned by service name. A downstream consumer writes them to search infrastructure or long-term storage.
e.g. api-server-1..N → "logs" topic → Elasticsearch consumer
Process a continuous stream of data in real time. Clickstream events, sensor readings, or financial transactions flow through topics. Consumer groups enable parallel processing — each consumer handles a subset of partitions.
e.g. "clicks" topic (8 partitions) → 4 consumers, 2 partitions each → real-time analytics
Distribute CPU-intensive work across a pool of workers. Producers enqueue tasks (image resizing, PDF generation, ML inference). Workers pull from the queue and process at their own pace. Offset tracking ensures no task is lost or duplicated.
e.g. "thumbnails" topic → 10 worker consumers → process and commit offset
Stream database changes as events. Every INSERT, UPDATE, DELETE becomes a message in a topic. Downstream consumers can build search indexes, update caches, replicate to a data warehouse, or trigger workflows — all from the same change stream.
e.g. "users.changes" topic → search indexer, cache invalidator, analytics pipeline
Absorb traffic spikes without overwhelming downstream services. During a flash sale, the API tier produces thousands of order events per second. The queue buffers them, and the fulfillment service consumes at a sustainable rate.
e.g. 10K req/sec spike → "orders" topic → fulfillment at 500/sec, catches up in minutes
Here's what each file does. Everything is in the internal/ directory.
You'll need Go 1.24+ installed. That's it — no other dependencies.
git clone https://github.com/matteso1/sentinel.git
cd sentinelgo test -v ./...
=== RUN TestSkipList_BasicOperations
--- PASS: TestSkipList_BasicOperations (0.00s)
=== RUN TestCluster_LeaderElection
raft_test.go:177: Leader elected: node-0
--- PASS: TestCluster_LeaderElection (0.50s)
=== RUN TestCluster_NetworkPartition
raft_test.go:309: Partitioned leader: node-1
raft_test.go:319: New leader: node-0 (term 4)
--- PASS: TestCluster_NetworkPartition (1.80s)
=== RUN TestLSM_CompactionWithUpdates
INFO compaction complete sourceLevel=0 targetLevel=1 sourceFiles=3
--- PASS: TestLSM_CompactionWithUpdates (0.70s)
...
PASS (45 tests, 0 failures)go run ./cmd/sentinel-server --port 9092 --data ./data
Metrics available at http://localhost:2112/metrics
Sentinel server listening on port 9092go run ./cmd/sentinel-cli produce \
-broker localhost:9092 \
-topic orders \
-partition 0 \
-message "order:12345" \
-count 100
Produced 100 record(s) to orders (offset: 0)go run ./cmd/sentinel-cli consume \
-broker localhost:9092 \
-topic orders \
-partition 0 \
-from-beginning
Consuming from orders (partition 0, offset 0)...
------------------------------------------------------------
offset=0 | key=(null) | value=order:12345-0
offset=1 | key=(null) | value=order:12345-1
offset=2 | key=(null) | value=order:12345-2
...
------------------------------------------------------------
Consumed 100 message(s)curl http://localhost:2112/metrics
sentinel_messages_produced_total 100
sentinel_messages_consumed_total 100
sentinel_produce_latency_ms 0.42
sentinel_active_connections 1
sentinel_uptime_seconds 34.20go test -bench=. ./internal/storage/...
BenchmarkSkipList_Put-18 4229517 266.7 ns/op
BenchmarkSkipList_Get-18 9040447 132.6 ns/op
BenchmarkLSM_Put-18 2618402 415.5 ns/op
# 3.7M writes/sec, 7.5M reads/sec, 2.4M LSM writes/sec