rain sky
I've been working on Walrus, a message streaming system (think Kafka-like) written in Rust. The focus was on making the storage layer as fast as possible.
it has:
1.2 million writes/second on a single node, scales horizontally the more nodes you add
Beats both Kafka and RocksDB in benchmarks (see graphs in README)
How it's fast:
The storage engine is custom-built instead of using existing libraries. On Linux, it uses io_uring for batched writes. On other platforms, it falls back to regular pread/pwrite syscalls. You can also use memory-mapped files if you prefer(although not recommended)
Each topic is split into segments (~1M messages each). When a segment fills up, it automatically rolls over to a new one and distributes leadership to different nodes. This keeps the cluster balanced without manual configuration.
Distributed setup:
The cluster uses Raft for coordination, but only for metadata (which node owns which segment). The actual message data never goes through Raft, so writes stay fast. If you send a message to the wrong node, it just forwards it to the right one.
You can also use the storage engine standalone as a library (walrus-rust on crates.io) if you just need fast local logging.
I also wrote a TLA+ spec to verify the distributed parts work correctly (segment rollover, write safety, etc).
Code: https://github.com/nubskr/walrus
would love to hear your thoughts on it :))
bronze panther
winter folio
humble steeple