I want to allow events to be processed in parallel but make sure that events for the same partition key/hash are processed sequentially. How would I do this in Convex?
#Does Convex have a way to do partitioned queues?
safe heath
velvet pollenBOT
Hi @safe heath. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Convex does not have a built-in, first-class partitioned queue feature where events are automatically routed and processed sequentially per partition key/hash. However, you can implement similar behavior using Convex’s transactional guarantees and some design patterns:
-
Queue Pattern with Indexes: You can create a table (e.g., “events”) and use an index on your partition key. When processing, select events for a specific partition and process them in order. Since Convex transactions are serializable, you can ensure that only one worker processes events for a given partition at a time by having your worker claim and update the status of events within a transaction. This prevents race conditions and ensures sequential processing per partition key. However, you would need to implement the logic to ensure only one worker processes a partition at a time, possibly by locking or marking the partition as “in progress” in your table structure. This is similar to the “queue” pattern described for high-throughput mutations, where you process a subset of items based on an index to avoid conflicts and allow parallelism across partitions Optimize Transaction Throughput: 3 Patterns for Scaling with Convex and ACID Databases.
-
Workpool Component: The Workpool component allows you to configure pools with different parallelism, but it does not natively support partitioned sequential processing. You could create a separate workpool per partition, but this is only practical if the number of partitions is small and known in advance Workpool.
- Work Stealing and Sharding: The work stealing pattern described for compute-heavy tasks suggests that you can shard work queues and have workers subscribe to subsets of queues (e.g., by partition key). Each worker would process events for its assigned partition(s) sequentially, and could “steal” work from other partitions if idle. This approach requires you to implement the logic for partition assignment and sequential processing within each partition Implementing work stealing with a reactive database.