Kafka cluster clarifications | NestJS | Page 1

scarlet pebble Jan 30, 2024, 7:22 AM

#

Hello,

I'm using kafka as a large work queue and microservices as consumers to process and async as much work as I can.

We have 2 very different types of data (analytics/telemetry and customer data) that we'd like to segment into a separate kafka cluster, and I want to ask directly after having spent a while going in circles.

Is it technically possible to set up multiple microservices that each read from their own kafka cluster? (I was setting up a separate config for each microservice, but it seems like they're stepping on each other)
Is it possible to tune the number of consumers on a topic explicitly? or is it all implicit based on server * worker counts?

zinc herald Jan 30, 2024, 9:13 AM

#

This is really more a question for a Kafka community, don't you think?

But, I'll give it a shot...

You should be able to assign each microservice app access to a cluster with the broker property. Is this not working for you?
I believe you can only control the number of consumers via consumer groups. The more groups you have, the more consumers you can have that pull from a (the same) topic.

scarlet pebble Jan 30, 2024, 5:53 PM

#

Thanks for the reply. I've used kafka before in prod, and my questions are about any idiosyncrasies of how nest configures and has opinions about kafka, which I think are very relevant to this discord.

This all spun out of an error I got locally when adding a 2nd microservice, which caused a "address already in use ::3000"

zinc herald Jan 30, 2024, 6:13 PM

#

Yeah, what environment are you running things in?

scarlet pebble Jan 30, 2024, 6:48 PM

#

I was seeing the port 3000 error when spinning up the app locally inside Docker.

I also pass in an empty config for both kafka microservices when im running tests that fully mock out kafka (but the microservice still gets set up)., which might cause the conflict.

latent pasture Feb 2, 2024, 9:59 PM

#

Regarding multiple clusters. It is possible to connect to two separate ones but it's not possible (by default) to discriminate between the topics each of them should consume, because the transportId is the same for both

#

One way to go around it is to extend the ServerKafka and override the transportId property with a custom symbol. Then pass the custom instance as a strategy to microservice options

#

An use the custom transport Id as the second parameter on @EventPattern decorators

#Kafka cluster clarifications