#Anyone running kafka as service

1 messages · Page 1 of 1 (latest)

tiny totem
hushed heart
thorn oar
#

From my memories of docker networking… The biggest problem will likely be mismatch between what zookeeper expects from the network, and what dagger can promise

#

zookeeper needs a LOT of control. Maybe more than Dagger can reasonably offer. But maybe not! It could go either way IMO.

#

@hushed heart can you live with single-node ZK? Setting up the cluster with full quorum will be one area of complication

#

My biggest fear is actually that it works accidentally. Dagger doesn’t have an equivalent to docker networks. What we have is closer to the now deprecated docker “links” : point-to-point dependencies between containers. There’s no guarantee that containers can communicate if they’re not linked by a service dependency- but they might anyway in the current implementation.

There’s also the issue of the globally unique (content-addressed) hostname: the role it plays in getting kafka & zk service discovery to work; and if it’s possible to use it for that purpose today

hushed heart
#

The idea is to run some e2e tests, so a single node zk is probably okay.

fierce mica
thorn oar
#

single-node kafka too?

fierce mica
#

looks like with this Kraft thing, you can run kafka withing a single container as the controller is embedded in kafka itself. It's also the default mode enabled in kafka now

image.png
thorn oar
#

Ok I was looking at the docker compose example linked above which seems to be 1-node ZK + 2-node kafka. That one I worry a little bit about

verbal wraith
fierce mica
verbal wraith
# fierce mica seems like not even that if using `Kraft` which seems to be the default now in t...

Just try from bitnami docker-compose.yml

$ docker compose up -d 
.... kafka starting ....

And then trying with kafka client from kafka website

$ ./bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

We will encounter an error like:

[2023-06-01 23:11:14,929] WARN [AdminClient clientId=adminclient-1] Error connecting to node d98b336a48d7:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: d98b336a48d7
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:948)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1628)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1494)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:990)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1141)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1401)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
    at java.base/java.lang.Thread.run(Thread.java:833)
#

Looks like kafka bootstrap server trying to point to container id instead of localhost. So I hack it a bit by modify /etc/hosts/ with

d98b336a48d7 127.0.0.1

And run

$ ./bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
Created topic quickstart-events.

And it works!

verbal wraith
# 
package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
    "github.com/wingyplus/must"
)

func main() {
    ctx := context.Background()
    client := must.Must(dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr)))
    defer client.Close()

    kafka := client.
        Container().
        From("docker.io/bitnami/kafka:3.4").
        WithEnvVariable("ALLOW_PLAINTEXT_LISTENER", "yes").
        WithEnvVariable("KAFKA_CFG_ADVERTISED_LISTENERS", "PLAINTEXT://kafka:9092").
        WithExposedPort(9092)

    out := must.Must(
        client.
            Container().
            From("alpine").
            WithServiceBinding("kafka", kafka).
            WithExec([]string{"apk", "add", "wget", "openjdk11", "bash"}).
            WithExec([]string{"wget", "https://dlcdn.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz"}).
            WithExec([]string{"tar", "xzvf", "kafka_2.13-3.4.0.tgz"}).
            WithWorkdir("/kafka_2.13-3.4.0").
            WithExec([]string{"sh", "-c", "./bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server kafka:9092"}).
            Stdout(ctx),
    )
    fmt.Println(out)
}

Hopefully, this can help.

NOTE: kafka client requires bash and need to set KAFKA_CFG_ADVERTISED_LISTENERS to the name of service binding.

fierce mica