Spark Streaming from Kafka has error numRecords must not be negative

Its kind of strange error because I still push data to kafka and consume message from kafka and Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: numRecords must...

Kafka access inside and outside docker

I'm trying to start a kafka service using docker-compose, and it should be able to be accessed inside and outside docker. So, it should be matter of setting the right advertisers inside and...

Kafka, why do I need to pass the certificate chain and not only the root ca when testing with kafka-console-producer (or Kafkcat)

I have a Kafka cluster which is configured to be used over ssl on port 9093. In order for me successfully send data on port 9093 I need to pass the certificate chain (intermediate + root CA) (all...

How to expose Kafka from Docker to the outside world?

Having to communicate with Kafka from a Dockerized Spring-Boot application, the only option I was able to get working was Dockerizing Kafka too. Here is my docker-compose-yml: version:...

Kafkacat: how to delete a topic or all its messages?

I am looking for a way to delete a topic or all its messages using kafkacat. Is it possible or the only way is through the script listed here? bin/kafka-topics.sh --zookeeper localhost:2181...

Can talk to Zookeeper but not to the message brokers

I'm using kafka-python to produce messages for a Kafka 2.2.1 cluster (a managed cluster instance from AWS's MSK service). I'm able to retrieve the bootstrap servers and establish a network...

Kafka Listener is not working! It is isolated in intranet

My Kafka node is hosted in Google Cloud Dataproc. However, we realized that the Kafka installed through default initialization script is set up in such a way that it only allows intranet access....

Kafka MirrorMaker 2.0 duplicate each messages

I am trying to replicate Kafka cluster with MirrorMaker 2.0. I am using following mm2.properties: name = mirror-site1-site2 topics = .* connector.class =...

How to install kafkacat on kafka container resulted from docker-compose

My context is I am trying to create a docker-compose wich will start few containers for running ELK+FileBeats and 3 Kafka containers. The whole docker compose and its yml configuration files can...

Check my Kafka and Zookeeper functionality and connections

As part of my system, I need to use Kafka and Zookeeper cluster on top of Kubernetes. I'm using Statefulset for deploying them and use headless service for Kafka's Broker to be able to talk with...

Cannot connect to kafka which is outside of network

I'm trying to connect my job inside a container which sends events to kafka cluster in another container. No matter what i've tried, i can't send the event to kafka topic I 've tried telnet and...

kubectl exec fails with the error "Unable to use a TTY - input is not a terminal or the right kind of file"

I am running a jenkins pipeline with the following command: kubectl exec -it kafkacat-5f8fcfcc57-2txhc -- kafkacat -b cord-kafka -C -t BBSim-OLT-0-Events -o [email protected] which is running fine on...

Connecting to kafka from another namespace inside k8s

I have the following configuration for my kafka and zookeeper in my minikube: apiVersion: v1 kind: Service metadata: name: kafka-service namespace: kafka spec: selector: app: kafka ...

Cannot write to kafka, brokers are down

I'm running Kafka locally. When I try to write to Kafka I get the following error: kafkacat -b localhost:9092 -t req -T -P -l msgs hello world % ERROR: Local: Broker transport failure:...

Kafkacat Produce message from a file with headers

I need to produce batch messages to Kafka so I have a file that I feed kafkacat: kafkacat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs The content of /tmp/msgs is as follows -H...

How to sink structured records directly from KSQL into a connector (e.g., InfluxDB)

I'm trying to sink data directly from KSQL into InfluxDB (or any other connector that would require definitions). I'm able to get things working in the simple case, but I start having trouble when...

Healthcheck not working at all when using docker-compose (My service do not wait for Kafka to be started before launching)

I have three services on my docker-compose: version: '3.4' setup-topics: image: 'bitnami/kafka:2' hostname: setup-topics container_name: setup-topics command: "bash -c 'echo...

NiFi in docker container fails to talk to kafka: TimoutException, kafkacat ist working just fine

I have set up NiFi (1.11.4) & Kafka(2.5) via docker (docker-compose file below, actual NiFi flow definition https://github.com/geoHeil/streaming-reference). When trying to follow up on basic...

Set timestamp using kafkacat when producing

I want to be able to transmit messages to kafka using kafkacat but also set the timestamp of the messages. But it seems like this is not possible. Is this true or is there something that I am...

Unable to produce/consume to Kafka on Kubernetes when using loadBalancer Service

Background I am running Kafka on kuberentes using confluent open source helm charts. I already have an eks cluster running with managed node groups. When i expose the brokers using NodePort it...

kafkacat for event hub

hi I tried to check connection to eventhub using kafkacat in one of my VM in azure I gave the following parametrs(fill my hab name and all): kafkacat \ -b...

Kafka on Kubernetes in minikube not working

I am trying to setup Kafka on Minikube, a very basic setup. I can't validate if Kafka and Zookeeper have been setup correctly because kafkacat fails. Here is my config: zookeeper kind:...

container startup issue for k8 for kafka & zookeeper

i am trying to create a spring boot producer and consumer with k8 zoo keeper & kafka but not able to set the k8 deployment its failing Not sure what is wrong configured here becoz same...

Kafka Java Producer is not able to send message to kafka instance

I am running a kafka instance in a docker container with following docker-compose.yml file. version: "3" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: -...

How to debug AvroConsumer in confluent kafka?

I'm trying to read Kafka from python but recieve message is None , No errors in CLI. I'm using port forwarding to destination host via putty, and than test ports over telnet - it work's...

Serialization for @KafkaListener

I'm using spring-boot 2.3.2.RELEASE with spring-kafka -> 2.5.4.RELEASE kafka-clients -> 2.5.0 I have the following simple listener @Slf4j @Component class SampleKafkaListener { private final...

Problem with ADVERTISED_LISTENER on macos

I start kafka with this docker-compose.yml on my Mac: mydb: [...] environment: kafka_bootstrap_servers: kafka:9093 kafka: image: wurstmeister/kafka:2.13-2.6.0 ports: -...

Kafka issue WARN [Producer clientId=console-producer] Error connecting to node.......java.net.UnknownHostException

I am trying to setup kafka in kubernetes. Everything starts fine including zookeeper and kafka-server as per their logs but when i try to produce message on a topic it says "Error connecting to...

Apache Beam Python SDK ReadFromKafka does not receive data

I'm trying out a simple example of reading data off a Kafka topic into Apache Beam. Here's the relevant snippet: with beam.Pipeline(options=pipeline_options) as pipeline: _ = ( ...

Consuming with Kafkacat shows no messages, although using -o beginning

I'm trying to consume from a topic using Kafkacat , using the following command: kafkacat -b *****:9093 -C -K\t -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=****...