Kafka connect confluent elasticsearch sink (no class found error)

I am very new to Kafka connect. I want to push my messages from Kafka topic to elasticsearch. After following the available documentation.. I downloaded and compiled elastic search sink from...

Kafka-connect sink task ignores file offset storage property

I'm experiencing quite weird behavior working with Confluent JDBC connector. I'm pretty sure that it's not related to Confluent stack, but to Kafka-connect framework itself. So, I define...

Connect to Kafka running in Docker

I setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3). In addition, I also exposed Zookeeper's port 2181 and Kafka's port...

Schema-Registry rejects unchanged schema as incompatible

We have a kafka cluster running with Avro schemas stored in Confluent's Schema-registry. On a recent re-deploy of (one of) our streams applications we starting seeing incompatible schema errors on...

how to get partition info and offset for Kafka topic without knowing consumer group info

I am totally a squat in Kafka land If I run the command /cfintools/confluent-4.0.0/bin/kafka-avro-console-consumer --topic $t --bootstrap-server $bt --consumer.config...

Not able to access kafka(confluent) installed on Azure VM using public IP

I have installed confluent-oss-5.0.0 on Azure VM and exposed all necessary ports to access using public IP Address. I tried to change the etc/kafka/server.properties below things to achieve but no...

Asynchrony with Kafka C# Client

I am trying to use Kafka for a pub/sub architecture in a .NET Core project. But Confluent's Kafka c# client only offers a synchronous "Consume" method for listening to published messages. Here is...

Spring Boot Embedded Kafka can't connect

I'm trying to write an integration test for my Kafka consumer. I've followed through with the official reference documentation but when I start my test I only see this repeated ad...

Confluent Kafka Golang Client Producer "Broker: Not enough in-sync replicas"

I am attempting to test out a producer writing messages to a topic on a kafka cluster using the Golang client. This works fine writing to a topic on a local cluster, I just copied and pasted the...

How to convert uint8 array to string

What's the best way to convert from []uint8 to string? I'm using http://github.com/confluentinc/confluent-kafka-go/kafka To read events from kafka. But it does not return plain string event. It...

Use Confluent Hub without Confluent Platform installation

I am using these installation instructions for getting confluent hub client https://docs.confluent.io/current/connect/managing/confluent-hub/client.html But, when I get to the line to install the...

Kafka does not always crash on failing flush

I have a Kafka topic named a_topic_that_does_not_exist and I want to publish a message to that topic using Confluent Kafka Python client. As the name suggests, the above topic does not exist, so I...

Do Quarkus & MicroProfile supports Confluent Schema Registry Avro Schema?

Confluent SchemaRegistry not working with Quarkus and MicroProfile end up with the below error ***********ERROR Caused by: io.confluent.common.config.ConfigException: Missing required ...

Kafka Connect assigns same task to multiple workers

I'm using Kafka Connect in distributed mode. A strange behavior I observed multiple times now is that, after some time (can be hours, can be days), what appears to be a balancing error happens:...

Mirrormaker2.0 vs confluent replicator

Apache Kafka introduced Mirrormaker2(MM2) with the release of Kafka 2.4. MM2 is significantly better than MM1. I know that from the architecture standpoint, MM1 used to work using producer and...

Unable to resolve confluent kafka dependencies in gradle?

While adding dependencies for confluent kafka in build gradle file, its unable to resolve it. compile group: 'io.confluent', name: 'kafka-avro-serializer', version: '4.0.0' compile group:...

How to get full document when using kafka mongodb source connector when tracking update operations on a collection?

I am using Kakfa MongoDB Source Connector [https://www.confluent.io/hub/mongodb/kafka-connect-mongodb] with confluent platform v5.4. Below is my MongoDB Source Connector config { "name":...

confluent-kafka Python library consumer.poll(timeout) doesn't work as expected

when I set msg = consumer.poll(timeout=10.0) the consumer waits 10 seconds and return None as expected, but when I change this to msg = consumer.poll(timeout=3600.0) this consumer just return None...

How to connect Kafka with Camel Spring Boot

What options do I have for connecting Kafka with Spring Boot Camel? I'm running ActiveMQ Artemis and Camel to make JMS/MQTT and REST routes to/from clients. I would like to add Kafka to that duo...

Quarkus SmallRye Messaging Kafka Avro native compilation failed

I'm using SmallRye Reactive Messaging with Kafka with the Confluent Registry and AVRO. It works fine as explained in this blog https://quarkus.io/blog/kafka-avro/ But it seems not work in native...

Teradata and Apache Kafka using Kafka connect

Apologies if my question is all over the place but I am new to Apache Kafka and currently I am trying to import data from a Teradata table (on a VM) into Kafka using my Windows OS. So far I've...

Kafka S3 Source Connector

I have a requirement where sources outside of our application will drop a file in an S3 bucket that we have to load in a kafka topic. I am looking at Confluent's S3 Source connector and currently...

Confluent Schema registry failed failed POST request

My local dev Kafka schema registry was working and I was able to POST the schema from my producer and get an ID back. I use auto-register=false, same as my production server. But I made an update...

InvalidPidMappingException causes kafka stream application to shut down

I have this application written in kafka streams. Every now and then it throws InvalidPidMappingException. Caused by: org.apache.kafka.common.KafkaException:...

Confluent/ Kafka support for the upcoming Java 17 LTS

Has Confluent shared any plans for supporting the upcoming Java 17 LTS release? I am mostly interested in the java client libraries. The official documentation does not mention any version after...

Kafka connector "Unable to connect to the server" - dockerized kafka-connect worker that connects to confluent cloud

I'm following similar example as in this blog post: https://rmoff.net/2019/11/12/running-dockerised-kafka-connect-worker-on-gcp/ Except that I'm not running kafka connect worker on GCP but...

Flink Kafka : Expecting type to be a PojoTypeInfo

My customer class is already created using maven-avro plugin.When i try to run this program i am getting error as Exception in thread "main" java.lang.IllegalStateException: Expecting type to be a...

Unable to debug Go code on Windows using VSCode/GoLand (Getting '%1 is not a valid Win32 application')

When I am trying to debug this Go application using VS Code and dlv, I get the following error: Failed to launch: could not launch process:...

Kafka connect failed with OCI Stream service

Im trying to setup Kafka connect with OCI Stream Service. Below is Kafka connect docker configuration, connect: image: confluentinc/cp-kafka-connect-base:7.0.0 hostname: connect ...

Apple M1: can't start the confluent control center using docker

I would like to start the Confluent control center using docker as prescribed in the document: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html This is the...