How to create a Topic in Kafka through Java

I want to create a topic in Kafka (kafka_2.8.0-0.8.1.1) through java. It is working fine if I create a topic in command prompt, and If I push message through java api. But I want to create a topic...

Persisting Spark Streaming output

I'm collecting the data from a messaging app, I'm currently using Flume, it sends approx 50 Million records per day I wish to use Kafka, consume from Kafka using Spark Streaming and persist it to...

Error creating Kafka Topics- Replication factor larger than available brokers

I am trying to create topics in Kafka by following the guide on Apache Kafka website through command line. While running the command: bin/kafka-topics.sh --create --zookeeper localhost:2181...

Launching multiple Kafka brokers fails - Configured brokerId doesn't match stored brokerId

While trying to launch multiple Kafka brokers with different brokerId's. One being the default server.properties and the other being serverTest.properties with 2 lines changed, those being...

spark-streaming-kafka-0-10 auto.offset.reset is always set to none

Does anyone come across the issue when assign auto.offset.reset->"latest" doesn't affect this property in spark-streaming-kafka 0-10 here is my code: val config =...

Kafka Consumer Vs Apache Flink

I did a poc in which I read data from Kafka using spark streaming. But our organization is either using Apache Flink or Kafka consumer to read data from Apache kafka as a standard process. So I...

How to view kafka message

Is there any way I can view the message content sent to kafka for a given topic? Say some thing like view last 5 messages for this topic, if that it possible.

Kafka server not returning anything

Zookeper is up and running in windows env.Now when i start Kafka server using .\bin\windows\kafka-server-start.bat .\config\server.properties nothing is displayed G:\Apache...

Using Kafka with Android

I am trying to do a sample application with kafka to showcase it's streaming capabilities. Here is exactly what I want to do: Create an Android Application to Send Data to Kafka Topics through...

Kafka Stream with Avro in JAVA , schema.registry.url" which has no default value

I have the following configuration for my Kafka Stream application Properties config = new Properties(); config.put(StreamsConfig.APPLICATION_ID_CONFIG,this.applicaionId); ...

Spring Boot Kafka: Verify if the kafka server is running or not

I am writing an edge case of know if the kafka server is running or not, so I have written a scheduler @Service @AllArgsConstructor public class KafkaConnectionVerificationScheduler { private...

Are Kafka partitions consumed evenly?

I have a consumer group with several consumers. Each consumer is assigned to a set of partitions. When the consumer polls for messages where the consumed partition is selected? Is it done on the...

Registering kafkalistenerContainers with KafkaListenerEndpointRegistry

I want to pragmatically control when to start/stop my kafka listeners.So looking through some previous posts and discussions it looks like I could use...

Kafka access inside and outside docker

I'm trying to start a kafka service using docker-compose, and it should be able to be accessed inside and outside docker. So, it should be matter of setting the right advertisers inside and...

How to create kafka compacted topic

I have a Kafka application that has a producer who produces messages to a topic. A consumer then takes the messages from the topic, does some logic to the given messages and then produces them to...

Unable to connect to Kafka Broker on Hortonworks Sandbox (HDP 2.6) from my Windows Host Machine on Port 6667

I have downloaded Hortonworks Sandbox (IP: 192.179.144.XXX) on my Windows 10 laptop. I have the HDP Sandbox running on VMWare. Below is my sandbox information: Created on: 19_04_2017_19_09_16 for...

delete topic not working for kafka java client 2.1.0

I want to delete a kafka topic programatically and struggling a lot to make it work. Am using below maven dependency. <dependency> <groupId>org.apache.kafka</groupId> ...

I can't run Kafka on windows

I'm learning about Microservice Architecture these days and I need to run Kafka to follow some tutorials. However, zookeeper-server-start on cmd which is the first step for running up Kafka...

how to get partition info and offset for Kafka topic without knowing consumer group info

I am totally a squat in Kafka land If I run the command /cfintools/confluent-4.0.0/bin/kafka-avro-console-consumer --topic $t --bootstrap-server $bt --consumer.config...

Kafka Connect - Failed to flush, timed out while waiting for producer to flush outstanding messages

I am trying to use the Kafka Connect JDBC Source Connector with following properties in BULK...

How to setup proxy layer on kafka broker?

I am trying to setup a proxy server (preferred in java) that can pass my connection stream to kafka broker. Client (Consumer/Producer) <--> [PROXY SERVER] <--> Kafka Broker My use case is consume...

Debezium Error : ConnectException: Data row is smaller than a column index

this error is on all my debezium connectors now ( on sql-server), i tried to clear a few K-Connect topics to clear meta data but no luck, any help is appreciated... Have a feel the meta data is...

How to override KafkaListenerEndpointRegistry in KafkaBootstrapConfiguration

I need to add some logic to KafkaListenerEndpointRegistry - i want to register additional listener for every topic (i want to create chain of retry topics consumers with different time of...

Why should I use KStream or KTable?

I read but I couldn't understand too much. I read that I can use KTable instead of log compaction. Or it has many more features. However, I could not find a good example in this regard. I also...

Kafka consumer is taking time to recognize new partition

I was running a test where kafka consumer was reading data from multiple partitions of a topic. While the process was running I added more partitions. It took around 5 minutes for consumer thread...

How to tell spring-kafka consumer to stop consuming, at runtime?

I'm using spring-kafka v2.5.2, with KafkaListener annotations. At runtime, I would like be able to send a signal to a consumer to stop consuming. I see the autoStartup parameter but it seems to...

Confluent Schema registry failed failed POST request

My local dev Kafka schema registry was working and I was able to POST the schema from my producer and get an ID back. I use auto-register=false, same as my production server. But I made an update...

How to Read data from kafka topic with different schema (has some optional objects) in structured streaming

i have data coming in kafka topic which has an optional object , and since its optional i am missing those records when reading with a defined schema ex : schema i have : val schema =new...

Webhook data to Kafka Connect

I'm using Kafka connect to pull data from different places: mysql, mongodb, etc. And send to elastic search. I would like to pull data where the origin is a webhook. So, can I configure some Kafka...

How to use the Tumbling window function for the non keyed streaming data in Flink?

I want to use the tumbling window function for my program (non keyed data) as it is processing streaming data but only 300 messages/sec. I want to take it to at least 5K/sec. For this purpose, I...