How to send key, value messages with the kafka console producer
I found out the solution after some research and the solution is here.
kafka-console-producer.sh --broker-list localhost:9092 --topic topic-name --property "parse.key=true" --property "key.separator=:"
After running this command you will enter in producer console and from there you can send key, value messages.
key1:value1 key2:value2 key3:value3
For more clarity, I am providing sample key-value message here,
emp_info is a key and
JSON object is a value.
Note: Simply sending lines of text will result in messages with
null keys. In order to send messages with both
keys and values you must set the
key.separator properties on the command line when running the producer.
By default, the producer doesn’t care about the topic-partition on which the messages are written to and will balance the messages fairly over all the partitions of a topic. Producer picks the partition based on the hash of the record’s key or in round-robin fashion if the record has no key.
Kafka uses the key to specify the target partition. The default
strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.
Kafka works with key-value pairs, if key not specified, it will be considered default as null and partition will identified as round-robin fashion.
If we specify key, messages/records with same key goes to same partition
To enable sending full key-value pairs, from the command-line, we need to use two properties as below:
parse.key : if it’s true – key is mandatory, by default it’s set as false.
key.separator : as below
Kafka Console Producer command
kafka-console-producer --broker-list MY-KAFKA:29092 --topic kafka-prod --property parse.key=true --property key.separator=,