How to send key, value messages with the kafka console producer

I found out the solution after some research and the solution is here.

kafka-console-producer command

kafka-console-producer.sh --broker-list localhost:9092 --topic topic-name --property "parse.key=true" --property "key.separator=:"

After running this command you will enter in producer console and from there you can send key, value messages.

For example

key1:value1
key2:value2
key3:value3

For more clarity, I am providing sample key-value message here, emp_info is a key and JSON object is a value.

emp_info: {"emp_id":100,"first_name":"Keshav","last_name":"Lodhi","designation":"DataEngineer"}

Note: Simply sending lines of text will result in messages with null keys. In order to send messages with both keys and values you must set the parse.key and key.separator properties on the command line when running the producer.

By default, the producer doesn’t care about the topic-partition on which the messages are written to and will balance the messages fairly over all the partitions of a topic. Producer picks the partition based on the hash of the record’s key or in round-robin fashion if the record has no key.

1. Kafka uses the key to specify the target partition. The default
   strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

2. Kafka works with key-value pairs, if key not specified, it will be considered default as null and partition will identified as round-robin fashion.

3. If we specify key, messages/records with same key goes to same partition

4. To enable sending full key-value pairs, from the command-line, we need to use two properties as below:

5. properties

• parse.key : if it’s true – key is mandatory, by default it’s set as false.

• key.separator : as below

Examples:

• key.separator=,
• key.separator=-
• key.separator=:

Kafka Console Producer command

kafka-console-producer --broker-list MY-KAFKA:29092 --topic kafka-prod --property parse.key=true --property key.separator=,

Reference: https://shashirl9.medium.com/kafka-producer-internals-d971ac582688