How to test whether log compaction is working or not in Kafka?

It is a good point to take a look also on log.roll.hours, which by default is 168 hours. In simple words: even in case you have not so active topic and you are not able to fill the max segment size (by default 1G for normal topics and 100M for offset topic) in a week you will have a closed segment with size below log.segment.bytes. This segment can be compacted on next turn.

In order check a Topics property from CLI you can do it using Kafka-topics cmd :

https://grokbase.com/t/kafka/users/14aev0snbd/command-line-tool-for-topic-metadata

even though this question is a few months old, I just came across it doing research for my own question. I had created a minimal example for seeing how compaction works with Java, maybe it is helpful for you too:

https://gist.github.com/anonymous/f78184eaeec3ee82b15182aec24a432a

Furthermore, consulting the documentation, I used the following configuration on a topic level for compaction to kick in as quickly as possible:

min.cleanable.dirty.ratio=0.01
cleanup.policy=compact
segment.ms=100
delete.retention.ms=100

When run, this class shows that compaction works - there is only ever one message with the same key on the topic.

With the appropriate settings, this would be reproducible on command line.

Actually, the log compaction is visible only when the number of logs reaches to a very high count eg 1 million. So, if you have that much data, it's good. Otherwise, using configuration changes, you can reduce this limit to say 100 messages, and then you can see that out of the messages with the same keys, only the latest message will be there, the previous one will be deleted. It is better to use log compaction if you have full snapshot of your data everytime, otherwise you may loose the previous logs with the same associated key, which might be useful.