How to prevent logging of pyspark 'answer received' and 'command to send' messages

You can set logging level for each logger separately

 pyspark_log = logging.getLogger('pyspark')
 pyspark_log.setLevel(logging.ERROR)

I had the same issue, I used following and all worked fine.

pyspark_log = logging.getLogger('pyspark').setLevel(logging.ERROR)
py4j_logger = logging.getLogger("py4j").setLevel(logging.ERROR)
matplotlib_logger = logging.getLogger("matplotlib").setLevel(logging.ERROR)

I was getting some matplotlib lib logs also, so I changed matplotlib logger level too, but if you don't have that issue you can remove that line.

The key component is "py4j". You just need to add a line of code to the beginning of your program:

py4j_logger = logging.getLogger("py4j").setLevel(logging.INFO)

Or just:

logging.getLogger("py4j").setLevel(logging.INFO)