Why is PySpark logger not logging INFO statements?

In the PySpark code below, I am attempting to ensure INFO statements are logged. However, I am only seeing WARN, ERROR and FATAL messages. How do I update the logger ('Example Processor') to have a log level of INFO, and log everything out?

Note: DEBUG logging works, and logs everything for DEBUG, INFO, WARN, ERROR, and FATAL. Thanks!

    self.spark = SparkSession.builder \
        .master("local[1]") \
        .appName("DemoProcessor") \
        .getOrCreate()
    

    log4jLogger = self.spark.sparkContext._jvm.org.apache.log4j
    self.log = log4jLogger.LogManager.getLogger('Example Processor')

    self.log.setLevel(log4jLogger.Level.INFO)
    # self.log.setLevel(log4jLogger.Level.DEBUG)
    self.log.trace("Trace Message!")
    self.log.debug("Debug Message!")
    self.log.info("Info Message!")
    self.log.warn("Warn Message!")
    self.log.error("Error Message!")
    self.log.fatal("Fatal Message!")

INFO level Log:

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/09 17:22:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/02/09 17:22:44 WARN Example Processor: Warn Message!
23/02/09 17:22:44 ERROR Example Processor: Error Message!
23/02/09 17:22:44 FATAL Example Processor: Fatal Message!

DEBUG level log:

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/09 17:35:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/02/09 17:35:01 INFO Example Processor: Info Message!
23/02/09 17:35:01 DEBUG Example Processor: Debug Message!
23/02/09 17:35:01 WARN Example Processor: Warn Message!
23/02/09 17:35:01 ERROR Example Processor: Error Message!
23/02/09 17:35:01 FATAL Example Processor: Fatal Message!

It seems like the logging level is not being set correctly for the logger 'Example Processor'. In the PySpark code, you are trying to set the log level to log4jLogger.Level.INFO, but it's not taking effect.

One thing to check is to make sure that the log level is not being overridden elsewhere in your code. For example, if you have the following code in your script, it will set the log level for the root logger to log4jLogger.Level.WARN:

log4jLogger.LogManager.getRootLogger().setLevel(log4jLogger.Level.WARN)

This would explain why you're only seeing WARN, ERROR, and FATAL messages, even though you set the log level for the 'Example Processor' logger to log4jLogger.Level.INFO.

To resolve the issue, you can either remove the line setting the log level for the root logger or set the log level for the root logger to a more verbose level, such as log4jLogger.Level.DEBUG.