pyspark log4j2: How to log full exception stack trace?

I tried

• logger.error('err', e)
• logger.error('err', exc_info=e) # syntax for python's logging

>>> 
>>> logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger('my-logger')
>>> 
>>> try: 1/0
... except Exception as e: logger.error('err', e)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1313, in __call__
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1283, in _build_args
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1283, in <listcomp>
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 298, in get_command_part
AttributeError: 'ZeroDivisionError' object has no attribute '_get_object_id'
>>> 
>>> 
>>> 
>>> try: 1/0
... except Exception as e: logger.error('err', exc_info=e)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: __call__() got an unexpected keyword argument 'exc_info'
>>> 

of course I can convert the stacktrace myself and pass it as a string to log4j instead of exception object. But don't wanna do all that if I can avoid it.

>>> try: 1/0
... except Exception as e: l.error(f'err {"".join(traceback.TracebackException.from_exception(e).format())}')
... 
23/03/09 11:38:47 ERROR my-logger: err Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

>>> 

tl;dr - We can not pass a pure python object in a py4j function call, because that object doesn't exist in target JVM.

Reason for AttributeError: 'ZeroDivisionError' object has no attribute '_get_object_id' error is that: org.apache.logging.log4j.Logger.info/err() expects a Java exception object to be passed to it.

For that to happen:

1. the exception should originate in the JVM (or see note below)
2. your python code should get a handle to that Java exception object as a python wrapper object (py4j.java_gateway.JavaObject).
3. pass that wrapper object as a param to Logger.info/error()

NOTE: If there is a pure python Error/Exception object that originated in python's runtime, then you must create a corresponding Java Exception object in JVM, then pass a handle (wrapper py4j.java_gateway.JavaObject) to that object to Logger.info/error().

We ended up just passing it as a string as part of the log message.

>>> 
>>> import traceback
>>> def e2s(ex: Exception):
...     return {''.join(traceback.TracebackException.from_exception(ex).format())}
... 
>>> 
>>> l = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger('my_logger')
>>> 
>>> try: 1/0
... except Exception as e: l.error(f'Some exception: {e2s(e)}')
... 
ERROR Some exception: {'Traceback (most recent call last):\n  File "<stdin>", line 1, in <module>\nZeroDivisionError: division by zero\n'}
>>>