pyspark.inheritable_thread_target#

pyspark.inheritable_thread_target(f=None)[source]#

Return thread target wrapper which is recommended to be used in PySpark when the pinned thread mode is enabled. The wrapper function, before calling original thread target, it inherits the inheritable properties specific to JVM thread such as InheritableThreadLocal, or thread local such as tags with Spark Connect.

When the pinned thread mode is off, it return the original f.

New in version 3.2.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters

ffunction, or SparkSession: the original thread target, or SparkSession if Spark Connect is being used. See the examples below.

Notes

This API is experimental.

It is important to know that it captures the local properties or tags when you decorate it whereas InheritableThread captures when the thread is started. Therefore, it is encouraged to decorate it when you want to capture the local properties.

For example, the local properties or tags from the current Spark context or Spark session is captured when you define a function here instead of the invocation:

>>> @inheritable_thread_target
... def target_func():
...     pass  # your codes.

If you have any updates on local properties or tags afterwards, it would not be reflected to the Spark context in target_func().

The example below mimics the behavior of JVM threads as close as possible:

>>> Thread(target=inheritable_thread_target(target_func)).start()  

If you’re using Spark Connect, you should explicitly provide Spark session as follows:

>>> @inheritable_thread_target(session)  
... def target_func():
...     pass  # your codes.

>>> Thread(target=inheritable_thread_target(session)(target_func)).start()  