pyspark.BarrierTaskContext.barrier¶
-
BarrierTaskContext.barrier() → None[source]¶ Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, this function blocks until all tasks in the same stage have reached this routine.
New in version 2.4.0.
Notes
This API is experimental
In a barrier stage, each task much have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout.