pyspark.sql.DataFrameWriter.text#

DataFrameWriter.text(path, compression=None, lineSep=None)[source]#

Saves the content of the DataFrame in a text file at the specified path. The text files will be encoded as UTF-8.

New in version 1.6.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

pathstr: the path in any Hadoop supported file system

Other Parameters

Extra options: For the extra options, refer to Data Source Option for the version you use.

Notes

The DataFrame must have only one column that is of string type. Each row becomes a new line in the output file.

Examples

Write a DataFrame into a text file and read it back.

>>> import tempfile
>>> with tempfile.TemporaryDirectory(prefix="text") as d:
...     # Write a DataFrame into a text file
...     df = spark.createDataFrame([("a",), ("b",), ("c",)], schema=["alphabets"])
...     df.write.mode("overwrite").text(d)
...
...     # Read the text file as a DataFrame.
...     spark.read.schema(df.schema).format("text").load(d).sort("alphabets").show()
+---------+
|alphabets|
+---------+
|        a|
|        b|
|        c|
+---------+