Why is .simpleString() method of spark schema truncating my output?

I have a very long schema that I want to return as string

import org.apache.spark.SparkConf;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

...

SparkSession spark = SparkSession.builder().config(new SparkConf().setAppName("YourApp").setMaster("local")).getOrCreate();

Dataset<Row> parquetData = spark.read().parquet("/Users/demo/test.parquet");

String schemaString = parquetData.schema().simpleString();

The problem is the resulting schema looks like (see "10 more fields"):

struct<test:struct<countryConfidence:struct<value:double>,... 10 more fields> etc etc>

Using

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.12</artifactId>
  <version>3.2.4</version>
</dependency>

Is there some configuration option I can use that means .simpleString does not truncate? I've tried parquetData.schema().toDDL(), but it doesn't print the format I need.

If you take a deeper look inside simpleString method, you can see that Spark uses a truncatedString, where SQLConf.get.maxToStringFields is passed as third argument.

The definition of this configuration is described as:

val MAX_TO_STRING_FIELDS = buildConf("spark.sql.debug.maxToStringFields")
  .doc("Maximum number of fields of sequence-like entries can be converted to strings " +
    "in debug output. Any elements beyond the limit will be dropped and replaced by a" +
    """ "... N more fields" placeholder.""")
  .version("3.0.0")
  .intConf
  .createWithDefault(25)

Solution

Tweaking spark.sql.debug.maxToStringFields to a higher number than 25, like 50 (arbitrary, but should be determined by your use case), for example:

SparkSession spark = SparkSession.builder()
  .appName("Spark app name")
  .master("local[*]")
  .config("spark.sql.debug.maxToStringFields", 50)
  .getOrCreate();

Good luck!