Comments convention "// $example on:" and "// $example off:" in Scala and Java

I found it in "examples" folder of standard Spark distribution, comments such as this:

// $example on:programmatic_schema$
import org.apache.spark.sql.Row
// $example off:programmatic_schema$
// $example on:init_session$
import org.apache.spark.sql.SparkSession
// $example off:init_session$
// $example on:programmatic_schema$
// $example on:data_types$
import org.apache.spark.sql.types._
// $example off:data_types$
// $example off:programmatic_schema$

object SparkSQLExample {

  // $example on:create_ds$
  case class Person(name: String, age: Long)
  // $example off:create_ds$

Really hard to find what is it for, I suspect for some auto-documentation tool? Same with Java and Scala.

Spark uses a custom Jekyll plugin to generate their documentation, called include_example.rb. This allows them to use the include_example tag in their Markdown sources to include the file from the repo.

The plugin contains this description:

> # Select lines according to labels in code. Currently we use "$example on$" and "$example off$"
> # as labels. Note that code blocks identified by the labels should not overlap.

Thus, these comments are there so that they can auto-generate their documentation better.

The file you have shown in the question is included in getting-started.md. Via {% include_example create_df scala/org/apache/spark/examples/sql/SparkSQLExample.scala %}. You can see how this looks fully rendered in Getting Started - Spark 3.0.0 Documentation.

As you can see, they use those tags to strip out irrelevant information/boilerplate of each language, and only show specific bits. The different labels allow them to select different bits of the file.