How can we load a non delimited text file using spark scala and save it as a CSV file where column lengths are given dynamically?

For ex - if we have three columns Name , address and age as schema and a line in file has 92 chars where first 50 are name, next 40 are address and last 2 chars is age, and if these column lengths might vary and will be given dynamically, how to read a file and make it delimited and save it as text file?

Could not get the idea at all

Your question has two parts.

Read file

I assume you have a file named input.txt like this:

1 John 100
2 Jack 200
3 Jonah300
10JJ   400

I also assumed that these are the columns:

// Column Name, Length
val columns = Vector(("id", 2), ("name", 5), ("value", 3))

You should find starting position for each column:

case class ColumnInfo(name: String, length: Int, position: Int)

val columnInfos = columns.tail.foldLeft(Vector(ColumnInfo(columns.head._1, columns.head._2, 1))) { (acc, current) =>
  acc :+ ColumnInfo(current._1, current._2, acc.last.position + acc.last.length)
}

This will be the result:

Vector(ColumnInfo(id,2,1), ColumnInfo(name,5,3), ColumnInfo(value,3,8))

Now, you can read and parse this file using this code:

val sparkCols = columnInfos map { columnInfo =>
  trim(substring(col("value"), columnInfo.position, columnInfo.length)) as columnInfo.name
}

val df = spark.read
  .text("input.txt")
  .select(sparkCols: _*)

df.show()

This will be the result:

+---+-----+-----+
| id| name|value|
+---+-----+-----+
|  1| John|  100|
|  2| Jack|  200|
|  3|Jonah|  300|
| 10|   JJ|  400|
+---+-----+-----+

Save file

You can save file using this code:

df.repartition(1).write.option("header", true).csv("output.csv")

This will be the result:

id,name,value
1,John,100
2,Jack,200
3,Jonah,300
10,JJ,400