Volume Mount in SparkApplication resource not working

I am toying with the spark operator in kubernetes, and I am trying to create a Spark Application resource with the following manifest.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark-jobs
spec:
  batchScheduler: volcano
  batchSchedulerOptions:
    priorityClassName: routine
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "<image_name>"
  imagePullPolicy: Always
  mainApplicationFile: local:///spark-files/csv_data.py
  arguments:
  - "10"
  sparkVersion: "3.0.0"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  timeToLiveSeconds: 86400
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.0.0
    serviceAccount: driver-sa
    volumeMounts:
    - name: sparky-data
      mountPath: /spark-data
  executor:
    cores: 1
    instances: 2
    memory: "512m"
    labels:
      version: 3.0.0
    volumeMounts:
    - name: sparky-data
      mountPath: /spark-data
  volumes:
  - name: sparky-data
    hostPath:
      path: /spark-data

I am running this in kind, where I have defined a volume mount to my local system where the data to be processed is present. I can see the volume being mounted in the kind nodes. But when I create the above resource, the driver pod crashes by giving the error 'no such path'. I printed the contents of the root directory of the driver pod and I could not see the mounted volume. What is the problem here and how do I fix this?

The issue is related to permissions. When mounting a volume to a pod, you need to make sure that the permissions are set correctly. Specifically, you need to make sure that the user or group that is running the application in the pod has the correct permissions to access the data.You should also make sure that the path to the volume is valid, and that the volume is properly mounted.To check if a path exists, you can use the exec command:

kubectl exec <pod_name> -- ls

Try to add security context which gives privilege and access control settings for a Pod

For more information follow this document.