How the executor memory is determined while running pyspark in local mode?

If I submit spark program as

->submit-spark --driver-memory 500M --executor-memory 300M spark_dataframe_example.py

then executor memory allocated is "110 MiB" even though I have set it as "300 M" .See image below:-

If I submit spark program as

-> spark-submit --driver-memory 1G --executor-memory 200M --num-executors 2 spark_dataframe_example.py

then executor memory allocated is "413.9 MiB" even though I have set it as "200 M".See image below:-

So could someone confirm how this executor memory is allocated?

As was said in the comments, the --executor-memory flag is ignored in local mode. To confirm this, try running your spark-submit command with a ridiculously high --executor-memory flag (bigger than your machine): it won't complain because it is ignored.

Running Spark in local mode is a bit of an exception, since your driver and executor run inside of a single JVM. So the value that counts here is your --driver-memory flag.

Now, where do those 110MB and 413.9MB come from? In version 3.3.2 - the most recent version at the time of this post - they are the result of this calculation:

    val systemMemory = conf.get(TEST_MEMORY)
    val reservedMemory = conf.getLong(TEST_RESERVED_MEMORY.key,
      if (conf.contains(IS_TESTING)) 0 else RESERVED_SYSTEM_MEMORY_BYTES)

    // skipping some irrelevant lines
    ...

    val usableMemory = systemMemory - reservedMemory
    val memoryFraction = conf.get(config.MEMORY_FRACTION)
    (usableMemory * memoryFraction).toLong

There are 3 values which we need to find out where they come from:

• systemMemory comes from TEST_MEMORY which finally is the Xmx value of your running JVM process. This will always be a bit smaller than your --driver-memory but close to it. You don't set this yourself, Spark does that for you in function of your memory requirements.
• reservedMemory will be defined by RESERVED_SYSTEM_MEMORY_BYTES (since we're not in a testing scenario) and this value is 314572800 (300MB).
• memoryFraction is 0.6 in the default scenario, which seems to be the case for you.

The final calculation is thus: `(systemMemory - reservedMemory)*memoryFraction

Now we can do our calculations!

Your first case

--driver-memory was 500M, so let's calculate:

• (524288000 - 314572800) * 0.6 = 125829120 = 120MB
• since we know the Xmx value of your JVM process is close to, but smaller than --driver-memory this is very close to 110MB!

Your second case

--driver-memory was 1G, so let's calculate:

• (1048576000 - 314572800) * 0.6 = 440401920 = 420MB

• since we know the Xmx value of your JVM process is close to, but smaller than --driver-memory this is very close to 413.9MB!