How does spark.python.worker.memory relate to spark.executor.memory?

Question

This diagram is quite clear on the relationship between the different YARN and Spark memory related settings, except when it comes to spark.python.worker.memory.

How does spark.python.worker.memory fit into this memory model?

Are the Python processes governed by spark.executor.memory or yarn.nodemanager.resource.memory-mb?

Update

This question explains what the setting does, but doesn't answer the question concerning the memory governance, or how it relates to other memory settings.

Appears to be discussed here http://stackoverflow.com/questions/31565497/what-is-spark-python-worker-memory — David, Apr 13 '16 at 14:02
That answer explains what the setting does, not how it relates to the rest of the spark memory model. — Douglas Willcocks, Apr 14 '16 at 15:19

score 7 · Accepted Answer · answered Apr 14 '16 at 16:01

7

Found this thread from the Apache-spark mailing list, and it appears that spark.python.worker.memory is a subset of the memory from spark.executor.memory.

From the thread: "spark.python.worker.memory is used for Python worker in executor"

answered Apr 14 '16 at 16:01

David

11,245
3
41
46

How does spark.python.worker.memory relate to spark.executor.memory?

1 Answers1

Linked