Set hadoop configuration values on spark-submit command line

Question

We want to set the aws parameters that from code would be done via the SparkContext:

sc.hadoopConfiguration.set("fs.s3a.access.key", vault.user)
sc.hadoopConfiguration.set("fs.s3a.secret.key", vault.key)

However we have a custom Spark launcher framework that requires all the custom Spark configurations to be done via --conf parameters to the spark-submit command line.

Is there a way to "notify" the SparkContext to set --conf values to the hadoopConfiguration and not to its general SparkConf ? Looking for something along the lines of

spark-submit --conf hadoop.fs.s3a.access.key $vault.user --conf hadoop.fs.s3a.access.key $vault.key

or

spark-submit --conf hadoopConfiguration.fs.s3a.access.key $vault.user --conf hadoopConfiguration.fs.s3a.access.key $vault.key

@vanza -Yes! i was trying to remember that . Please add as answer — WestCoastProjects, Mar 14 '17 at 21:33

score 43 · Accepted Answer · edited Mar 14 '17 at 22:50

43

You need to prefix Hadoop configs with spark.hadoop. in the command line (or SparkConf object). For example:

spark.hadoop.fs.s3a.access.key=value

edited Mar 14 '17 at 22:50

zero323

322,348
103
959
935

answered Mar 14 '17 at 21:37

vanza

9,715
2
31
34

1

Yay! I was looking for it! It's working! That's what SO is for (-: – matemaciek Aug 02 '17 at 15:06
And this ends 2 days of searching. Thank you! – johnnyaug Jun 18 '20 at 11:49
they do not recommend using `s3a://` anymore https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html. Do you know how to setup to use `s3://`? – Gustavo Jan 28 '22 at 12:05

Set hadoop configuration values on spark-submit command line

1 Answers1