0

Imagine that you are working hours with a spark-shell, you have existing Dataframes that you don't want to lose them and later you figure out that you need something from another jar...

Do i have to kill the existing spark-shell session before loading again with --packages?

aironman
  • 837
  • 5
  • 26
  • 55
  • 1
    I have seen how to do this in Java but I'm not sure how it translates to Scala/Spark: https://stackoverflow.com/questions/60764/how-should-i-load-jars-dynamically-at-runtime Truthfully, – Jeremy May 22 '17 at 14:59

1 Answers1

1

Do i have to kill the existing spark-shell session before loading again with --packages?

I doubt it is possible in spark-shell with all the "goodies" to make Spark work nicely with Scala implicits and such.

Even if you managed to create a SparkContext or SparkSession with the new jar file loaded what about the existing data structures you have already created? They use the other "incompatible" session and hence would become unusable (or would lead to hard to trace classloader issues).

So, yes, I'd recommend leaving the existing spark-shell session and start over.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420