How to update to SparkR 2.0.0 package in R

Question

I want to update from SparkR 1.4.0 to SparkR 2.0.0, but I get the following error:

had non-zero exit status

This is because SparkR 2.0.0 is not available on CRAN.

Similarly, from SparkR 1.6.2 to SparkR 2.0.0, we get:

Warning in install.packages :
package ‘SparkR’ is not available (for R version 3.3.0)

How does one install Spark 2.0.0 from there? (i dont want to install the SparkR because it's already installed as 1.6.2 nut i have to update it as 2.0.0)

i want to used read.jdbc function which is available in SparkR (2.0.0) — Sahil Desai, Aug 29 '16 at 09:02
Read how to install R packages from source. Additionally it may fails if SparkR added some OS dependency you don't have, you need to read installation output to spot that. — jangorecki, Aug 31 '16 at 10:06
@jangorecki yup i already read it but its for SparkR(1.1.4). — Sahil Desai, Aug 31 '16 at 10:09
Possible duplicate of [Installing of SparkR](http://stackoverflow.com/questions/31184918/installing-of-sparkr) — Béatrice Moissinac, Sep 08 '16 at 20:56

score 3 · Accepted Answer · answered Sep 09 '16 at 08:49

SparkR requires not just an R package but an entire Spark backend to be pulled in. When you want to upgrade SparkR, you are upgrading Spark, not just the R package.

Nowadays you may want to refer to the sparklyr package as it makes all of this a whole lot easier.

install.packages("devtools")
devtools::install_github("rstudio/sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
spark_install(version = "2.0.0")

It also offers more functionality than SparkR.

Installation complete. Warning message: In value[[3L]](cond) : Failed to set logging settings — Sahil Desai, Sep 09 '16 at 09:08

How to update to SparkR 2.0.0 package in R

1 Answers1