作者rleaner (rleaner)
看板R_Language
标题[问题]sparkR.init问题
时间Sun Apr 2 15:26:26 2017
[sparkR.init问题]
我sparkR初始化上一直出问题,
想请教一下前辈们
[目的]=========================================
用spakR进行平行化计算
[系统环境]=======================================
我在vm上模拟一台master和一台node
没有装hadoop直接就装spark,
版本:
R version 3.2.3 (2015-12-10)
ubuntu:16.04
spark:2.1.0
sparkR:1.4.1
[sparkR安装]====================================
如下是我的安装过程(以下只安装在master)
[Part1.下载套件rJava和devtools]
Step1.设订R CMD javareconf
sudo JAVA_HOME=/usr/lib/jvm/jdk/ R CMD javareconf
Step2.更新ubuntu函式库,以下载devtools]
sudo apt-get -y install libcurl4-gnutls-dev libcurl4-openssl-dev libssl-dev
Step3.於R-shell下载rJava和devtools套件
sudo R
install.packages("rJava")
install.packages("devtools")
[Part2.於R-shell用devtools下载sparkR套件、启动]
devtools::install_github('apache/
[email protected]', subdir='R/pkg')
简单来说,我就只是安装rJava套件和devtools
[初始化code]====================================
我是在Rstudio server上启动sparkR的
Sys.setenv(SPARK_HOME='/home/hduser/spark/')
.libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')
library(SparkR, lib.loc = "/home/hduser/spark/R/lib/")
library(devtools)
#初始化sparkr为sc
sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
sqlContext <- sparkRSQL.init(sc)
[错误讯息]=======================================
在sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
这行会出现讯息
17/03/31 02:30:17 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
java.lang.IllegalArgumentException: Invalid type
.......
Error: returnStatus == 0 is not TRUE
附图:
https://www.facebook.com/photo.php?fbid=1490142967725208&set=p.1490142967725208&type=3&theater
https://www.facebook.com/photo.php?fbid=1490143787725126&set=p.1490143787725126&type=3&theater
不知道怎麽了!!!QAQ
希望有贵人指点,感谢
--
※ 发信站: 批踢踢实业坊(ptt.cc), 来自: 140.128.101.15
※ 文章网址: https://webptt.com/cn.aspx?n=bbs/R_Language/M.1491117989.A.24F.html