spark 报错 py4j.protocol.Py4JJavaError错误-昊天SEO

java:0
>>> r = sc.textFile("~/Desktop/heimao.md")
>>> r.take(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 1345, in take
    totalParts = self.getNumPartitions()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 409, in getNumPartitions
    return self._jrdd.partitions().size()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.
: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)
    at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:745)

java:0

>>> r = sc.textFile("~/Desktop/heimao.md")

>>> r.take(1)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 1345, in take

totalParts = self.getNumPartitions()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 409, in getNumPartitions

return self._jrdd.partitions().size()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco

return f(*a, **kw)

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.

: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)

at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:282)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:238)

at java.lang.Thread.run(Thread.java:745)

原因:

 text = sc.textFile("shakespeare.txt")

1 2	text = sc.textFile("shakespeare.txt")

spark 读取本地文件

spark默认去hdfs寻找，然而我的放在本地，应该改为：
默认是从hdfs读取文件，也可以指定sc.textFile("路径").在路径前面加上hdfs://表示从hdfs文件系统上读
本地文件读取 sc.textFile("路径").在路径前面加上file:// 表示从本地文件系统读，如file:///home/user/spark/README.md

 text = sc.textFile("file:///root/shakespeare.txt")

text = sc.textFile("file:///root/shakespeare.txt")

spark 报错 py4j.protocol.Py4JJavaError错误-新乡seo|网站优化,网站建设_微信公众号:zeropython—昊天博客

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！

最热点击文章

WordPress 去除Akismet模块 This site uses Akismet to reduce spam.

Python 执行 javascript PyExecJS 模块使用教程

最新sublime text 3 注册码license 3176 2018年9月更新 [亲测]

关于我

Python 3 执行 JavaScript 脚本,js2py教程

It is a distutils installed project and thus we cannot accurately determine [已解决]

Python3 用xlrd模块读取xls/xlsx文档

Pytest 使用教程

python采集百度新闻源并自动发布文章到phpcms

Firefox 事件绑定监听-爬虫进阶

最新评论

热门标签

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

猜你喜欢

Pyspark DataFrame 保存到MongoDB 数据库

Pyspark rdd 常用的执行动作 Action Pyspark(三)

Pyspark rdd 常用的转换 Transformation Pyspark(二)

Pyspark RDD 概念 属性，如何创建RDD Pyspark(一)

pyspark ipython Notebook 配置

Python3 连接spark,spark集群 [亲测]

发表评论 点击这里取消回复。

关注微信公众号有惊喜 么么哒！

最热点击文章

WordPress 去除Akismet模块 This site uses Akismet to reduce spam.

Python 执行 javascript PyExecJS 模块使用教程

最新sublime text 3 注册码license 3176 2018年9月更新 [亲测]

关于我

Python 3 执行 JavaScript 脚本,js2py教程

It is a distutils installed project and thus we cannot accurately determine [已解决]

Python3 用xlrd模块读取xls/xlsx文档

Pytest 使用教程

python采集百度新闻源并自动发布文章到phpcms

Firefox 事件绑定监听-爬虫进阶

最新评论

热门标签

关注我们的公众号

Pyspark RDD 概念属性，如何创建RDD Pyspark(一)

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！