spark 报错 py4j.protocol.Py4JJavaError错误-昊天SEO

java:0
>>> r = sc.textFile("~/Desktop/heimao.md")
>>> r.take(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 1345, in take
    totalParts = self.getNumPartitions()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 409, in getNumPartitions
    return self._jrdd.partitions().size()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.
: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)
    at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:745)

java:0

>>> r = sc.textFile("~/Desktop/heimao.md")

>>> r.take(1)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 1345, in take

totalParts = self.getNumPartitions()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 409, in getNumPartitions

return self._jrdd.partitions().size()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco

return f(*a, **kw)

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.

: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)

at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:282)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:238)

at java.lang.Thread.run(Thread.java:745)

原因:

 text = sc.textFile("shakespeare.txt")

1 2	text = sc.textFile("shakespeare.txt")

spark 读取本地文件

spark默认去hdfs寻找，然而我的放在本地，应该改为：
默认是从hdfs读取文件，也可以指定sc.textFile("路径").在路径前面加上hdfs://表示从hdfs文件系统上读
本地文件读取 sc.textFile("路径").在路径前面加上file:// 表示从本地文件系统读，如file:///home/user/spark/README.md

 text = sc.textFile("file:///root/shakespeare.txt")

text = sc.textFile("file:///root/shakespeare.txt")

spark 报错 py4j.protocol.Py4JJavaError错误-新乡seo|网站优化,网站建设_微信公众号:zeropython—昊天博客

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！

最热点击文章

在MacOSX中安装Android SDK

Pycharm Flask 开启 debug 模式坑

安卓 uiautomator设备测试和选择器~Python详解

Python 编码解码

python3 字符串转列表出现\ufeff的解决方法

一篇文章搞定Datetime 模块

mac Pycharm 安装 autopep8

unittest 单元测试教程

Python 3 协程研究

异步任务神器 Celery 简单实用

最新评论

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

猜你喜欢

Pyspark 从Kafka 接收数据

Pyspark DataFrame 保存到MongoDB 数据库

Pyspark rdd 常用的执行动作 Action Pyspark(三)

Pyspark rdd 常用的转换 Transformation Pyspark(二)

Pyspark RDD 概念 属性，如何创建RDD Pyspark(一)

pyspark ipython Notebook 配置

发表评论 点击这里取消回复。

关注微信公众号有惊喜 么么哒！

最热点击文章

在MacOSX中安装Android SDK

Pycharm Flask 开启 debug 模式 坑

安卓 uiautomator设备测试 和选择器~Python详解

Python 编码 解码

python3 字符串转列表出现\ufeff的解决方法

一篇文章搞定Datetime 模块

mac Pycharm 安装 autopep8

unittest 单元测试 教程

Python 3 协程 研究

异步任务神器 Celery 简单实用

最新评论

关注我们的公众号

Pyspark RDD 概念属性，如何创建RDD Pyspark(一)

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！

Pycharm Flask 开启 debug 模式坑

安卓 uiautomator设备测试和选择器~Python详解

Python 编码解码

unittest 单元测试教程

Python 3 协程研究