spark 报错 py4j.protocol.Py4JJavaError错误-昊天SEO

java:0
>>> r = sc.textFile("~/Desktop/heimao.md")
>>> r.take(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 1345, in take
    totalParts = self.getNumPartitions()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/rdd.py", line 409, in getNumPartitions
    return self._jrdd.partitions().size()
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/<span class='wp_keywordlink'><a href="http://www.168seo.cn/python" title="python">python</a></span>/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/<span class='wp_keywordlink_affiliate'><a href="https://www.168seo.cn/tag/pyspark" title="View all posts in pyspark" target="_blank">pyspark</a></span>/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.
: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
    at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)
    at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:745)

java:0

>>> r = sc.textFile("~/Desktop/heimao.md")

>>> r.take(1)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 1345, in take

totalParts = self.getNumPartitions()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/rdd.py", line 409, in getNumPartitions

return self._jrdd.partitions().size()

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco

return f(*a, **kw)

File "/Users/songhao/apps/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o32.partitions.

: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/songhao/Desktop/demo/log_168/~/Desktop/heimao.md

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

at scala.Option.getOrElse(Option.scala:121)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)

at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)

at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:282)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:238)

at java.lang.Thread.run(Thread.java:745)

原因:

 text = sc.textFile("shakespeare.txt")

1 2	text = sc.textFile("shakespeare.txt")

spark 读取本地文件

spark默认去hdfs寻找，然而我的放在本地，应该改为：
默认是从hdfs读取文件，也可以指定sc.textFile("路径").在路径前面加上hdfs://表示从hdfs文件系统上读
本地文件读取 sc.textFile("路径").在路径前面加上file:// 表示从本地文件系统读，如file:///home/user/spark/README.md

 text = sc.textFile("file:///root/shakespeare.txt")

text = sc.textFile("file:///root/shakespeare.txt")

spark 报错 py4j.protocol.Py4JJavaError错误-新乡seo|网站优化,网站建设_微信公众号:zeropython—昊天博客

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！

最热点击文章

Mac Flask 报错:RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encodin

vim YouCompleteMe Python3 组合安装

Python 的 type 和 object 之间是怎么一种关系？{转}

Centos 安装 Kafka 后台启动教程

（解决方法）MySQL 8 + macOS 错误：Authentication plugin ‘caching_sha2_password’ cannot be loaded

pipenv gives TypeError: ‘module’ object is not callable [已解决]

Mac 安装 Mysql 教程 brew安装

为什么用pycharm在同目录下import，pycharm会报错，但是实际可以运行？

Pymong pymongo.errors.AutoReconnect: 127.0.0.1:27017 [已解决]

Kafka.errors.NoBrokersAvailable: NoBrokersAvailable. [亲测]

最新评论

spark 报错 py4j.protocol.Py4JJavaError错误

spark 读取本地文件

pyspark ipython Notebook 配置

闽系“汇泉西悦城”，业主维权，黑心开发商

猜你喜欢

Pyspark 从Kafka 接收数据

Pyspark DataFrame 保存到MongoDB 数据库

Pyspark rdd 常用的执行动作 Action Pyspark(三)

Pyspark rdd 常用的转换 Transformation Pyspark(二)

Pyspark RDD 概念 属性，如何创建RDD Pyspark(一)

pyspark ipython Notebook 配置

发表评论 点击这里取消回复。

关注微信公众号有惊喜 么么哒！

最热点击文章

Mac Flask 报错:RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encodin

vim YouCompleteMe Python3 组合安装

Python 的 type 和 object 之间是怎么一种关系？{转}

Centos 安装 Kafka 后台启动 教程

（解决方法）MySQL 8 + macOS 错误：Authentication plugin ‘caching_sha2_password’ cannot be loaded

pipenv gives TypeError: ‘module’ object is not callable [已解决]

Mac 安装 Mysql 教程 brew安装

为什么用pycharm在同目录下import，pycharm会报错，但是实际可以运行？

Pymong pymongo.errors.AutoReconnect: 127.0.0.1:27017 [已解决]

Kafka.errors.NoBrokersAvailable: NoBrokersAvailable. [亲测]

最新评论

关注我们的公众号

Pyspark RDD 概念属性，如何创建RDD Pyspark(一)

发表评论点击这里取消回复。

关注微信公众号有惊喜么么哒！

Centos 安装 Kafka 后台启动教程