Apache Spark spark-submit k8s API https ERROR

huangapple go评论72阅读模式
英文:

Apache Spark spark-submit k8s API https ERROR

问题

Spark版本:2.4.4

k8s版本:1.18

我有一个Spark集群和一个k8s集群。

我按照Spark文档:https://spark.apache.org/docs/2.4.4/running-on-kubernetes.html 进行操作。

当我在k8s上使用HTTP代理提交作业时:一切正常。

但是,使用k8s上的本机HTTPS API时,我收到以下错误:

之前我不得不将k8s API证书导入到我的主Spark中(keytool)。

internal.WatchConnectionManager: 执行失败:HTTP 403,状态:403 - 禁止“spark-pi-1598541432880-driver”:用户“system:anonymous”无法在命名空间“default”中观察资源“pods”。
java.net.ProtocolException: 预期的HTTP 101响应,但是实际是'403 Forbidden'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

这很奇怪,因为没有匿名用户。

我已经尝试将“anonymous”用户添加到k8s,但没有效果。

如果我尝试不导入API证书,则会收到以下错误:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:PKIX路径构建失败:sun.security.provider.certpath.SunCertPathBuilderException:无法找到到请求目标的有效认证路径
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)

我提交作业的命令:

-bash-4.2$ spark-submit --master k8s://https://ip:port --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=docker.io/spawnxx/spark:fink-test-2 hdfs://hdfs_ip/user/sacha.pateyron/spark-examples_2.11-2.4.4.jar 1000

Apache Spark本地不支持k8s的https API(因此我必须导入证书)?

有什么想法吗?

英文:

Spark version :2.4.4

k8s version : 1.18

I have a Spark and a k8s cluster.

I followed Spark documentation : https://spark.apache.org/docs/2.4.4/running-on-kubernetes.html

When I submit a job with an HTTP proxy on k8s : everything is ok.

However with the native HTTPS API on k8s I got this error :

Previously I had to import k8s API cert to my master Spark (keytool).

internal.WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 - pods "spark-pi-1598541432880-driver" is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

This is curious because there is no user anonymous.

I already try to add an 'anonymous' user to k8s but it has no effect.

If I try without import the API cert I got this error :

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)

My command to submit my job :

-bash-4.2$ spark-submit --master k8s://https://ip:port  --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=docker.io/spawnxx/spark:fink-test-2 hdfs://hdfs_ip/user/sacha.pateyron/spark-examples_2.11-2.4.4.jar 1000

K8s https API is not supported natively by Apache Spark (so I must import the cert) ?

Any idea ?

答案1

得分: 4

Solution:

HTTPS k8s API使用证书和令牌进行身份验证。

首先下载k8s HTTPS API:

在主节点spark上 -

echo -n|openssl s_client -connect ip_master_k8s:port_https_api|openssl x509 -outform PEM > selfsigned_certificate.pem

在k8s主节点机器上获取spark令牌:

kubectl get secret

kubectl describe secret spark-token-XXX

然后在Spark主节点上使用证书和令牌提交作业:

spark-submit --master k8s://https://ip_master_k8s:port_https_api --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=your_image --conf spark.kubernetes.authenticate.submission.caCertFile=selfsigned_certificate.pem --conf spark.kubernetes.authenticate.submission.oauthToken=spark-token-XXX hdfs://ip_master_hdfs/my_jar
英文:

Solution :

HTTPS k8s API use cert and token for authentication.

First download k8s HTTPS API :

On master spark ->

echo -n|openssl s_client -connect ip_master_k8s:port_https_api|openssl x509 -outform PEM > selfsigned_certificate.pem

On k8s master machine get spark token :

kubectl get secret

kubectl describe secret spark-token-XXX

Then on Spark master we submit a job with cert and token :

spark-submit --master k8s://https://ip_master_k8s:port_https_api  --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.container.image=your_image --conf spark.kubernetes.authenticate.submission.caCertFile=selfsigned_certificate.pem --conf spark.kubernetes.authenticate.submission.oauthToken=spark-token-XXX hdfs://ip_master_hdfs/my_jar

huangapple
  • 本文由 发表于 2020年8月28日 16:08:11
  • 转载请务必保留本文链接:https://go.coder-hub.com/63629870.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定