spark on k8s原理学习

# 原理

spark-operator：管理
sprak-driver：调度
node-executor: 具体干活的

1
2
3

# 启动

# driver 启动
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0/jre/bin/java -cp /opt/spark/jars/* -Xmx4g org.apache.spark.deploy.SparkSubmit --deploy-mode client --conf spark.driver.bindAddress=xx.xx.xx.xx --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.JavaWordCount local:///opt/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar oss://{wordcount-file-oss-bucket}/

# spark on k8s 根据下面的配置进行启动
--master k8s://https://10.6.16.32:6443 \
--deploy-mode cluster \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \

1
2
3
4
5
6
7
8

# Operator

SparkApplication
ScheduledSparkApplication

1
2

# 参考文档

上次更新: 2023-10-13 14:07:43

← spark shell和sqark sql 控制台使用 GPT家族常用术语汇总→