本文共 1750 字，大约阅读时间需要 5 分钟。

在bin/spark-shell 中会调用spark-submitfunction main() {  if $cygwin; then    # Workaround for issue involving JLine and Cygwin    # (see http://sourceforge.net/p/jline/bugs/40/).    # If you're using the Mintty terminal emulator in Cygwin, may need to set the    # "Backspace sends ^H" setting in "Keys" section of the Mintty options    # (see https://github.com/sbt/sbt/issues/562).    stty -icanon min 1 -echo > /dev/null 2>&1    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Djline.terminal=unix"    "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"    stty icanon echo > /dev/null 2>&1  else    export SPARK_SUBMIT_OPTS    "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"  fi}"bin/spark-submit" 中会继续调用spark-class，并制定执行类org.apache.spark.deploy.SparkSubmitif [ -z "${SPARK_HOME}" ]; then  source "$(dirname "$0")"/find-spark-homefi# disable randomized hash for string in Python 3.3+export PYTHONHASHSEED=0exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"在bin/spark-class中首先找到runner，这里看到是调用java# Find the java binaryif [ -n "${JAVA_HOME}" ]; then  RUNNER="${JAVA_HOME}/bin/java"else  if [ "$(command -v java)" ]; then    RUNNER="java"  else    echo "JAVA_HOME is not set" >&2    exit 1  fifi可以看到是调用java 运行org.apache.spark.launcher.Main 这个类build_command() {  "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"  printf "%d\0" $?}下面将build_command 中的保存到CMDset +o posixCMD=()while IFS= read -d '' -r ARG; do  CMD+=("$ARG")done < <(build_command "$@")最后执行javaCMD=("${CMD[@]:0:$LAST}")exec "${CMD[@]}"最终在launcher.Main 中调用sparksubmit->repl->sparkloop.process->initializespark->createsparkcontext 至此最重要的sparkcontext登场了

转载地址：http://qinmi.baihongyu.com/