Implementing Spark in Spring-boot

I am here to share you guys the integration of spark in the spring-boot application.

Usually spark is useful in multi-node/cluster environment.If you are using a single node cluster and using sparing-boot to submit jobs and getting workflow results to show somewhere in your web application.Then this approach will be straight forward way.The spring rest-api will launch spark jobs and the computed results will be given as a response.

yashwanth2804 / spring-spark-example

An example of setting up Spring-Boot with Spark.

spring-spark-example

An example of setting up Spring-Boot with Spark.

View on GitHub

Job Submissions

Typical way

1. CLI

# Run application locally on 8 cores
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local[8] \
  /path/to/examples.jar \
  100

1. CURL

curl -X POST -d http://master-host:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
  "action": "CreateSubmissionRequest",
  "appResource": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
  "clientSparkVersion": "2.0.0",
  "appArgs": [ "10" ],
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass": "org.apache.spark.examples.SparkPi",
  "sparkProperties": {
    "spark.jars": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
    "spark.driver.supervise":"false",
    "spark.executor.memory": "512m",
    "spark.driver.memory": "512m",
    "spark.submit.deployMode":"cluster",
    "spark.app.name": "SparkPi",
    "spark.master": "spark://master-host:6066"
  }
}

Springboot rest api

http://localhost:8056/api/mongodb