How to run Apache Spark 2.1 Driver Program in docker container with bridge mode

I’m trying to connect to Standalone Apache Spark cluster from a dockerized Apache Spark application using Client mode with a shaded jar.

I have set following properties

  • How do I compile python for Alpine?
  • Using a Docker image to build and deploy as part of a build matrix
  • Differences in the ways of creating a Docker swarm
  • Using Amazon ECS with google container registry
  • Sort by memory usage in docker stats
  • How to write parameterized Bash alias for Docker command
  •  - spark.driver.port
     - spark.driver.blockManager.port
     - spark.driver.bindAddress with Docker Host's IP
     - SPARK_PUBLIC_DNS environment variable with Docker Host's IP.

    I have exposed and mapped the set ports with host ports. The application is shown running on Spark Master, but no response is being returned from workers.
    Earlier without setting these properties it was shown in waiting state on spark master, after setting these it’s shown as running but no response is being returned from the action, it hangs.

    If i do not set the spark.driver.bindAddress as Docker Host’s IP,on checking the logs on workers the driver-url generated is-

     --driver-url spark://CoarseGrainedScheduler@X.X.X.X:7000
      The X.X.X.X is the container IP.
      All workers binds to container's IP (like, which cannot be
      connected to and from master or driver.
      If i set it to Docker Host's IP, It fails as that IP is not visible within the container, 
      but would be reachable by others as port-forwarding is configured.

    What am i missing? Is there another property that needs to be set?

  • docker-machine error, when I try to create
  • Rails console into Docker on Heroku
  • How does docker0 bridge work internally inside the host?
  • Can Docker map multiple container ports to one host port?
  • docker-py: how can I check if the build was successful?
  • What is the overhead of creating docker images?
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.