How to run Apache Spark 2.1 Driver Program in docker container with bridge mode

I’m trying to connect to Standalone Apache Spark cluster from a dockerized Apache Spark application using Client mode with a shaded jar.

I have set following properties

  • Why docker (Container-based technologies) are useful
  • Is zap docker generate .xml report?
  • How to change docker bridge address space (subnet)?
  • docker run -v does not work on Windows using docker-toolbox
  • How can I create a docker volume container in specific directory?
  • Dockerized Mongo takes ages to load then doesn't run mongo
  •  - spark.driver.port
     - spark.driver.blockManager.port
     - spark.driver.bindAddress with Docker Host's IP
     - SPARK_PUBLIC_DNS environment variable with Docker Host's IP.
    

    I have exposed and mapped the set ports with host ports. The application is shown running on Spark Master, but no response is being returned from workers.
    Earlier without setting these properties it was shown in waiting state on spark master, after setting these it’s shown as running but no response is being returned from the action, it hangs.

    If i do not set the spark.driver.bindAddress as Docker Host’s IP,on checking the logs on workers the driver-url generated is-

     --driver-url spark://CoarseGrainedScheduler@X.X.X.X:7000
      The X.X.X.X is the container IP.
      All workers binds to container's IP (like 172.17.0.45), which cannot be
      connected to and from master or driver.
      If i set it to Docker Host's IP, It fails as that IP is not visible within the container, 
      but would be reachable by others as port-forwarding is configured.
    

    What am i missing? Is there another property that needs to be set?

  • Why this docker file is not running with a shell script?
  • Background/foreground with docker-compose
  • docker container not able to write on host machine
  • ssh pseudo-tty causes PTY allocation request failed on channel 0
  • Best workflow for developing and debugging - initially deploy in a docker environment
  • Howto run a Prestashop docker container with persistent data?
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.