How to run Apache Spark 2.1 Driver Program in docker container with bridge mode
I’m trying to connect to Standalone Apache Spark cluster from a dockerized Apache Spark application using Client mode with a shaded jar.
I have set following properties
- spark.driver.port - spark.driver.blockManager.port - spark.driver.bindAddress with Docker Host's IP - SPARK_PUBLIC_DNS environment variable with Docker Host's IP.
I have exposed and mapped the set ports with host ports. The application is shown running on Spark Master, but no response is being returned from workers.
Earlier without setting these properties it was shown in waiting state on spark master, after setting these it’s shown as running but no response is being returned from the action, it hangs.
If i do not set the spark.driver.bindAddress as Docker Host’s IP,on checking the logs on workers the driver-url generated is-
--driver-url spark://CoarseGrainedScheduler@X.X.X.X:7000 The X.X.X.X is the container IP. All workers binds to container's IP (like 172.17.0.45), which cannot be connected to and from master or driver. If i set it to Docker Host's IP, It fails as that IP is not visible within the container, but would be reachable by others as port-forwarding is configured.
What am i missing? Is there another property that needs to be set?