How can I connect to JMX through Kubernetes managed Docker containers?

I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error:

java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
     java.net.ConnectException: Connection timed out
     ...

Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:

  • Docker-Machine stuck on creating default
  • Not able to access tomcat application on Docker VM with host(windows) IP while using docker toolbox
  • How to check if the docker image has all the files?
  • How to scale application with multiple exposed ports and multiple volume mounted by using docker swarm?
  • Why do we need run update before run install in Dockerfile?
  • libicui18n.so.52: cannot open shared object file
  • jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi 
    

    I know that JMX opens a random high port, and I have tried to resolve that by including a custom @Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran

    netstat -tulpn
    

    to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.

    As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:

    -Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0
    

    The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.

    I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.

    Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.

    Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?

    EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?

  • Taking files off a docker container
  • How to make pull public and push requiring login in docker registry v2?
  • Alpine based Docker images fully compatible on Ubuntu host?
  • Kubernetes. HTTPS API return `Unauthorized`
  • How to create an serverless infrastructure with docker?
  • When, where and how does Visual Studio 2017 set the DOCKER_BUILD_SOURCE environment variable
  • One Solution collect form web for “How can I connect to JMX through Kubernetes managed Docker containers?”

    The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.

    Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.