How can I connect to JMX through Kubernetes managed Docker containers?

I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error:

java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
     java.net.ConnectException: Connection timed out
     ...

Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:

  • docker hornetQ standalone … jnp issue
  • docker network port binding
  • Docker for Windows: ASP.NET site unable to read environment variables defined in Dockerfile
  • grunt-cli doesn't work with docker volume mounted, fine without
  • How to check if docker machine exists programmatically?
  • How can i create a folder inside the shared volume with the correct permission in docker?
  • jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi 
    

    I know that JMX opens a random high port, and I have tried to resolve that by including a custom @Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran

    netstat -tulpn
    

    to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.

    As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:

    -Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0
    

    The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.

    I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.

    Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.

    Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?

    EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?

  • How do I add a container user to a user group in the Docker host?
  • How to use ebtables inside Docker?
  • Run command as --privileged in Dockerfile
  • Can't connect to docker published port for tftp
  • Docker, and small production server infrastructure advices
  • Filesystems content of docker container on host
  • One Solution collect form web for “How can I connect to JMX through Kubernetes managed Docker containers?”

    The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.

    Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.