How can I connect to JMX through Kubernetes managed Docker containers?

I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error:

java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
     java.net.ConnectException: Connection timed out
     ...

Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:

  • docker error FATA[0000] Get http:///var/run/docker.sock/v1.18/images/search?term=node: dial unix /var/run/docker.sock: no such file or directory
  • howto append/modify files in docker container from 'docker run … [command]'
  • How do I run a command on an already existing Docker container?
  • Access db as localhost from docker container
  • Docker postgresql container with external data container
  • Docker: combine docker layers into image
  • jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi 
    

    I know that JMX opens a random high port, and I have tried to resolve that by including a custom @Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran

    netstat -tulpn
    

    to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.

    As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:

    -Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0
    

    The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.

    I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.

    Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.

    Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?

    EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?

  • docker-compose swarm: force containers to run on specific hosts
  • How do I automate two layers of SSH plus a docker exec?
  • How to make Secure Gateway Client to resolve the hostname of the destination's host
  • docker-compose on Windows - Container command 'sh' not found or does not exist
  • How can one Docker container call another Docker container
  • Connecting to postgres from a docker container
  • One Solution collect form web for “How can I connect to JMX through Kubernetes managed Docker containers?”

    The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.

    Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.