The daily bug hunt

Yesterday we got a support request that a certain route request fails and just says ‘route not found’. We wanted to quickly reply that the open source routing engine has an option where you can reduce to a sufficient minimum, but as it turned out he meant the hosted GraphHopper Routing API where we already use a properly configured engine. So something was wrong with it. And in the geographical area we detected nothing strange with the OpenStreetMap data, (although for Google Maps we found one ;)) and so it had to be something in the routing engine.

Grabbing the area via the export button at openstreetmap.org was easily done, but we could not reproduce this on my local machine. And so we would need to somehow use the remote data, which is a bit too big to fit on my local computer (~50GB).

How do you remotely debug a Java application in a safe way?

Let me explain this for NetBeans but it should work similar for any other Java IDE. Add the following options to your remote Java application:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8888 ...

You can find out more about these settings via

java -agentlib:jdwp=help

As a side note: there is also an old settings “-Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n” but for that we got an error: “OutOfMemoryError: Java heap space” which did not go away even when trippling the -Xmx setting (maximum RAM for the JVM). After switching to a different off-heap graph storage method (MMAP_STORE_SYNC) we got it working with the old method.

To make the JVM wait for the agent to connect before starting the application just specify suspend=y.

Now, how can you open the port 8888 without making it public to anyone? Instead just 8888 use localhost:8888, then the socket is not publicly accessible and so you can safely access it through a SSH tunnel:

ssh -L 8888:localhost:8888 user@server

Then click ‘Debug’ in the Netbeans menu, then ‘Attach Debugger’ and press ‘okay’. Keep localhost and 8888 or if you do not have a SSH tunnel use the remote server IP. It should then open a new output window telling you that you are connected. In our case we still had a problem:

Not able to submit breakpoint LineBreakpoint GraphHopper.java. Reason: Breakpoint's source file does not belong into the preferred source root '.../.m2/repository/com/graphhopper/graphhopper/0.7.0/graphhopper-0.7.0-sources.jar'

This is easy to fix. Go to Window->Debugging->Sources and deselect the entry for graphhopper 0.7.0 as the open project with 0.8-SNAPSHOT itself is sufficient in my case.

Finally you were able to remotely debug into the JVM

Nice! But we were still not able to figure out what went wrong ;(

The reason was that this bug was not related to runtime (i.e. finding the route) but it was related to how we post-process the OpenStreetMap network to make it less error-prone. Read more about the details here and if you liked this maybe you would like to contribute 🙂 ? Have a look into our quickstart and sign-up to our forum.

We hope you enjoyed reading this bug hunt and you can apply the findings to your own debug sessions!