Btrace is a wonderful tool for debugging complex application problems. I have used it recently to debug a strange finalizer problem in an application running on Glassfish V3. Since getting it to work with v3 wasn't very straightforward, I will share my findings here in case someone else is trying to do the same...
BTrace works as a JVM Agent, which means it gets the chance to modify classes before they are loaded by the platform. My initial attempts to attach it to a running Glassfish v3 domain failed with the error "connection refused".
The way the btrace command-line tool works is this:
First it attaches the agent to the running JVM process
The agent listens on tcp port 2020 for tracing scripts to run
The command line tool then connects to the agent and submits the tracing script
The connection refused message indicates the tool is not being able to connect to the agent. Why is that?
If you look at the command-line of a running Glassfish V3 server you will notice that it is started with a jvm option similar to the following one::
It seems like Glassfish comes with its own version of Btrace for internal use only.The noServer=true option disables the agent server feature, making the it unreacheable by external tools. The btrace command line tool tries to attach its own agent to the JVM but only one agent can be attached at a time, and since the existing one has no server you end up with the connection refued message.
I tried to find a way of changing this startup option but I couldn't find any documentation related to it or even any BTrace-related option in domain.xml.
Luckly it turns out that if you disable the monitoring feature in Glassfish it will start without its internal Btrace agent. You can do that from the admin console under the "Monitoring" set of options or from the command line like this: