This discussion is archived
11 Replies Latest reply: Jan 25, 2013 1:21 AM by gimbal2 RSS

Hot Reload of Classes

801508 Newbie
Currently Being Moderated
Hello,

I have a networked application that consists of 1 server and multiple telnet connections. I would like to be able to do a Hot Reboot of the binaries. I do not care about the state of the program because it will be reset. My main goal is to get the new binaries without disconnecting all of the telnet connections. So there will be an interruption on the program and the state would be completely reset/reloaded - but no loss of connectivity.

I've done a lot of research about this and played around with a few different libraries. But I figured I'd make a post to see if I missed something.

Here is what I've looked up/played with:
1. I looked at JRebel and Javeleon. JRebel has limitations. I've actually used Javeleon - it works OK. I don't like having to have a Java Agent and the fact I need a licensee that costs monthly (there are other packages), turns me off. This project is just a hobby.
2. I have done a bit of research on custom class loaders. I've played with a few code snippets. Though, one thing I didn't like about it was I lost the ability to use the "new" operator to create objects. I would have to use reflection. Is there a way to write a custom class loader such that I don't have to make large code changes elsewhere in my program? As mentioned above, I'm completely fine with the state being lost and reloaded.
3. This server is intended to run on Linux, so I don't need it to be portable. As such, I could possibility use JNI to be able to call C execl and reload the binaries and transfer file descriptors. I've seen this done for other telnet based servers (text based MUDs). Though, I'm not sure what the effect of execl would be on a JVM/etc.
4. Much like 4, but instead us JNA in order to do this from within the Java code. I'm not sure if this is possible - I haven't used JNA before.
5. Write a proxy server, that sits in front of the main server. This proxy server will contain a list of connections (telnet) and it will monitor the status of the main server. If the main server will want to reload the binaries, the proxy server would shut it down, restart it, and then reconnect to the new server. For anything else, it'd just simply pass along the input from the telnet connections to the main server.
6. Something else?

I think my best option would be 2, but I haven't really been able to make much headway with classloaders initially. It seems everything is a huge project (and could lead no where so I don't want to waste time) except 2.

Does anyone else have suggestions/thoughts/comments?

Thanks for your help.

Edited by: kilosi on Dec 6, 2012 12:19 PM
A bit more information.
  • 1. Re: Hot Reload of Classes
    Kayaman Guru
    Currently Being Moderated
    kilosi wrote:
    I think my best option would be 2, but I haven't really been able to make much headway with classloaders initially. It seems everything is a huge project (and could lead no where so I don't want to waste time) except 2.

    Does anyone else have suggestions/thoughts/comments?
    I would also consider approach number 5, but that's just 'cos I'm all enterprisey and stuff. Besides, having a proxy server might be advantageous in other ways as well later on.

    Regarding solution 2, you don't need to stop using "new" even if you use a custom classloader. Loading a single "root" class dynamically would be enough, since all the classes loaded from there will be loaded using the same classloader.
    You'll need to make some modifications in your code naturally, but how extensive I can't really say.
  • 2. Re: Hot Reload of Classes
    801508 Newbie
    Currently Being Moderated
    Kayaman wrote:

    I would also consider approach number 5, but that's just 'cos I'm all enterprisey and stuff. Besides, having a proxy server might be advantageous in other ways as well later on.
    It would be certainly useful for other things. I would really like to have the program be able to monitor itself in some way. This proxy server could do that - like send me an email if the main server crashed, or something similar. Would be cool!

    Now, my concern about this is - besides the amount of work in developing it (which I don't mind once I commit), the added performance aspect. Since any data/input would have to go through a new layer, which would consist of something very minimal such as a string check (to check the input to see if it was a hotreboot command) and then it'd be pass to the server through a socket. Then, anything from the main server would just have to be ferried back through the proxy server and then to the client. I'd like to think it would be very low, and beyond noticeable. If I had to guess, the majority of connections I'd have would not go higher than 50, and most likely in the 10-20 range.

    Based on your reply, it seems something like this is fairly common in enterprise?

    Kayaman wrote:
    Regarding solution 2, you don't need to stop using "new" even if you use a custom classloader. Loading a single "root" class dynamically would be enough, since all the classes loaded from there will be loaded using the same classloader.
    You'll need to make some modifications in your code naturally, but how extensive I can't really say.
    When you refer to a root class, do you mean "Object"? Or a root class of the code I want to reload? Do you happen to have a web link handy, or some good terms I can google with?

    Thanks for your help!
  • 3. Re: Hot Reload of Classes
    800268 Expert
    Currently Being Moderated
    You'll want some sort of plugin system.
    Your main program maintains the connection and then delegate the handling of the input/output to the plugin. Then you reload or load a different (newer) plugin on the reboot command.

    See OSGI or http://jpf.sourceforge.net/ (there are probably more or you can build your own with custom class loaders as suggested).
  • 4. Re: Hot Reload of Classes
    Kayaman Guru
    Currently Being Moderated
    kilosi wrote:
    >
    It would be certainly useful for other things. I would really like to have the program be able to monitor itself in some way. This proxy server could do that - like send me an email if the main server crashed, or something similar. Would be cool!
    Exactly.
    Now, my concern about this is - besides the amount of work in developing it (which I don't mind once I commit), the added performance aspect. Since any data/input would have to go through a new layer, which would consist of something very minimal such as a string check (to check the input to see if it was a hotreboot command) and then it'd be pass to the server through a socket. Then, anything from the main server would just have to be ferried back through the proxy server and then to the client. I'd like to think it would be very low, and beyond noticeable. If I had to guess, the majority of connections I'd have would not go higher than 50, and most likely in the 10-20 range.
    The performance aspect with a proxy server is indeed minimal, so that shouldn't be an issue.
    Based on your reply, it seems something like this is fairly common in enterprise?
    It is, with all sorts of load balancers and clusters involved.
    Kayaman wrote:
    When you refer to a root class, do you mean "Object"? Or a root class of the code I want to reload?
    The code you want to reload. Let's say you have a Main class you run, then in that class you load dynamically your MyServer class. Inside MyServer class you can use new to instantiate classes as per usual.
    However you'll need to make sure that the system classloader won't load the MyServer class, so you need to keep it out of the classpath (if my memory serves me correctly).
    Do you happen to have a web link handy, or some good terms I can google with?
    Google for "java custom class loader" and perhaps class reloading.
    Thanks for your help!
    No problem, also check out OSGi, I forgot that completely. It has module reloading built-in.
  • 5. Re: Hot Reload of Classes
    801508 Newbie
    Currently Being Moderated
    HI,

    I was in the process of moving so took me awhile to respond :)

    Thanks for the replies - I'll check out OSGi and do a bit of googling around for more class loading information.


    I'll also make a 'final' reply on the thread after I implement my decision.
  • 6. Re: Hot Reload of Classes
    801508 Newbie
    Currently Being Moderated
    Hello!

    I'm currently looking at Equinox framework. I haven't been able to get some other OSGi frameworks to work except this one.
    I'll post up in more detail if I get something working.

    Edited by: kilosi on Jan 22, 2013 6:45 PM
  • 7. Re: Hot Reload of Classes
    801508 Newbie
    Currently Being Moderated
    Hi again,

    This will be my final update to this thread.

    I have managed to successfully write a prototype that does what I want with OSGi. It took a few days but after playing around with it, it quickly came together.

    Basically, this is what I did:
    1. Write two bundles. The first bundle will be very simple - a simple list of IDs and sockets. The second bundle will be the actual server. The dependency for the server(second bundle) is the first bundle.
    2. When the server detects a connection typed "update", it'll save all of the connections and IDs, and place them in the first bundle.
    3. Then the second bundle will update itself with the update call.
    4. Upon reloading, the second bundle will determine if it was updated or not via checking the contents of the list in the first bundle (ie: if the list is empty, then we didn't update). If true, it'll reestablish all of the correct sockets with the login IDs, otherwise, it'll create new ones.
    5. After the second bundle will clear the list out.

    This has worked perfectly, and I'm quite pleased with it. I'm now going to take the time to integrate my real application into this and see how it truly works. I have high hopes.

    Thanks all for suggesting OSGi, I've learned a lot even though at times, it was quite time consuming. However, this framework seems quite powerful.
  • 8. Re: Hot Reload of Classes
    gimbal2 Guru
    Currently Being Moderated
    kilosi wrote:
    Thanks all for suggesting OSGi, I've learned a lot even though at times, it was quite time consuming. However, this framework seems quite powerful.
    Good things take time, congratulations on your success. What references did you use to figure this out, if you don't mind me asking?
  • 9. Re: Hot Reload of Classes
    801508 Newbie
    Currently Being Moderated
    gimbal2 wrote:
    kilosi wrote:
    Thanks all for suggesting OSGi, I've learned a lot even though at times, it was quite time consuming. However, this framework seems quite powerful.
    Good things take time, congratulations on your success. What references did you use to figure this out, if you don't mind me asking?
    It involved A TON of googling to find a whole host of different websites and guides. I ran into a bunch of little problems. At first, I couldn't get Felix or the Knopflerfish to work. I then tried Equinox and that seemed to work great. I should of tried that first because I use Eclipse and Eclipse uses it.

    But some of the good references I found useful were:
    http://www.osgi.org/Technology/HowOSGi (Scroll down to the Equinox section, there were a few good pages there)
    This one was quite helpful:
    http://www.javaworld.com/javaworld/jw-03-2008/jw-03-osgi1.html

    Then a few pages on Stackoverflow, here is one:
    http://stackoverflow.com/questions/1305307/the-activator-bundle-is-invalid

    A few other pages that helped me with some issues:
    http://eclipsesource.com/blogs/2009/06/10/osgi-and-start-levels/
    http://eclipse.1072660.n5.nabble.com/prov-Strange-error-message-org-osgi-framework-BundleException-State-change-in-progress-for-bundle-td30397.html


    I just got done putting in the functionality into my main telnet server. I ran into a bunch of big problems with this - it took around 6 hours to do. The MAIN issue was, and I didn't really know this, was that it seems OSGi won't terminate any threads created by the user. They'll still exist. So...my prototype was a single connection based test. My main server, obviously can handle multiple connections. So, what was happening was after my hot reboot (bundle update) within the program, I would try to connect with a second connection. But it would not connect to the main menu and the logging wouldn't appear on the first connection. This stumped. What I ended up doing was creating a time stamp for server creation and I displayed that in many debug statements. Turns out, the second connection was still trying to be used by the first thread!! Not the second, new one.

    So I then had to find out how to stop this issue: How do I interrupt serverSocket.accept()? After trying a bunch of things and thinking about it, the solution is pretty easy...it was to give it a 50 MS timeout. If the server socket doesn't get a connection, it can check to see if it should terminate after (ie: the bundle shutdown, a flag is set when this happens), then the thread can terminate. The only downside, well two at least is:
    1. Someone could connect during this and screw up their connection. It's very unlikely, since 50 MS is not a lot of time. I may lower this to 10 MS or something.
    2. This deals with the 50 MS issue again, if a bunch of people try to connect, there may be some delay, I suspect, because the server has to service them in at least 50 MS increments. I will benchmark this and see if it really is a problem.

    Hopefully you and others found this insightful! I certainly enjoyed typing hotreboot on the program over and over again though. :D

    Edited by: kilosi on Jan 24, 2013 4:52 PM
  • 10. Re: Hot Reload of Classes
    r035198x Pro
    Currently Being Moderated
    For the class loading hopefully JSR 277 will standardize things.
  • 11. Re: Hot Reload of Classes
    gimbal2 Guru
    Currently Being Moderated
    kilosi wrote:
    Hopefully you and others found this insightful! I certainly enjoyed typing hotreboot on the program over and over again though. :D
    Thanks man, very useful indeed.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points