1 2 Previous Next 24 Replies Latest reply: Oct 19, 2010 5:09 PM by 800745 RSS

    Local Port Exhaustion

    800745
      Ref:
      Re: Remote Registy & Logical Port Exhaustion with multiple JVMs

      Hi,

      I am executing an RMI program with Threads implemented. and my program just stops executing without any result after few execution cycles (Each Execution cycle is independent from others).

      My Set Up:

      Server : on single machine starting 10 Threads caling the Clients

      Clients : 10 of them also Servers, running on 5 Machines with 5 Registries

      Slaves : 30 of them 3 report to a client on top, Will get registered with the respective client for Call Backs.

      I need serious help... If You need the whole code I am willing to send it for clarifications...

      I am always getting stuck at this stage.

      .... Please help me to get out of this.....
        • 1. Re: Local Port Exhaustion
          EJP
          This explains precisely nothing. 'Getting stuck' doesn't describe any error condition I am aware of.
          • 2. Re: Local Port Exhaustion
            800745
            I think whether it is Linux or Windows whenever a concurrent activity is started it gets hold of a socket and it scales until it reaches the MAX number of ports configured...

            The problem is that I am not getting any error at any time... the program just freezes.

            Experiment results :

            1. With 1 Slave per Client running concurrently the program ran for 1000+ cycles and freezes

            2. With 3 Slaves per Client ... the program ran for 297 cycles and freezes.

            I assume the problem is as stated below..........

            The Kernel, OOM, LOWMEM, and You
            We first tested our code on a local Linux box that had Ubuntu 64-bit with 6GB of RAM, connecting with several Ubuntu VMs per client using bridged network adapters so we could ramp up our connections. We’d fire up the server and run our clients locally to see just how many connections we could hit. We noticed that we could hit 512,000 with our Java server not even breaking a sweat.

            The next step was to test on EC2. We first wanted to see what sort of numbers we could get on “Small” instances, which are 1.7GB 32-bit VMs. We also had to fire up a number of other EC2 instances to act as clients.

            We watched the numbers go up and up without a hitch until, seemingly randomly, the Java server fell over. It didn’t print any exceptions or die gracefully—it was killed.

            We tried the same process again to see if we could replicate the behavior. Killed again.

            Grepping through syslog, we found this line:

            Out of Memory: Killed process 2178 java

            The OOM-killer killed the Java process. Having watched the free RAM closely, this was odd because we had at least 500MB free at the time of the kill.

            +.............. Cont+



            Ref : http://blog.urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/.

            My Question is is there a way to release the ports that Java program is holding on to once I have completed a cycle?
            • 3. Re: Local Port Exhaustion
              796440
              797742 wrote:
              My Question is is there a way to release the ports that Java program is holding on to once I have completed a cycle?
              Going only by the dearth of information presented, and not being a network guru like EJP, I can only guess that maybe "close each socket when you're done using it" might come into play at some point. And maybe "don't open a new connection for messages that could better be sent over an existing one."
              • 4. Re: Local Port Exhaustion
                800745
                I can only guess that maybe "close each socket when you're done using it" might come into play at some point.

                Any Idea As to How I could use this?


                And maybe "don't open a new connection for messages that could better be sent over an existing one."

                I don't open connections deliberately but I wonder if JRE is doing it? How to avoid this ?
                • 5. Re: Local Port Exhaustion
                  796440
                  797742 wrote:
                  I can only guess that maybe "close each socket when you're done using it" might come into play at some point.

                  Any Idea As to How I could use this?
                  I don't know how to be any more explicit than that. What part don't you understand? Do you know how to close a socket? Do you know when you're done using a given socket? Do you know about finally blocks?
                  And maybe "don't open a new connection for messages that could better be sent over an existing one."

                  I don't open connections deliberately but I wonder if JRE is doing it? How to avoid this ?
                  Whatever is being done, it's being caused by your code (or, if your code is doing nothing to prompt it, then by a horrible bug in whatever library you're using). Since I have no idea what your code is doing, it's impossible to say what the actual cause of the problem is. (Do NOT provide your whole code. If you're going to post code, provide an [url http://sscce.org]SSCCE that demonstrates the problem. Make sure to use code tags.)
                  • 6. Re: Local Port Exhaustion
                    800745
                    jverd wrote:
                    Do you know how to close a socket?
                    NO
                    jverd wrote:
                    Do you know when you're done using a given socket?
                    YES
                    jverd wrote:
                    Do you know about finally blocks?
                    YES

                    Here is a sequence of my Program process
                    <START UP>
                    1. RMI registry is up on locla host
                    2. "Class" Server is up and exports an Object and binds it in localhost registry
                    3. "Class" Client will look up for the Server Remote Object and pass their loclahost address to Server
                    4. Client is up and will export 10 Objects and binds it in the local host registry (They also act as servers)
                    5. "Class" Slaves will look up the "Server Object" and get the respective Clients IP
                    6. Slave will export 30 objects (3 Slaves per Client)
                    7. The Slave instance will look up their respective Clients through the 'LocateRegistry.getRegistry(o_RegistryReference)' and will be register for Call-Backs
                    8. Once respective slaves have been reported to Clients, Each Client will report to the Server

                    <EXECUTION>
                    1. Then Server will initiate tasks concurrently (via threads) to Clients
                    2. Clients will then intern command its 3 slaves to do the job
                    3. each slave will do the job and report results to its clients
                    4. once Clients have received responses from its 3 slaves that client will do the calculation and report the results back to Server
                    5. Server will count until all 10 clients have reported and will give final result.
                    ** Once final result is completed server will restart the process from 1.... and the rest continues until termination cycle number is met.
                    ** * Each execution cycle is independent of the other.
                    • 7. Re: Local Port Exhaustion
                      EJP
                      2. "Class" Server is up and exports an Object and binds it in localhost registry
                      Does this class use an RMIClientSocketFactory? Do any of your remote objects do that?
                      3. "Class" Client will look up for the Server Remote Object and pass their loclahost address to Server
                      Why? It should just pass its own remote reference if it's a callback, otherwise just call the server with any RMI method: the server can get the client's IP address via RemoteServer.getClientHost().
                      4. Client is up and will export 10 Objects and binds it in the local host registry (They also act as servers)
                      They don't act any other way. They are servers.
                      5. "Class" Slaves will look up the "Server Object" and get the respective Clients IP
                      Why? Why not just get the client's remote references directly? e.g. from the original server?
                      6. Slave will export 30 objects (3 Slaves per Client)
                      7. The Slave instance will look up their respective Clients through the 'LocateRegistry.getRegistry(o_RegistryReference)' and will be register for Call-Backs
                      Why? See above.
                      1. Then Server will initiate tasks concurrently (via threads) to Clients
                      2. Clients will then intern command its 3 slaves to do the job
                      Why? Why can't the server talk directly to the slaves? What's the middle tier for?
                      • 8. Re: Local Port Exhaustion
                        800745
                        EJP, Thank you for Excellent Answers... CLEAR.
                        EJP wrote:
                        Does this class use an RMIClientSocketFactory? Do any of your remote objects do that?
                        No It does not
                        EJP wrote:
                        Why? It should just pass its own remote reference if it's a callback, otherwise just call the server with any RMI method: the server can get the client's IP address via RemoteServer.getClientHost().
                        So you mean, If a client exports its instance and passes the reference to Serve's Remote Method Register(Client Client_Instance), Server should be able to get the IP of that Client Instance?
                        OR
                        Should I have a method as below in Client to sent the IP to Server ....
                        +public String getClientHost() throws ServerNotActiveException {+
                        return RemoteServer.getClientHost();
                        +}+
                        EJP wrote:
                        Why? Why not just get the client's remote references directly? e.g. from the original server?
                        The Clients IP is registered with the Server, so the slave can find the intermediate level servers IP from the place that they have registered them (In the Original Server)
                        EJP wrote:
                        Why? Why can't the server talk directly to the slaves? What's the middle tier for?
                        This is my Architecture

                        1 Server : who is acting as the commander and decision maker
                        10 Clients : Who will do intermediate work and and Report to the Server above
                        30 Slaves : Who does the only task assigned by the respective client and Report back to the Client

                        Foe instance : Client 1 will have subordinates Slave (1,2 & 3) and Client 2 will have subordinates Slave (4,5 & 6)......

                        Edited by: JLearner on Oct 5, 2010 3:57 PM
                        • 9. Re: Local Port Exhaustion
                          EJP
                          So you mean, If a client exports its instance and passes the reference to Serve's Remote Method Register(Client Client_Instance), Server should be able to get the IP of that Client Instance?
                          I didn't just 'mean' that. I said that.
                          Should I have a method as below in Client to sent the IP to Server ....
                          No you shouldn't. Java provides the method. I told you its name and the class it is in. The server calls it. What exactly is the problem here? Did you look it up in the Javadoc?
                          Why? Why not just get the client's remote references directly? e.g. from the original server?
                          The Clients IP is registered with the Server, so the slave can find the intermediate level servers IP from the place that they have registered them (In the Original Server)
                          You're just repeating yourself here. I asked you why? You don't need IP addresses. What you need is remote references to clients. And the server has all those. You're building in a lot of unnecessary stuff here.
                          • 10. Re: Local Port Exhaustion
                            800745
                            EJP wrote:
                            Did you look it up in the Javadoc?
                            YES : So what I will do is RemoteServer o_Remote = new RemoteServer(ClientInstance)

                            EJP wrote:
                            What you need is remote references to clients. And the server has all those.
                            So the Slaves will just get the REFERENCE of their superiors from the Original server.

                            Thanks you this answers my questions....
                            • 11. Re: Local Port Exhaustion
                              EJP
                              Did you look it up in the Javadoc?
                              YES:
                              NO, on this evidence.
                              So what I will do is RemoteServer o_Remote = new RemoteServer(ClientInstance)
                              No, that isn't what you will do. You can't do that. It's a static method. You don't need an instance of RemoteServer to call it, and you can't construct one that way anyway.

                              If that's what you call reading Javadoc, you need to try harder. Much harder. I'm serious. I gave you this answer yesterday. You've wasted 15 hours by not reading properly.

                              And if you had read my response properly, you would have seen that you don't need this method at all.
                              • 12. Re: Local Port Exhaustion
                                800745
                                I will try and give you feedback.

                                tks
                                • 13. Re: Local Port Exhaustion
                                  EJP
                                  Re the bind exception, you are running out of outbound ports.

                                  This usually only happens in RMI if you are using RMIClientSocketFactories without properly defined equals() methods.

                                  Or else you have something else using up all the outbound ports other than RMI connections.

                                  What does netstat -an show?
                                  • 14. Re: Local Port Exhaustion
                                    800745
                                    It is holding many ports

                                    Eg;....

                                    TCP 136.186.6.157:3942 136.186.6.157:2659 TIME_WAIT
                                    TCP 136.186.6.157:3943 136.186.6.157:2658 TIME_WAIT
                                    TCP 136.186.6.157:3944 136.186.6.157:2647 ESTABLISHED
                                    TCP 136.186.6.157:3945 136.186.6.157:2657 ESTABLISHED
                                    TCP 136.186.6.157:3946 136.186.6.157:2649 ESTABLISHED
                                    TCP 136.186.6.157:3947 136.186.6.157:2668 ESTABLISHED
                                    TCP 136.186.6.157:3948 136.186.6.157:2658 ESTABLISHED
                                    TCP 136.186.6.157:3949 136.186.6.157:2668 TIME_WAIT
                                    TCP 136.186.6.157:3950 136.186.6.157:2658 TIME_WAIT
                                    TCP 136.186.6.157:3951 136.186.6.157:2657 TIME_WAIT
                                    TCP 136.186.6.157:3952 136.186.6.157:2659 ESTABLISHED
                                    TCP 136.186.6.157:3953 136.186.6.157:2659 TIME_WAIT
                                    TCP 136.186.6.157:3954 136.186.6.157:2647 TIME_WAIT
                                    TCP 136.186.6.157:3955 136.186.6.157:2649 TIME_WAIT
                                    TCP 136.186.6.157:3956 136.186.6.157:2657 ESTABLISHED
                                    TCP 136.186.6.157:3957 136.186.6.157:2668 ESTABLISHED
                                    TCP 136.186.6.157:3958 136.186.6.157:2636 ESTABLISHED
                                    TCP 136.186.6.157:3959 136.186.6.157:2661 ESTABLISHED
                                    TCP 136.186.6.157:3960 136.186.6.157:2643 ESTABLISHED
                                    TCP 136.186.6.157:3961 136.186.6.157:2660 ESTABLISHED
                                    TCP 136.186.6.157:3962 136.186.6.157:2636 ESTABLISHED
                                    TCP 136.186.6.157:3963 136.186.6.157:2660 TIME_WAIT
                                    TCP 136.186.6.157:3964 136.186.6.157:2657 TIME_WAIT
                                    TCP 136.186.6.157:3965 136.186.6.157:2661 TIME_WAIT
                                    TCP 136.186.6.157:3966 136.186.6.157:2668 TIME_WAIT
                                    TCP 136.186.6.157:3967 136.186.6.157:2643 TIME_WAIT
                                    TCP 136.186.6.157:3968 136.186.6.157:2649 ESTABLISHED
                                    TCP 136.186.6.157:3969 136.186.6.157:2647 ESTABLISHED
                                    TCP 136.186.6.157:3970 136.186.6.157:2636 ESTABLISHED
                                    TCP 136.186.6.157:3971 136.186.6.157:2647 TIME_WAIT
                                    TCP 136.186.6.157:3972 136.186.6.157:2647 ESTABLISHED
                                    TCP 136.186.6.157:3973 136.186.6.157:2647 TIME_WAIT
                                    TCP 136.186.6.157:3974 136.186.6.157:2636 TIME_WAIT
                                    TCP 136.186.6.157:3975 136.186.6.157:2649 TIME_WAIT
                                    TCP 136.186.6.157:3976 136.186.6.157:2668 ESTABLISHED
                                    TCP 136.186.6.157:3977 136.186.6.157:2649 ESTABLISHED
                                    TCP 136.186.6.157:3978 136.186.6.157:2625 ESTABLISHED
                                    TCP 136.186.6.157:3979 136.186.6.157:2649 ESTABLISHED
                                    TCP 136.186.6.157:3980 136.186.6.157:2661 ESTABLISHED
                                    TCP 136.186.6.157:3981 136.186.6.157:2661 ESTABLISHED
                                    TCP 136.186.6.157:3982 136.186.6.157:2643 ESTABLISHED
                                    :
                                    :
                                    Cont.....

                                    Keep increasing.

                                    *** I wonder if there is an impact of using following statements every time I pass a message to the upper layer or lower...

                                    For example... once a slave has completed the job and needs to pass the results back to the Clients, Slaves will do following

                                    Registry registry = LocateRegistry.getRegistry(o_ClientRef);
                                    Client_Interface client = (Client_Interface) registry.lookup("Client_Obj" + i_ClientID);
                                    client.reportResults(i_SlaveID, i_Cycle, d_ShortestDistance);

                                    as Slaves are as many as 50.. there will be 50 lookup's every time and is this necessary?

                                    Can I do this once and use "client" instance to report the result back to clients?
                                    1 2 Previous Next