1 2 Previous Next 15 Replies Latest reply: May 19, 2011 10:04 AM by 796440 RSS

    Need advice

    800271
      Dear All,
      Currently I have a socket listener where it accept and create a new thread and in it I will write to the database it self. The problem now I am planning to do some heavy processing e.g checking the poi location, geo fence etc of the data receive from the listener. Do you think I should create a new separate thread for its processing or just write the raw data into the system and later do the processing separately?
        • 1. Re: Need advice
          802316
          You could do all that but without more details I would implement the simplest solution first and look at optimising it later.
          • 2. Re: Need advice
            796440
            Impossible to say without more details. It's not even clear what you're doing. All I can tell is that you receive some data over a socket, do some processing, and insert something into a DB. It's not clear at all what the sequence or interdependence of these steps is. I also have no idea what you mean by "checking the poi location" or "geo fence".

            However, as a general rule, I would consider a good design to be to have one thread pool for pulling data off the wire and putting it into a queue, and another thread pool for pulling it off the queue, doing whatever pre-processing is necessary, and sticking it into the DB, and then one or more additional thread pools for whatever processing tasks are necessary to process stuff after it's been inserted into the DB.

            Note that I most definitely would NOT start a new thread for each received message. Use something like java.util.concurrent.ThreadPoolExecutor.

            Edited by: jverd on May 17, 2011 11:50 AM
            • 3. Re: Need advice
              854422
              Just food for thought. A web/http server listens on a socket with its thread pool. If you subclassed Servlet instead of HttpServlet & defined Tomcat's or Jetty's port to be the one you want, how would the sender even know the listening side changed. You might get a more robust system, & quicker by not starting so low. If there is not some detail not provided that rules this out.
              • 4. Re: Need advice
                800271
                Dear Jverd,
                Below is part of my codes. I have remove the insertion into the db as it is a lot.So here only one thread is created. How to create further thread for the preprocessing of the data receive from the socket? What will be your idea?
                public class commServer {
                  
                  
                   public static void main(String[] args) {
                
                      try {
                                   final ServerSocket serverSocketConn = new ServerSocket(9000);
                                    
                                    while (true) 
                                         {
                                              try 
                                              {
                                                     Socket socketConn1 = serverSocketConn.accept();
                                                new Thread(new ConnectionHandler(socketConn1)).start();                           
                                              }
                                              catch(Exception e)
                                              {
                                                   System.out.println("MyError:Socket Accepting has been caught in main loop."+e.toString());
                                                  e.printStackTrace(System.out);
                                              }
                                         }
                               
                      } 
                      catch (Exception e) 
                      {
                         System.out.println("MyError:Socket Conn has been caught in main loop."+e.toString());
                         e.printStackTrace(System.out);
                         //System.exit(0); 
                      }
                   }
                   
                }
                  class ConnectionHandler implements Runnable {
                    private Socket receivedSocketConn1;
                   
                
                    ConnectionHandler(Socket receivedSocketConn1) {
                      this.receivedSocketConn1=receivedSocketConn1;
                    }
                
                 
                   //@Override
                   public void run() { 
                      
                      Connection dbconn = null;
                      BufferedWriter w = null;
                      BufferedReader r = null;
                      
                
                      try {
                      
                         PrintStream out = System.out; 
                            BufferedWriter fout = null;
                         w =  new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream()));
                         r = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream()));
                         
                        
                         
                         
                         int m = 0, count=0;
                         String line="";
                         String n="";
                         //w.write("$PA\n");
                         //w.flush();
                         while ((m=r.read()) != -1) 
                         {
                                  
                                  ///**DB processing here.
                              
                                  n="";
                                 
                         }
                      } 
                      catch (IOException ex)  
                      { 
                           System.out.println("MyError:IOException has been caught in in the main first try");
                           ex.printStackTrace(System.out);
                      }      
                      finally
                      {
                        try 
                            {
                        
                             if ( w != null ) 
                             {
                                    w.close();
                             }
                             else 
                             {
                                  System.out.println("MyError:w is null in finally close");
                             }
                        }
                        catch(IOException ex){
                           System.out.println("MyError:IOException has been caught in w in finally close");
                           ex.printStackTrace(System.out);
                        }
                        
                      }
                   }
                }
                • 5. Re: Need advice
                  796440
                  I'm not going to read your code and address it in particular. I'm just talking general case here.

                  First, my comment about a thread for preprocessing only applies if there is preprocessing that you have to do before inserting, and it's relatively expensive in terms of CPU or needing lots of other I/O. If you just get a comma-separated String and just have to split it up and insert it, or something simple like that, there's probably no point in a separate preprocess thread pool. I only mentioned it because it wasn't clear to me what your situation is. Another case for the preprocessing model might be if a lot of data is arriving in more-or-less constant stream, you could just enqueue it, and then have a separate thread that pulls chunks off the queue and inserts them in batchdes, to reduce the JDBC overhead.

                  In either case, the basic model is this:

                  One thread reads from the socket and immediately inserts the data into a queue, such as java.util.concurrent.ArrayBlockingQueue. There's a separate thread, or perhaps a thread pool--managed by java.util.concurrent.ThreadPoolExecutor--that reads from the queue, does any preprcoessing if you require it, and inserts into the DB.

                  Note that you only create one reader thread for a single socket, and you only create it once per socket, not once for each message. Similarly, you don't create a new thread every time you read a message off the queue. Just create one single thread, or a thread pool, once at the beginning.

                  This is the same thing I said in my first post. It's not clear to me what part you're not understanding.

                  If you're looking for me to write the code for you, that's not going to happen.

                  Edited by: jverd on May 18, 2011 8:34 AM
                  • 6. Re: Need advice
                    800271
                    Dear Jverd,
                    No the thing is the previous code is the current code so is just a basic thing split using come and then insert into the db. The preprocessing here is that based on the lat and long I am using then look back the poi table, geo fence table and see which poi and geo fence is it. In addition I am also going to process the alerts. All this have been implemented here. So what is your idea do it in the java or insert into the db first and then process from the db ?
                    • 7. Re: Need advice
                      796440
                      797268 wrote:
                      Dear Jverd,
                      No the thing is the previous code is the current code so is just a basic thing split using come and then insert into the db. The preprocessing here is that based on the lat and long I am using then look back the poi table, geo fence table and see which poi and geo fence is it. In addition I am also going to process the alerts. All this have been implemented here. So what is your idea do it in the java or insert into the db first and then process from the db ?
                      I have no idea what you're saying. All I can tell is that you still haven't provided any more details that I could use to offer a different suggestion that what I already have. I don't know what your actual problem is, or what you didn't understand about my previous suggestion.

                      I'm sorry, but I don't think I can help you. Good luck.
                      • 8. Re: Need advice
                        800271
                        Dear Jverd,
                        Ok sorry let me re-word it. Currently upon receiving the data from the wire one thread is created. Then in this thread I just insert the data into the database table. Now before inserting into the database table I want to process the data then only enter that data. Do you think from my current thread I should create another thread? That is where I am trying to adapt as you suggested "pulling data off the wire and putting it into a queue, and another thread pool for pulling it off the queue, doing whatever pre-processing is necessary, and sticking it into the DB, and then one or more additional thread pools for whatever processing tasks are necessary to process stuff after it's been inserted into the DB."
                        • 9. Re: Need advice
                          DrClap
                          797268 wrote:
                          Now before inserting into the database table I want to process the data then only enter that data. Do you think from my current thread I should create another thread?
                          If your requirement is "I want to do A, then B" then putting A and B into separate threads is absolutely the wrong thing to do. This is just basic concurrency design: if you have to do two things in sequence then you can't do them concurrently.
                          • 10. Re: Need advice
                            796440
                            797268 wrote:
                            Dear Jverd,
                            Ok sorry let me re-word it. Currently upon receiving the data from the wire one thread is created.
                            Don't do that. Put it in a queue for a thread pool to handle. I don't know how to make it any clearer.
                            Then in this thread I just insert the data into the database table. Now before inserting into the database table I want to process the data then only enter that data.
                            First get the producer/consumer model with the thread pool working. Then change the consumer to do your additional processing before inserting.
                            Do you think from my current thread I should create another thread?
                            No.

                            Do as I suggested above. It's possible that after that you might want to add another queue, where the consumers of the first queue do the preprocessing, then put it into the second queue, and the consumers of the second queue do the DB inserts, but I can't comment on that without knowing more details about your situation than I'm interested in knowing.
                            • 11. Re: Need advice
                              796440
                              DrClap wrote:
                              797268 wrote:
                              Now before inserting into the database table I want to process the data then only enter that data. Do you think from my current thread I should create another thread?
                              If your requirement is "I want to do A, then B" then putting A and B into separate threads is absolutely the wrong thing to do. This is just basic concurrency design: if you have to do two things in sequence then you can't do them concurrently.
                              @OP: DrClap makes a good point that I had ignored. I was giving you a better thread management model, but it might be that you don't need multiple threads at all. A single thread that just reads, processes, and inserts is the simplest approach.

                              Why do you want multiple threads? How exactly do you think that will help you?
                              • 12. Re: Need advice
                                800271
                                Dear Jverd & Dr. Clap,
                                I have to agree I am quite new into this thread management. Actually due to my problem of heavy db processing that is why I thought first one thread capture then next have another thread to process and enter into db. I was thinking this will help not to lock the overall system because of heay db processes. I might be wrong please correct me.
                                • 13. Re: Need advice
                                  796440
                                  Multiple threads might help, but it's impossible to say without more details.

                                  For instance, if you have one or more threads doing the preprocessing, and one or more other threads doing the DB inserts, then you might get some benefit, as one DB insert can be happening at the same time as another preprocess. However, if both preprocess and insert are already fast, or if one of them takes 10 or more times as long as the other, then you won't see much benefit.

                                  If your preprocess is very CPU-intensive, and you have multiple CPUs or cores, then dedicating multiple threads to the preprocessing might help.
                                  • 14. Re: Need advice
                                    800271
                                    Dear All,
                                    My current situation is simple the data received via the java listener is gps data e.g latitude,logittude, date time, alerts etc. So in order not to burden the java listener we immediately store the data after receiving. So if I start to process the point of interest location, geo fence etc I will need to do a lot of select query from the db and calculation etc. So what is the best suggestion to handle this heavy burden. I am worried I might loose some of the transaction because the queue of data waiting to come in may build up. I might be wrong thinking multi thread could help.
                                    1 2 Previous Next