1 2 3 Previous Next 31 Replies Latest reply on Sep 5, 2009 10:29 AM by 782681

    threading taking loong time to process

    807580
      Hi Gurus,
      I have a class which reads the file and send the record to separate classes to process that record. so:


      |------ wclass1
      |
      |
      |------ wclass2
      |
      |
      mainclass |------ wclass3
      |
      |
      |------ wclass4
      |
      |
      |------ wclass5


      so at the moment, mainclass calls wclass1.process(String record), wclass2.process(String record),... one after another.

      I tried to make the wclass1,2..4 a Runnable and had a following loop in mainclass to check each wlcassx thread by updating a flag to see if the wclassx.process(String record) finished processing the current record so i can send the next record to process. But my threading didnt work.

      while(true)
      {
      boolean recordProcessed = false;
      for (int i = 0; i < tables.size(); i++)
      {
      recordProcessed = ((Extractor) tables.elementAt(i)).recordProcessed();
      if(!recordProcessed)
      {
      break;
      }
      }
      if(recordProcessed) break;
      }

      Please tell me the best to thread it.

      Thanks a BUNCH
        • 1. Re: threading taking loong time to process
          796440
          1. Google for java concurrency tutorial

          2. "Didn't work" contains no useful information. You'll need to provide an SSCCE (with properly indented code and the code tags, of course) and a precise description of what "didn't work" means.
          • 2. Re: threading taking loong time to process
            807580
            1. i hvae done threading but this seems a little complicated to me.

            2. sorry i shdnt have said, didnt work, i meant it took longer than processing it in sequential way.

            i dont have code anymore since i discard it, but to start again i wanted to get an idea.so plz help.
            thanks
            • 3. Re: threading taking loong time to process
              796440
              javalover2002 wrote:
              2. sorry i shdnt have said, didnt work, i meant it took longer than processing it in sequential way.
              Then either you're using threads incorrectly, or you're making invalid assumptions about what multithreading really means, or your test is bogus.
              i dont have code anymore since i discard it,
              So re-create it.
              but to start again i wanted to get an idea.so plz help.
              Sure, as soon as you show what you're trying to do and what exact problem you're having with it, you'll get help.
              • 4. Re: threading taking loong time to process
                807580
                here is the code, there are lots of connected programs, i pasted related programs only. 1. WorkerThread, which reads a file, and sends records to different extractor programs to process ie subextract1 and subextract2 here. SubExtract1 and SubExtract2 are implemented runnable and extended from Extractor.
                please take a look and see why its taking sooo long time and also hogging my cpu.
                package com.packaging;
                import java.util.*;
                import com.packaging.extract.*;
                import com.common.*;
                public class WorkerThread extends Thread
                {
                    packaging parent;
                    FileUtility logFile;
                    FileUtility readFile;
                    ConfigUtil confUtil;
                    String inputFileName = "";
                    String tn = "";
                    String separator = "\\|";
                    Vector tables = new Vector();
                    Vector tableThreads = new Vector();
                    SubExtract1 subextract1;
                    SubExtract2 subextract2;
                    public WorkerThread(packaging parent, ConfigUtil confUtil, String inputFileName) throws Exception
                    {
                        logFile = parent.logFile;
                        this.confUtil = confUtil;
                        readFile = new FileUtility();
                        this.inputFileName = inputFileName;
                    }
                    public void run()
                    {
                        tn = this.getName();
                        int tid = Integer.parseInt(tn.substring(tn.indexOf("-")+1));
                        try
                        {
                            logFile.writeToFile(tn+" to process file: "+inputFileName,true,false);
                            readFile.getInputFile(inputFileName);
                            subextract1 = new SubExtract1(logFile, confUtil, tid);
                            tables.addElement(subextract1);
                            tableThreads.addElement(new Thread(subextract1));
                            subextract3 = new SubExtract3(logFile, confUtil, tid);
                            tables.addElement(subextract3);
                            tableThreads.addElement(new Thread(subextract3));
                           for(int i = 0; i < tableThreads.size(); i++)
                               ((Thread) tableThreads.elementAt(i)).start();
                            processFile();
                        } catch(Exception e)
                        {
                            e.printStackTrace();
                        } finally
                        {
                            for (int i = 0; i < tables.size(); i++)
                                ((Extractor) tables.elementAt(i)).finish();
                        }
                    }
                    private void processFile() throws Exception
                    {
                       try
                        {
                            readFile.readFile(this);
                        } catch (Exception e)
                        {
                            throw e;
                        }
                    }
                    public String processRecord(String record) throws Exception
                    {
                        try
                        {
                            record = record.substring(0,record.length()-1);
                            Map rcd = new HashMap();
                            rcd.put("TAG",tag);
                            for(int i = 1; i < ra.length; i++)
                                rcd.put(fieldNames[i-1].trim(),ra.trim());
                for (int i = 0; i < tables.size(); i++)
                ((Extractor) tables.elementAt(i)).setRecord(rcd);
                while(true)
                {
                boolean recordProcessed = false;
                for (int i = 0; i < tables.size(); i++)
                {
                recordProcessed = ((Extractor) tables.elementAt(i)).recordProcessed();
                if(!recordProcessed)
                break;
                }
                if(recordProcessed) break;
                }

                if(ra[0].startsWith("END"))
                {
                for (int i = 0; i < tables.size(); i++)
                ((Extractor) tables.elementAt(i)).flush();
                }
                } catch (Exception e)
                {
                throw e;
                }
                return null;
                }
                }
                ------------------------------------------
                package com.packaging.extract;

                public class Extractor
                {
                boolean exit = false;
                Map record = new HashMap();

                public Extractor() throws Exception
                {
                }

                public void initialize() throws Exception
                {
                try
                {
                procesing...
                } catch (Exception e)
                {
                throw e;
                }
                }

                public void setRecord(Map record)
                {
                this.record = new HashMap(record);
                }

                public boolean recordProcessed()
                {
                return record.isEmpty();
                }

                public void process(Map rcd) throws Exception
                {
                try
                {
                record = new HashMap(rcd);
                if(record.isEmpty()) return;
                // logFile.log(FileUtility.DEBUG,tid+"New Record: "+record,true,true,true);
                recordCounter++;
                //processging....
                scenarioFound = true;
                } catch(Exception e)
                {
                throw e;
                }
                }

                public void getTags() throws Exception
                {
                String fieldDetail = "";
                String[] fieldExtensions, conditions, conditionAnd;
                try
                {
                for (int i = 0; i < fields.length; i++)
                {
                fieldDetail = config.getProperty(ctable + "." + fields[i]);
                // processing...........
                }
                } catch (Exception e)
                {
                logFile.log(FileUtility.ERROR, tid+"getTags(): fields: "+fields);
                throw e;
                }
                }

                /// many methods here

                public void flush() throws Exception
                {
                try
                {
                for(int i = 0; i < records.size(); i++)
                {
                outputFile.writeToFile(((RecordUtil)records.elementAt(i)).getRecordString());
                }
                clearFieldData("*ALLB");
                tagCounter.clear();
                } catch (Exception e)
                {
                throw e;
                }
                }

                public void finish()
                {
                exit = true;
                }

                }
                --------------------------------------------------
                package com.packaging.extract;

                import com.packaging.*;
                import com.common.FileUtility;

                import java.math.*;
                import java.util.*;

                public class SubExtract1 extends Extractor implements Runnable
                {
                Map commonData = new HashMap();


                public SubExtract1(FileUtility logFile, ConfigUtil conf, int tid) throws Exception
                {
                this.logFile = logFile;
                threadId = tid;
                outputFileName = "dddddddddd";
                config = conf;
                ctable = "cccccccc";
                try
                {
                super.initialize();
                } catch (Exception e)
                {
                throw e;
                }
                }

                public void run()
                {
                try
                {
                while (!exit)
                {
                if(!record.isEmpty()) process(record);
                }
                } catch (Exception e)
                {
                e.printStackTrace();
                }
                }

                public void process(Map rcd) throws Exception
                {
                super.process(rcd);
                try
                {
                if (scenarioFound)
                {
                     //prrocessing...
                }
                } catch (Exception e)
                {
                throw e;
                }
                record.clear();
                }

                public void flush() throws Exception
                {
                try
                {
                                              //processing
                super.flush();
                } catch (Exception e)
                {
                throw e;
                }
                }
                }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                • 5. Re: threading taking loong time to process
                  EJP
                  Your threads are spending most of their time spinning while record.isEmpty(), using 100% CPU in the process.

                  Use a wait/notify technique, or a blocking queue, or something that doesn't set the CPU on fire like this.
                  • 6. Re: threading taking loong time to process
                    796440
                    Good eye. I'm way too lazy to read that much code if I'm not getting paid for it. :-)
                    • 7. Re: threading taking loong time to process
                      807580
                      ic, how do i use wait/notify here, please change/highlight in the code, how do i code that wait/notify stuff, never used it, tried to understand it too but not very much luck. without any proper example.
                      thanks
                      • 8. Re: threading taking loong time to process
                        807580
                        in addition to my last request:
                        ic, how do i use wait/notify here, please change/highlight in the code, how do i code that wait/notify stuff, never used it, tried to understand it too but not very much luck. without any proper example.
                        thanks

                        please also note the loop i am doing in WorkerThread to check if the record is processed or not.Is that okay or do i hvae to do the wait/notify there too??if yes , again how please?

                        i wish i could pay you jverd.
                        • 9. Re: threading taking loong time to process
                          796440
                          javalover2002 wrote:
                          ic, how do i use wait/notify here
                          By studying and learning how it works, understanding it, and then applying what you learned to your specific requirements.
                          , please change/highlight in the code, how do i code that wait/notify stuff
                          This is not a code service. If you want code, hire someone. If you want to learn about Java, show your work, and precisely describe your problems.
                          without any proper example.
                          When you went through at least 2 or 3 tutorials and/or books, typed in and ran the examples, modified them, wrote your own code, etc., what about it was not "proper"? What exactly did you not understand?
                          • 10. Re: threading taking loong time to process
                            807580
                            you are right jverd, but unfortunately i dont have enough time to read and do experiments, i have to deliver the code ASAP.

                            can someone please help and give me an example instead of lecturing me.
                            • 11. Re: threading taking loong time to process
                              807580
                              plus i am reading it already but a help would be GREATLY appreciated
                              • 12. Re: threading taking loong time to process
                                796440
                                javalover2002 wrote:
                                you are right jverd, but unfortunately i dont have enough time to read and do experiments, i have to deliver the code ASAP.
                                That doesn't alter the fact that this is not a code service. People come here to help others learn Java, not to fill in for their inability to do their jobs.
                                can someone please help and give me an example instead of lecturing me.
                                Look. If you post an SSCCE and explain clearly and precisely what problems you're having, you'll get help.

                                If you just say, "My code is too slow and I don't understand threading," you have much less chance of getting help other than broad suggesttions such as I gave (which was far from being a lecture, by the way).
                                • 13. Re: threading taking loong time to process
                                  796440
                                  javalover2002 wrote:
                                  plus i am reading it already but a help would be GREATLY appreciated
                                  How does anybody here know exactly which parts you don't understand?
                                  • 14. Re: threading taking loong time to process
                                    807580
                                    Anyone else please?
                                    1 2 3 Previous Next