3 Replies Latest reply on Dec 8, 2009 2:51 PM by alwu-Oracle

    How can I enhance loading performance?

    737663
      Hi, all.

      I'm loding LUBM 50 data set during 4 hours in 11g release2 with new jena apaptor. (Linux 86_64, 16GB RAM )
      At First, several data files(.nt) is loaded within several minutes.
      But, as time goes by, and as the amount of the used memory is increasing, loading speed remarkably down.
      I don't know the reason.
      Now, it looks like the loading stopped.



      Could you tell me an advice?

      I'm using below code.


           private void init() throws Exception {
                String jdbcURL = "jdbc:oracle:thin:@" + Constants.HOST + ":1521:" + Constants.SID;
                OracleDataSource ds = new OracleDataSource();
                ds.setURL(jdbcURL);
                ds.setUser(Constants.ORACLE_USER);
                ds.setPassword(Constants.ORACLE_PASSWORD);

                OracleConnection conn = (OracleConnection) ds.getConnection();
                oracle = new Oracle(conn);

                modelOracleSem = ModelOracleSem.createOracleSemModel(oracle, modelName);
                graphOracleSem = modelOracleSem.getGraph();

                graphOracleSem.createTables();
                graphOracleSem.dropApplicationTableIndex();

                graphOracleSem.clearRepository();
           }

           private void loadingData() throws Exception {
                String dataSetType = Constants.getDataSetType(dType);

                long start = System.nanoTime();
                
                Model model = ModelFactory.createDefaultModel();

                File dir = new File(Constants.DATA_PATH + File.separator + dataSetType);
                File[] fileList = dir.listFiles();

                File file = null;
                String inputFilePath = null;

                for (int i = 0; i < fileList.length; i++) {
                     file = fileList;

                     inputFilePath = file.getCanonicalPath();
                     
                     System.out.println("Loading triples from " + inputFilePath);

                     InputStream in = null;
                     
                     try {
                          in = FileManager.get().open(inputFilePath);
                     
                          if (in == null) {
                               throw new IllegalArgumentException("File: " + inputFilePath + " not found.");
                          }
                          
                          model.read(in, "", "N-TRIPLE");

                          graphOracleSem.getBulkUpdateHandler().addInBulk(GraphUtil.findAll(model.getGraph()), tableSpaceName);

                          graphOracleSem.commitTransaction();
                          graphOracleSem.flushAdd();
                     } finally {
                          if (in != null)
                               in.close();
                     }
                }
                
                graphOracleSem.rebuildApplicationTableIndex();
           }
        • 1. Re: How can I enhance loading performance?
          user61472
          Can you take some snapshots of your database and run off an AWR report during the period you are running your upload for. You need to see what your database is wating on.
          • 2. Re: How can I enhance loading performance?
            737663
            Thanks, DonnyDBA.

            Unfortunately, I don't have dba account for oracle 11g. I can not get the AWR report.
            I should find a way to enhance performance by programming code.
            But, I can see the information of system memory.

            --------------------------------------------------------------------------------------------------------------
            Mem: 16442560k total, 16357228k used, 85332k free, 27092k buffers
            Swap: 18808816k total, 408304k used, 18400512k free, 11611716k cached

            PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
            4361 oracle 15 0 7291m 2.4g 1.8g S 1 15.1 250:11.62 oracle
            3815 oracle 16 0 6771m 1.8g 1.7g D 1 11.2 3:43.82 oracle
            3797 oracle 15 0 6737m 18m 16m S 0 0.1 0:00.64 oracle
            3630 oracle 15 0 81204 12m 9248 S 0 0.1 0:00.15 tnslsnr
            ---------------------------------------------------------------------------------------------------------------

            Whenever oracle load triples, memory consuming is increased. Now, oracle is using almost all memory.
            In this system, running program is only the test code.

            Is this a memory leak?
            If this is a memory leak, I think my test code have some problems.

            When flushAdd() or commitTransaction() method is excuted, do oracle return the used memory?
            • 3. Re: How can I enhance loading performance?
              alwu-Oracle
              Hi,

              We have loaded LUBM 50 many times without a problem using Jena Adaptor.

              Can you first load all those ontology files into a single Jena in-memory model and then use addInBulk to load that Jena model into Oracle?

              Thanks,

              Zhe Wu