6 Replies Latest reply: Aug 14, 2014 2:34 AM by Srimant-Oracle RSS

    Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" />.Please help me in this.

    1011327

      Hi Experts,

      Is there any way to get the number of records present in a file ,while reading the same using a ftp adapter.

       

      Also in existing code do not read file content is enabled while reading the file( <property name="UseHeaders" value="true" />.Please help me in this.)

       

      I am Planning to use this Adapter in OSB project.

       

      Please help me in solving this issue.

      Thanks in Advance.

        • 1. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
          Daniel R. Dias

          Hello,

           

          To count the number of records you can use the xquery function word-count FunctX XQuery Functions: functx:word-count if you just want to count the records, or the function tokenize FunctX XQuery Functions: fn:tokenize if you want to parse the records for further use.

           

          Regarding your other question, you can select the option "Do not read file content" (step 5)

           

          Best Regards,

          Daniel Dias

          • 2. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
            Srimant-Oracle

            Hi,

             

            I wanted to understand what you mean by records. Does it mean the number of lines or could each record encompass more than one line?

             

            Could you explain the use case a bit more? One approach I can think of is to write a RecordCountingValve. More information is available here:

             

            http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_valve.htm#CHDDIICH

             

            What you would need to do is to copy the inputstream to a byte array or store it in file (depending on the size). Count the number of records and then add that as a property to the InputStreamContext e.g.

             

            public InputStreamContext execute(InputStreamContext inputStreamContext)

                              throws IOException, PipelineException {

                        // Get the input stream that is passed to the Valve

                        InputStream originalInputStream = inputStreamContext.getInputStream();

                        OutputStream out = ..();

                        try{

                        PipelineUtils.copyStream(originalInputStream, out);

                        long lines = countLines(...);

                        newInputStream = readBackInputStream(); //read from the out back into inputstream

                        inputStreamContext.addProperty("file.recountCount", String.valueOf(lines));

                        inputStreamContext.setInputStream(newInputStream);

                        return inputStreamContext;

                       finally{

                           close(originalInputStream);

                           close(out);

                       }

             

            You will need to compile this class, add to WLS classpath and then refer to it from the jca file.

            In the JCA File, you will need to add a property for this class name

            <property name="PipelineValves" value="myvalves.RecordCountingValve"/>

            • 3. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
              1011327

              Hi Daniel/Srimant ,

               

              Thanks for replying.As of now my exact requirement is like ,I have got two FTP jca files and corresponding wsdl. using first jca file i am reading the file but not the content(ie:property name="UseHeaders" value="true" ) ,and using second jca I am doing direct ftp move from FTP location to local file location(filemove.jca).(

              <adapter-config name="Filemove" adapter="FTP Adapter" wsdlLocation="Filemove.wsdl" xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">

                <connection-factory location="eis/ftp/FtpAdapter"/>

                <endpoint-interaction portType="FTPMove_ptt" operation="FTPMove">

                  <interaction-spec className="oracle.tip.adapter.ftp.outbound.FTPIoInteractionSpec">

                    <property name="SourcePhysicalDirectory" value="/XXX/OUT"/>

                    <property name="SourceFileName" value="TEMP.TXT"/>

                    <property name="TargetPhysicalDirectory" value="/localfolder"/>

                    <property name="FileNamingConvention" value="Temp.txt"/>

                    <property name="TargetIsRemote" value="false"/>

                    <property name="Type" value="COPY"/>

                  </interaction-spec>

                </endpoint-interaction>

              </adapter-config>).

              And in OSB level I have created  a proxy service out of this read jca and another one business service out of filemove.jca. From that proxy obtained using read.jca am directing calling business service created using filemove.jca .

               

              Now my requirement is i need to get the total no lines (based on eol in file which I am reading) in this proxy service.Since i am not reading content of file,am unable to get the record count from body level and in header information jca.file.batch is returning null.

               

              This is my issue ,but i need to get the count of the total no of lines present in the file which am reading using FTP adapter some how.

               

              I am trying the methods suggested by you to solve this issue.If you find some other solution please update me on this.

               

              Thank you.

              • 4. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
                Srimant-Oracle

                Hi,

                 

                I wanted to know what you would wish to do with the record count.

                 

                If this were a File on the File system, I would have suggested to use a Java Callout Action within OSB to read the number of records.  I am not aware of any other option but to write custom code to handle this case (either using a adapter valve or custom java callout in OSB).

                 

                If you were to use adapter valve,

                 

                public InputStreamContext execute(InputStreamContext inputStreamContext)

                                  throws IOException, PipelineException {

                            // Get the input stream that is passed to the Valve

                            InputStream originalInputStream = inputStreamContext.getInputStream();

                            OutputStream out = null;

                            try{

                            File newFile = PipelineUtils.getUniqueStagingFile(new File(System.getProperty("java.io.tmpDir")));

                            out = new FileOutputStream(newFile);

                            PipelineUtils.copyStream(originalInputStream, out);

                            out.close();

                            long lines = readCountFromFile(newFile);

                            newInputStream = new FileInputStream(newFile);

                            inputStreamContext.addProperty("file.recordCount", String.valueOf(lines));

                            inputStreamContext.setInputStream(newInputStream);

                            return inputStreamContext;

                           finally{

                               close(originalInputStream);

                               close(out);

                           }


                Now, within OSB pipeline, you will need to read the "file.recordCount" endpoint property that you had added in the valve.

                • 5. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
                  1011327

                  Hi

                   

                   

                  I want that record count because my requirement is to insert the record count(total no of lines in the file which we read) in a table,I will be also inserting some other parameters along with this record count.

                   

                  I got some doubts in your solution , please help me in solving this.Since i am not reading the file content(ie body will be empty(ie:property name="UseHeaders" value="true")),Even if i pass the body as argument to the method you have specified I hope it will pass null.Any how I will try it.

                   

                  Another issue is that my file may have 400000,In this scenario even if this method works,I doubt the performance.

                   

                  Thanks,

                  Kiran.

                  • 6. Re: Is there any way to get the number of records present in a file while reading the same using a ftp adapter,if <property name="UseHeaders" value="true" ></property>.Please help me in this.
                    Srimant-Oracle

                    Hi Kiran,

                     

                    I did not realize that you are using "No Payload' option. In this case, the Valve I mentioned will not be invoked.

                     

                    Can we approximate the line count e.g. we read the header for size e.g. "jca.ftp.size" and divide by approximate number of characters in a line (say 80). If the size of file is 1 MB (1000000) and we approximate each line to have 80 characters, then number of lines would come out to 1000000/80.

                     

                    If this is not an option, then we would need to do some custom java code in OSB pipeline to read the number of lines. In your case, you will get the ftp directory name and file name as properties e.g. jca.ftp.Directory and jca.ftp.FileName. We would then need to download the remote file to local using FTPIoInteractionSpec (with source being remote and target being local). And, then use the custom java code to read the number of lines.