This discussion is archived
11 Replies Latest reply: Feb 24, 2013 5:30 PM by vladodias RSS

File based rejection handling using Debatching

Tarak Newbie
Currently Being Moderated
Hi All,

I am facing very serious issue while using debatching option in file adapter.

I am reading file in de-batch mode. When any record is corrupted inside file then entire file is moving to default rejmsgs folder in SOA domain/soa_server1 folder.

Let say i have total 10 record in files. If record 2,5,and 7 have invalid record then entire message is moving to rejmsgs/process name folder. Because of this case size of the rejmsgs is increased to 20G.

I request some one to provide me the solution that how can avoid this case and movie only particular invalid record to rejection handler.

Appreciate for response.

Regards,
Tarak.
  • 1. Re: File based rejection handling using Debatching
    703425 Journeyer
    Currently Being Moderated
    Full file will get rejected only, this is default behavior.

    -Yatan
  • 2. Re: File based rejection handling using Debatching
    Tarak Newbie
    Currently Being Moderated
    Hi Yatan,

    Thanks for the update. Could you please let me how to avoid default functionality.

    I am using Debatching option in file and because of this it is causing issue.

    Regards,
    Tarak.
  • 3. Re: File based rejection handling using Debatching
    703425 Journeyer
    Currently Being Moderated
    If you read any file using file adapter which is not schema compliant it will be rejected and send to this folder inside the soa suite installation.

    You can have some control over where to send the file, or invoke another bpel process where you can handle the fault, or call some java class.

    I think we cannot control the default behavior if the file is corrupt.

    -Yatan
  • 4. Re: File based rejection handling using Debatching
    Tarak Newbie
    Currently Being Moderated
    Hi Yatan,

    Thanks again.

    Yes as you said we can control the rejection file location and name of the file in fault polices. But i am wondering how to control the entire rejected file,want to move only rejected record in the file, I think i need to raise SR with oracle. i will post the result if get good from Oracle.

    Regards,
    Tarak.
  • 5. Re: File based rejection handling using Debatching
    Ankitkalanoria Newbie
    Currently Being Moderated
    Hi Tarak,

    You can use attribute nxsd:uniqueMessageSeparator="${eol}" in xsd file of file adapter.

    Before this you will have to specify that input file consist of multiple messages while configuring file adapter. This is called debatching. After selecting this checkbox, adapter does translation line by line and publishes specific number of lines in one publish to jca based proxy service. You will see property "PublishSize" in jca file after updating this.

    This continues processing of messages even in case of some intermediate bad record. In the end rejection message consist of only bad records.

    Edited by: Ankit kalanoria on Feb 13, 2013 5:35 AM
  • 6. Re: File based rejection handling using Debatching
    Tarak Newbie
    Currently Being Moderated
    Hi Ankit kalanoria,

    Thank you for your response

    As you said i am using debatching mode and in my .Jca file i can see publishsize property value as 1.
    <adapter-config name="ReadRMSinput_file" adapter="File Adapter" wsdlLocation="ReadRMSinput_file.wsdl" xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">
      
      <connection-factory location="eis/FileAdapter" UIincludeWildcard="storetsf*.tsfxml"/>
      <endpoint-activation portType="Read_ptt" operation="Read">
        <activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
          <property name="DeleteFile" value="false"/>
          <property name="MinimumAge" value="0"/>
          <property name="PhysicalDirectory" value="/u01/textdata/SOADEV/SABAT/TSF2SIM/out"/>
          <property name="Recursive" value="false"/>
          <property name="PublishSize" value="1"/>
          <property name="PollingFrequency" value="10"/>
          <property name="IncludeFiles" value="storetsf.*\.tsfxml"/>
          <property name="UseHeaders" value="false"/>
        </activation-spec>
      </endpoint-activation>
    
    </adapter-config>
    XSD which i am using for debatching in fileadapter.
    <xs:schema elementFormDefault="qualified"
               targetNamespace="http://www.oracle.com/RMS/RMStoSIMTransfers/RMStoSIMTransfers.xsd"
               version="1.0"
               xmlns="http://www.oracle.com/RMS/RMStoSIMTransfers/RMStoSIMTransfers.xsd"
               xmlns:retailDoc="http://www.w3.org/2001/XMLSchema"
               xmlns:xs="http://www.w3.org/2001/XMLSchema">
             <xs:element name="Shipments">
              <xs:complexType>
                <xs:sequence>
                 <xs:element minOccurs="1" maxOccurs="unbounded" ref="Shipment" nxsd:uniqueMessageSeparator="${eol}"/>
                  </xs:sequence>
                  </xs:complexType>
                  </xs:element>
                  <xs:element name="Shipment">
                    <xs:complexType>
                      <xs:sequence>
                        <xs:element name="ActualShipDateTime" type="xs:dateTime"/>
                        <xs:element name="BolNumAlpha" type="xs:string"/>
                        <xs:element ref="Carrier" minOccurs="0"/>
                         <xs:element ref="Comments" minOccurs="0"/>
                        <xs:element ref="Customer" minOccurs="0"/> 
                        <xs:element name="InternalShipmentNum" type="xs:int"/>
                        <xs:element name="OrderType" type="xs:string"/>
                        <xs:element name="TotalContainers" type="xs:int"/>
                        <xs:element name="TotalLines" type="xs:int"/>
                        <xs:element name="TotalQuantity" type="xs:int"/>
                        <xs:element name="TotalVolume" type="xs:int"/>
                        <xs:element name="TotalWeight" type="xs:int"/>
                        <xs:element name="WeightUm" type="xs:string"/>
                         <xs:element ref="Details" minOccurs="0"/>
                              <xs:element ref="ItemDetails" minOccurs="0"/>
                         <xs:element ref="Containers" minOccurs="0"/>
                         </xs:sequence>
                        </xs:complexType>
                        </xs:element>
                        <xs:element name="Carrier">
                           <xs:complexType>
                            <xs:sequence>
                              <xs:element name="Carrier" type="xs:string"/>
                            </xs:sequence>
                          </xs:complexType>
                        </xs:element>
                        <xs:element name="Comments">
                          <xs:complexType>
                            <xs:sequence>
                              <xs:element name="Comment" minOccurs="0"
                                          maxOccurs="unbounded">
                                <xs:complexType>
                                  <xs:sequence>
                                    <xs:element name="Text" type="xs:string"/>
                                  </xs:sequence>
                                </xs:complexType>
                              </xs:element>
                            </xs:sequence>
                          </xs:complexType>
                        </xs:element>
                        <xs:element name="Customer">
                          <xs:complexType>
                            <xs:sequence>
                              <xs:element name="Customer" type="xs:string"/>
                            </xs:sequence>
                          </xs:complexType>
                        </xs:element>
                        <xs:element name="Details">
                          <xs:complexType>
                            <xs:sequence>
                              <xs:element ref="ShipmentDetail" minOccurs="0"
                                          maxOccurs="unbounded"/>
                               </xs:sequence>
                                </xs:complexType>
                                </xs:element>
                                <xs:element name="ShipmentDetail">
                                 <xs:complexType>
                                  <xs:sequence>
                                    <xs:element name="UserDef1" type="xs:string"/>
                                    <xs:element name="UserDef2" type="xs:string"/>
                                    <xs:element name="UserDef6" type="xs:string"/>
                                    <xs:element name="UserDef5" type="xs:string"/>
                                    <xs:element name="ErpOrder" type="xs:string"/>
                                  </xs:sequence>
                                </xs:complexType>
                              </xs:element>
                             <xs:element name="ItemDetails">
                          <xs:complexType>
                            <xs:sequence>
                              <xs:element ref="ItemDetail" minOccurs="0"
                                          maxOccurs="unbounded"/>
                               </xs:sequence>
                                </xs:complexType>
                                </xs:element>
                                <xs:element name="ItemDetail">
                                 <xs:complexType>
                                  <xs:sequence>
                                    <xs:element name="Item" type="xs:int"/>
                                    <xs:element name="Quantity" type="xs:decimal"/>
                                    </xs:sequence>
                                </xs:complexType>
                              </xs:element>
                        <xs:element name="Containers">
                          <xs:complexType>
                            <xs:sequence>
                              <xs:element ref="ShippingContainer" minOccurs="0"
                                          maxOccurs="unbounded"/>
                                </xs:sequence>
                                </xs:complexType>
                                </xs:element>
                                <xs:element name="ShippingContainer">
                                <xs:complexType>
                                <xs:sequence>
                                    <xs:element name="ContainerId" type="xs:string"/>
                                    <xs:element ref="ContainerDetails" minOccurs="0"/>
                                    <xs:element ref="ContainerType" minOccurs="0"/>                            
                                    <xs:element name="ManifestCarrServiceSymbol"
                                                type="xs:string"/>
                                    <xs:element name="ManifestId" type="xs:string"/>
                                    <xs:element name="TotalFreightCharge"
                                                type="xs:decimal"/>
                                    <xs:element name="TotalFreightDiscount"
                                                type="xs:decimal"/>
                                    <xs:element name="TotalQuantity"
                                                type="xs:decimal"/>
                                    <xs:element name="TotalWeight"
                                                type="xs:decimal"/>
                                    <xs:element name="TrackingNumber"
                                                type="xs:string"/>
                                    <xs:element name="WeightUm" type="xs:string"/>
    
                                    </xs:sequence>
                                    </xs:complexType>
                                    </xs:element>
                                    <xs:element name="ContainerDetails">
                                    <xs:complexType>
                                    <xs:sequence>
                                    <xs:element ref="ContainerDetail" minOccurs="0" maxOccurs="unbounded"/>
                                    </xs:sequence>
                                    </xs:complexType>
                                    </xs:element>
                                    <xs:element name="ContainerDetail">
                                    <xs:complexType>
                                     <xs:sequence>
                                    <xs:element name="Item" type="xs:int"/>
                                    <xs:element name="Quantity"   type="xs:decimal"/>
                                    </xs:sequence>
                                    </xs:complexType>
                                    </xs:element>
                                    <xs:element name="ContainerType">
                                      <xs:complexType>
                                        <xs:sequence>
                                          <xs:element name="DimensionUm"
                                                      type="xs:string"/>
                                          <xs:element name="Height"
                                                      type="xs:decimal"/>
                                          <xs:element name="Length"
                                                      type="xs:decimal"/>
                                          <xs:element name="MaxWeight"
                                                      type="xs:decimal"/>
                                          <xs:element name="Volume"
                                                      type="xs:decimal"/>
                                          <xs:element name="Weight"
                                                      type="xs:decimal"/>
                                          <xs:element name="Width"
                                                      type="xs:decimal"/>
                                        </xs:sequence>
                                      </xs:complexType>
                                    </xs:element>
                                </xs:schema>
    After adding the attribute uniqueMessageSeparator, if any record in the xml is invalid then my De-batching process is getting stopped. Could you pls let me know if any thing more i have to define in my xsd in order to avoid stopping debatching mode..
  • 7. Re: File based rejection handling using Debatching
    Ankitkalanoria Newbie
    Currently Being Moderated
    Hi Tarak,

    Everything looks correct. Just that uniqueMessageSeparator is attribute for schema and not element. So you will have to modify your xsd like this(Please observe that i have shifted position of attribute uniqueMessageSeparator from element level to schema level):

    <xs:schema elementFormDefault="qualified" nxsd:uniqueMessageSeparator="${eol}"
    targetNamespace="http://www.oracle.com/RMS/RMStoSIMTransfers/RMStoSIMTransfers.xsd"
    version="1.0"
    xmlns="http://www.oracle.com/RMS/RMStoSIMTransfers/RMStoSIMTransfers.xsd"
    xmlns:retailDoc="http://www.w3.org/2001/XMLSchema"
    xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="Shipments">
    <xs:complexType>
    <xs:sequence>
    <xs:element minOccurs="1" maxOccurs="unbounded" ref="Shipment"/>
    </xs:sequence>
    </xs:complexType>

    Please try this and see if it works.
  • 8. Re: File based rejection handling using Debatching
    Tarak Newbie
    Currently Being Moderated
    Hi Ankit kalanoria,

    After adding the attribute in xsd level still i am not able to move the invalid record. Also my batch process is getting effected because of in valid data. SOA is processing the records but when it finds invalid date then the batch process is terminating.

    Regards,
    Tarak.
  • 9. Re: File based rejection handling using Debatching
    vladodias Guru
    Currently Being Moderated
    Hi Tarak,

    Just to clarify, you are not using Native Format Builder, are you? I mean, your file is already xml, is that right?

    I believe you are using something like this... But in 11g...
    http://oraintpc.blogspot.com.au/2007/08/xml-debatching-in-file-ftp-adapter.html

    So, the debatching should create multiple instances of your BPEL, are you observing that?

    If the xml is not well-formed, the debatching will stop, there's not much do to about it...

    What you can do is to register a batch notification handler like described here...
    http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_file.htm#CACJBIGD

    In the handler you can then treat the exception, and maybe "reinject" the remainder of the file if that's what you want.

    Cheers,
    Vlad
  • 10. Re: File based rejection handling using Debatching
    Tarak Newbie
    Currently Being Moderated
    Just to clarify, you are not using Native Format Builder, are you? I mean, your file is already xml, is that right?
    Yes Vald i am not using native format builder and using XML file for debatching.
    So, the debatching should create multiple instances of your BPEL, are you observing that?
    Yes Vald, it is creating multiple instances.
    What you can do is to register a batch notification handler like described here...
    http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_file.htm#CACJBIGD
    Vald, i am not getting how to use java class in my process.

    I am not sure what it meant in the document You can register a batch notification callback (Java class) which is invoked when the last batch is reached in a debatching scenario.

    Can you please elaborate how use Java class.
    In the handler you can then treat the exception, and maybe "reinject" the remainder of the file if that's what you want.
    I need you help in sending reminder if any batch fails.


    Once again thanks so much for showing the way.

    Regards,
    Tarak.
  • 11. Re: File based rejection handling using Debatching
    vladodias Guru
    Currently Being Moderated
    Hi Takar,

    When you define a batchHandler and have a failed batch, onFailedBatch method should be called, the metadata should contain the location in the file where the batch has stopped. By the information given you should be able to figure out in your code how many lines were actually processed.

    In order to create a "recovered" file, you will need to skip the processed records and the error records, this may required some heuristic approach... You can just skip lines on the file until you find the next valid record and create a new file to process... This has to be done by the java code manually...

    Using StAX may help...
    http://docs.oracle.com/cd/E23943_01/web.1111/e13724/stax.htm#XMLPG173
    <service ...
      <binding.jca ...
        <property name="batchNotificationHandler">com.acme.batchHandler</property>
    where com.acme.batchHandler must implement
    
    package oracle.tip.adapter.api.callout.batch;
    public interface BatchNotificationCallout extends Callout
    {
        public void onInitiateBatch(String rootId,
                                    String metaData)
            throws ResourceException;
        public void onFailedBatch(String rootId,
                                  String metaData,
                                  long currentBatchSize,
                                  Throwable reason)
            throws ResourceException;
        public void onCompletedBatch(String rootId,
                                    String metaData,
                                                       long finalBatchSize)
            throws ResourceException;
    Cheers,
    Vlad

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points