1 2 Previous Next 18 Replies Latest reply: Aug 7, 2013 12:51 PM by user4769191 RSS

    RL Client clarification

    ankV

      Hi Experts,

       

      I am having a xml file which is having all the catalog items(categories,child categories,product's,child product's,sku's) and price item in sku level and will come from different system.

       

      My requirement is to load / dump this data into different ATG tables(DCS_CATALOG,DCS_PRODUCT,DCS_PRODUCT,DCS_PRICE) and some customized tables too.

       

      Can we use RL Client for the above scenario and Will that do parsing and all?

       

      Please help me to proceed this using ATG way.

       

       

      Thanks,

      ankV

        • 1. Re: RL Client clarification
          Nooruddin Shaik

          Try this:

          C:\ATG\ATG9.3\home\bin>startSQLRepository.bat -m MyModule -repository /atg/commerce/catalog/ProductCatalog -import C:\test\productCatalog.xml

           

          Peace

          Shaik

          • 2. Re: RL Client clarification
            ankV

            please find below details for input xml.

             

            <DeviceList>

                 <Device>

                      <SKU>sku name</SKU>

                      <Manufacturer code="manu">manufacturer</Manufacturer>

                      <ShortDescription>shortdescription></ShortDescription>

                      <LongDescription>longdescription</LongDescription>

                     <Category>category</Category>

                     <SubCategory>subcategory</SubCategory>

                     <Price>88.00</Price>

                 <Device>

            <DeviceList>

             

            Will startSQLRepository.bat work on this case ?


            Also, I have gone through the Repository Loader concepts in the oracle docs where I don't see this kind of approach at all.


            Please help me on this.

            • 3. Re: RL Client clarification
              Nitin Khare

              startSQLRepository can only be used if you have input XML in the ATG acceptable format (GSA DTD) which would be like:

               

              <?xml version="1.0" encoding="UTF-8" standalone="no"?>
              <!DOCTYPE gsa-template SYSTEM "dynamosystemresource:/atg/dtds/gsa/gsa_1.0.dtd">
              
              <gsa-template>
              <import-items>
              
              <add-item item-descriptor="sku" id="sku100001">
                <set-property name="name"><![CDATA[sku name]]></set-property>
                <set-property name="shortDescription"><![CDATA[sku short description]]></set-property>
                <set-property name="longDescription"><![CDATA[sku long description]]></set-property>
                <set-property name="category"><![CDATA[category]]></set-property>
                <set-property name="subcategory"><![CDATA[subcategory]]></set-property>
                <set-property name="listPrice"><![CDATA[88.00]]></set-property>
                <set-property name="creationDate"><![CDATA[7/5/2013 14:10:57]]></set-property>
                <set-property name="siteIds"><![CDATA[sitename]]></set-property>
                ..
                ..
              </add-item>
              ..
              ..
              </import-items>
              </gsa-template>
              

               

              For your input XML you can use RL to load the data. There is one /atg/dynamo/service/loader/typemapping/Xml2RepositoryContentHandler component provided in the RL module which uses ATG’s xml2repository system to transform files into XML files that use the xml2repository schema, which are then transformed into repository items. This component has a property "enableTransforms" which you have to set to true in order to use a xml transformer for preprocessing the input xml so that it uses XMLTransformer property which by default points to SampleXMLTransformer component. It can perform one or more XSL transforms on input source files. You will have to set the XSL stylesheets in the SampleXMLTransformer.styleSheets property which is an array of style sheet files to use in transforming XML files.

              • 4. Re: RL Client clarification
                ankV

                Should we need to create a manual xsd file based on the input xml file?

                 

                Here, The input xml file is having the category,child categories, child products,product sku and price information which needs to be transformed into repository items.

                So,Do we need to create a mapping for each and ever item descriptors or Can't we load repository level?

                • 5. Re: RL Client clarification
                  Nitin Khare

                  Not sure why do you want to create xml schema here... what purpose will it serve? ATG already has a generateXMLSchema utility which can generate a XML schema based on the given repository definition and a mapping file (which defines what repository item properties to be included and their names that can be different from the default property name).

                   

                  If you are importing catalog data from external system, one of the first basic step would involve modelling your catalog as per your system requirements in which you would extend, add or remove properties of various catalog items in the OOTB ProductCatalog repository so that you can map and populate external data in repository. You would have to analyze and design according to your requirements and what data you are having readily available for import and what you might want to let the business users create later from BCC.

                   

                  As I already mentioned you can use RL with a XSL transformation getting applied to the input XML to transform it in ATG specific format. In your case of input XML here is a possible XSL that can be applied (I have assumed a "skuno" element in your input XML to use it for sku-Id)

                   

                  <?xml version="1.0" encoding="ISO-8859-1" ?>
                  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
                      <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" media-type="text/xml"/>
                      <xsl:template match="/">
                          <xsl:element name="gsa-template">
                              <xsl:element name="import-items">
                                  <xsl:for-each select="Device">
                                      <add-items item-descriptor="sku" id="sku-{skuno}">
                                          <set-property name="name"><xsl:value-of select="SKU" /></set-property>
                                          <set-property name="shortDescription"><xsl:value-of select="ShortDescription" /></set-property>
                                          <set-property name="longDescription"><xsl:value-of select="LongDescription" /></set-property>
                                          <set-property name="listPrice"><xsl:value-of select="Price" /></set-property>
                                      </add-items>
                                  </xsl:for-each>
                              </xsl:element>
                          </xsl:element>
                      </xsl:template>
                  </xsl:stylesheet>
                  

                   

                  You can also check your output XML after XSL transformation is applied on the input XML using XMLTransform droplet. This may not be complete solution to your problem but I am just trying to explain the underlying concept and approach that you may want to take. Hope it helps.

                  • 6. Re: RL Client clarification
                    ankV

                    Thanks for your valuable inputs Nitin. But, I have confused totally here.

                     

                    Please clarify me , Don't we need to create a mapping file and schema file for the above requirement ? . If it is a case then where we need to configure a input xml file which is readable by the system and I hope it will be in FileSystemMonitorService component.

                     

                    Also, I don't see any utility to create a XSL file. Please help me to understand.

                     

                    I am triggering the job using Repository Loader interface where I can able to create a job and the status is complete but I don't see any entry on db or any transformed xml file.

                    • 7. Re: RL Client clarification
                      Nitin Khare

                      I guess you got confused by bunch of components present in <ATG>\RL\config\atg\repository\xml, especially the SchemaManager which has properties like  XMLSchemaFileDirectory and mappingFileDirectories. This is basically part of ATG's repository-XML binding system. You should refer to this documentation

                       

                      Oracle ATG Web Commerce - Repository to XML Data Binding

                       

                      Now the Repository Loader can use the same XML schemas so that it can create ATG repository items from the XML files by loading input XML files or export existing repository items in the form of XML files.

                       

                      You must also keep in mind that there is a FileSystemMonitorScheduler which is a scheduler that starts FileSystemMonitorService. In FileSystemMonitorService you would have to specify the rootPath for the location where your input files are present that will be used by RL. Having scheduler helps rescanning the input file locations to pick-up and load any updated input files.

                       

                      Now once you create a job and when that job is triggered it it calls specific content handlers through TypeMapper and TypeMappings components. In your case it should be using Xml2RepositoryContentHandler where you need to have enableTransforms=true and specify the "styleSheets" property in SampleXMLTransformer component to the stylesheets file having content similar to the XSL snippet that I posted earlier. On processing, your input xml will be transformed to the ATG acceptable format and RL would be able to create the repository items out of it. It should work that way.

                       

                      I recommend you also enable loggingDebug on the involved components to see the complete flow and possibly look at which point it is failing. Hope it helps.

                      • 8. Re: RL Client clarification
                        ankV

                        Hi Nitin,

                         

                        As per your latest post, I understood, We can't use the Oracle ATG Web Commerce - Repository to XML Data Binding concept in my case. Is that correct?. If yes, then I wonder, why can't we validate and parse using xsd file as ATG is providing Repository concept which is purely based on XML?.


                        Now, I can able to get the transformed XML file which is based on XSL for a given XML file.

                        Here, I have two questions.

                        1.Do we have any utility to create XSL file in ATG?.

                        2.How can we load this XML into DB using scheduler basis as I don't want to run manually using startSQLRepository.bat file?


                        Thanks in Advance.

                        ankV

                        • 9. Re: RL Client clarification
                          ankV

                          ++

                           

                          I am getting the below exception while transforming the XML file too.

                           

                          **** debug Thu Jul 11 19:29:12 IST 20131373551152366   /atg/repository/xml/LoaderAddServic
                          startDocument was called with public id: null and system Id: null
                          **** debug Thu Jul 11 19:29:12 IST 20131373551152366   /atg/repository/xml/LoaderAddServic
                          startElement called with localName: gsa-template and namespace URI:
                          **** debug Thu Jul 11 19:29:12 IST 20131373551152366   /atg/repository/xml/LoaderAddServic
                          calling end on TransactionDemarcation: transaction=Xid=BEA1-0CE71F9678D3(29146635),Status=A

                          ctive,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=30,activeThread=

                          Thread[RLJobThread#0,5,Pooled Threads],SCInfo[zig_domain+AdminServer]=(state=active),OwnerTransacti

                          onManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=AdminServer+localhost:7001+zig_domai

                          n+t3+, XAResources={ZIG_PROD_zig_domain, WSATGatewayRM_AdminServer_zig_domain},NonXAResources={})])

                           

                          **** Error Thu Jul 11 19:29:12 IST 20131373551152367   /atg/dynamo/service/loader/LoaderMa
                          nager   logging unhandled exception in XMLManifestBackedJob.runJob:java.lang.StringIndexOutOfB

                          oundsException: String index out of range: -22

                          at java.lang.String.substring(String.java:1937)
                          at java.lang.String.substring(String.java:1904)
                          at atg.repository.xml.RepositoryXMLTools.getItemMappingElement(RepositoryXMLTools.java:780)

                           

                          at atg.repository.xml.ModifyHandler.startElement(ModifyHandler.java:703)
                          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXPar

                          ser.java:501)

                          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSD

                          ocumentScannerImpl.java:400)

                          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRoo

                          tElementHook(XMLNSDocumentScannerImpl.java:626)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDr

                          iver.next(XMLDocumentFragmentScannerImpl.java:3104)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocu

                          mentScannerImpl.java:922)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerIm

                          pl.java:648)

                          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScann

                          erImpl.java:140)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLD

                          ocumentFragmentScannerImpl.java:511)

                          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.j

                          ava:808)

                          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.j

                          ava:737)

                          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
                          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.jav

                          a:1205)

                          at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.

                          java:522)

                          at weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:133)
                          at weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:173)
                          at atg.repository.xml.AddService.addItem(AddService.java:502)
                          at atg.repository.xml.AddService.addItem(AddService.java:407)
                          at atg.repository.loader.Xml2RepositoryContentHandler.processAdd(Xml2RepositoryContentHandl

                          er.java:908)

                          at atg.repository.loader.BaseJob.processOp(BaseJob.java:761)
                          at atg.repository.loader.XmlManifestBackedJob.handleAdd(XmlManifestBackedJob.java:619)
                          at atg.repository.loader.XmlManifestBackedJob$ManifestHandler.endElement(XmlManifestBackedJ

                          ob.java:894)

                          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParse

                          r.java:601)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XM

                          LDocumentFragmentScannerImpl.java:1782)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDr

                          iver.next(XMLDocumentFragmentScannerImpl.java:2939)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerIm

                          pl.java:648)

                          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLD

                          ocumentFragmentScannerImpl.java:511)

                          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.j

                          ava:808)

                          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.j

                          ava:737)

                          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
                          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.jav

                          a:1205)

                          at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.

                          java:522)

                          at weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:133)
                          at weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:173)
                          at atg.repository.loader.XmlManifestBackedJob.runJob(XmlManifestBackedJob.java:421)
                          at atg.repository.loader.JobThread.run(JobThread.java:158)
                          • 10. Re: RL Client clarification
                            Nitin Khare

                            Your queries required me to look into some more details and I realized that a point made by you earlier was right.

                            As per your latest post, I understood, We can't use the Oracle ATG Web Commerce - Repository to XML Data Binding concept in my case. Is that correct?. If yes, then I wonder, why can't we validate and parse using xsd file as ATG is providing Repository concept which is purely based on XML?

                            Perhaps I was not able to explain this clearly in earlier post. If you are using repository loader then you can very well make use of ATG's Respository-XML binding framework. You are right that we can validate the XML against a schema but you will have to do some groundwork for that. The XSL snippet I posted was just for reference and not complete. You will have to include the DTD references like below in order to make it fully acceptable:

                             

                            <xsl:output omit-xml-declaration="no" standalone="no" indent="yes" method="xml" encoding="UTF-8"
                              doctype-public="-//Art Technology Group, Inc.//DTD Dynamo Security//EN"  
                              doctype-system="http://www.atg.com/dtds/gsa/gsa_1.0.dtd" />
                            

                             

                            I found that I linked it a bit wrong above as the earlier XSL (with mentioned doctype declarations) will transform your input XML so that you can use it with startSQLRepository. Now in order to use your input XML with RL you can generate a XML schema using generateXMLSchema and the mapping file and then write a XSL which will transform your input XML to the XML conforming to that schema. For mapping files and repository to XML you can refer this for more details:

                             

                            Oracle ATG Web Commerce - Repository Operations

                             

                            1.Do we have any utility to create XSL file in ATG?

                            As far as I know there is no utility for generating the XSL provided in ATG. ATG only gives generateXMLSchema utility which uses a mapping file. There are some default mapping files corresponding to various items in ProductCatalog repository provided for the reference in <ATG>\DCS\mappingFiles folder. You may want to change the default mapping files after designing your catalog repository as per your needs. Once you design your catalog, update mapping files and finally generate the schema then you can generate a sample XML based on that schema and accordingly you would have to write your XSL. Here is one XSL that I have used successfully with RL to load sku item for your input file:

                             

                            <?xml version="1.0" encoding="UTF-8" ?>
                            <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
                            xmlns="http://www.atg.com/ns/skuMap/ProductCatalog/sku" 
                            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                            xsi:schemaLocation="http://www.atg.com/ns/skuMap/ProductCatalog/sku skuMap+ProductCatalog+sku.xsd" >
                                
                                <xsl:output omit-xml-declaration="no" standalone="no" indent="yes" method="xml" encoding="UTF-8" />
                            
                                <xsl:template match="/Device">
                                                <xsl:variable name="skuId"><xsl:value-of select="skuno" /></xsl:variable>
                                                <!-- <sku repositoryId="sku{skuno}" ID="sku{skuno}" > -->
                                                <sku ID="sku{$skuId}" >    
                                                    <sku.id>sku<xsl:copy-of select="$skuId" /></sku.id>
                                                    <sku.displayName><xsl:value-of select="SKU" disable-output-escaping="no"/></sku.displayName>
                                                    <sku.description><xsl:value-of select="LongDescription" /></sku.description>
                                                    <sku.displayNameDefault><xsl:value-of select="ShortDescription" /></sku.displayNameDefault>
                                                </sku>
                                </xsl:template>
                            </xsl:stylesheet>
                            

                             

                            For the schema generation in this case you can use following mapping file (skuMap.xml):

                             

                            <!DOCTYPE item-descriptor SYSTEM "http://www.atg.com/dtds/databinding/itemDescriptorMapping_1.0.dtd">
                            
                             <item-descriptor repository-path="/atg/commerce/catalog/ProductCatalog" name="sku" default-include="false">
                              <property name="id" include="true"/>
                              <property name="displayName" include="true"/>
                              <property name="description" include="true"/>
                              <property name="displayNameDefault" include="true"/>
                             </item-descriptor>
                            

                             

                            2.How can we load this XML into DB using scheduler basis as I don't want to run manually using startSQLRepository.bat file?

                            There is already a scheduler provided for this in form of component /atg/dynamo/service/loader/FileSystemMonitorScheduler. You will have to set its enable property true in order to use it. Once enabled it will get called by FileSystemMonitorService and will scan your input folders for the updated input files and reload them. You would have to enable it through properties file and restart your server and ensure you can see it under registered schedulers from dyn/admin.

                             

                            Also, check following configurations in your environment:

                             

                            /atg/dynamo/service/loader/typemapping/Xml2RepositoryContentHandler

                            enableTransforms=true

                             

                            /atg/dynamo/service/loader/LoaderManager

                            acceptingNewJobs=true

                            # loggingDebug=true

                             

                            /atg/dynamo/service/loader/typemapping/SampleTypeMapping

                            contentIsXML=true

                            itemDescriptorName=sku

                            parseContent=true

                            repository=/atg/commerce/catalog/ProductCatalog

                            # loggingDebug=true

                             

                            /atg/dynamo/service/loader/typemapping/SampleXMLTransformer

                            styleSheets=<PATH_TO_XSL_FOR_INPUT_XML_TRANSFORM>\style.xsl

                            #loggingDebug=true

                             

                            /atg/repository/xml/SchemaManager

                            XMLSchemaFileDirectory=<ATG>\home\schemas     # place your generated xsd here

                            mappingFileDirectory=<ATG>\home\mappingFiles     # place mapping files you are using here

                             

                            /atg/dynamo/service/loader/FileSystemMonitorScheduler

                            enabled=true     # for using the provided scheduler

                             

                            In case you are importing on versioned schema then you will also have to enble createProcesses and createProjects in /atg/epub/file/VersionedLoaderEventListener.

                             

                            Hope it helps you.

                            • 11. Re: RL Client clarification
                              ankV

                              You are absolutely correct Nitin. But, We need to watch / consider one thing here which is loading the content where it is in transformed XML based on the given XSL into ATG database where I am getting the NULL values. Please find below.

                               

                              <?xml version="1.0" encoding="UTF-8" standalone="no"?>

                              <sku xmlns="http://www.atg.com/ns/skuMapping/ProductCatalog/sku" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="sku">

                              <sku,id/>

                              <sku.displayName/>

                              <sku.description/>

                              <sku.displayNameDefault/>

                              </sku>

                               

                              Exception :

                               

                              ERROR [LoaderManager] handleException called for LoaderException: nested exception class: java.lang.IllegalArgumentException message: Attempt to set property named displayName (ProductCatalog:sku:sku230005)  with value = null (class=null).  A null value is not allowed for a required property.

                               

                              I am having one doubt here whether the OOTB components will do loading for the newly transformed XML on flow or at a time.

                              • 12. Re: RL Client clarification
                                Nitin Khare

                                You are absolutely correct Nitin. But, We need to watch / consider one thing here which is loading the content where it is in transformed XML based on the given XSL into ATG database where I am getting the NULL values. Please find below.

                                 

                                <?xml version="1.0" encoding="UTF-8" standalone="no"?>

                                <sku xmlns="http://www.atg.com/ns/skuMapping/ProductCatalog/sku" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="sku">

                                <sku,id/>

                                <sku.displayName/>

                                <sku.description/>

                                <sku.displayNameDefault/>

                                </sku>

                                 

                                Exception :

                                 

                                ERROR [LoaderManager] handleException called for LoaderException: nested exception class: java.lang.IllegalArgumentException message: Attempt to set property named displayName (ProductCatalog:sku:sku230005)  with value = null (class=null).  A null value is not allowed for a required property.

                                Not sure why you are getting null values in the transformed XML. For me it worked fine. Just ensure that if you do any changes in the XSL then you might have to restart your server as I think ATG inrternally creates a template for the transformation which does not get updated immediately if you make changes in the input XSL. To check the complete flow you can temporarily turn off the enforceRequiredProperties property on the involved repository component but I am not convinced about the null values part in the transformed XML. If you are still getting this error then post your complete input xml, xsl, mapping file and the schema you are using I will try looking into it.

                                 

                                I am having one doubt here whether the OOTB components will do loading for the newly transformed XML on flow or at a time.

                                Sorry but I didn't understand what exactly you are trying to ask here.

                                • 13. Re: RL Client clarification
                                  user4769191

                                  Hi Nitin

                                   

                                  I have a question on the usage of RL .

                                   

                                  Lets say i get the xml as required for the repository then there should not be a need for the XSL right?

                                  Am extending customcatalog by adding a new item-descriptor but extends media item-descriptor and now i want to create an XSD for the new schema and am trying to use generateXMLSchema script

                                  For example to create xsd for sku item-descriptor

                                  generateXMLSchema -s ATGPublishing -repository atg/commerce/catalog/ProductCatalog -itemDescriptor sku -outputDirectory .

                                   

                                  I am getting the following message

                                   

                                  The value specified for the repository argument: atg/commerce/catalog/ProductCatalog can not be resolved as a repository

                                   

                                  Let me know how to fix this.

                                   

                                  Thanks

                                  • 14. Re: RL Client clarification
                                    ClareR-Oracle

                                    The Nucleus component named ProductCatalog is defined in the DCS module.  You didn't mention any modules in your generateXMLSchema command, so the utility ran without DCS and without your extension module.  Try adding -m [your module name] to the command line.  Assuming your module's' manifest mentions DCS, or something that depends on DCS, on its ATG-Required line, that should get you past the unresolved component error.

                                    1 2 Previous Next