Forum Stats

  • 3,826,865 Users
  • 2,260,714 Discussions
  • 7,897,108 Comments

Discussions

How to check if service is up and running or instaled ?

FredericoSB
FredericoSB Member Posts: 419
edited Jul 9, 2014 5:47AM in Endeca Information Discovery

Hello,

I've been having a problem with the performance of my modify records component and i read that it uses the Data Ingest Web Service (DIWS) to modify the assignments.

How do i know if this service is up or if it is running without problems ?

I noticed that on this component, i have a speed rate of 1470 records updated every 30 seconds and this is a terrible performance..

Any ideas regarding the status of this service or how to check if it has any problems?

I'm using version 3.1 of Endeca.

Regards and thanks everyone.

Answers

  • JM-Oracle
    JM-Oracle Member Posts: 198

    Hello Frederico,

    Here is the doc about Data Ingest Web Service: http://docs.oracle.com/cd/E40518_01/server.761/es_data_loading/toc.htm#Overview%20of%20the%20Data%20Ingest%20Web%20Service

    You can access it by:

    http://localhost:<port>/endeca-server/ws/ingest/<dataDomain>?wsdl

    where the localhost and port are the host and port of the running Oracle Endeca Server, endeca-server is the default context root for the Endeca Server Java application running in the WebLogic Server, and dataDomain is the name of the Endeca data domain.

    Here is also information about this web service's logging:

    http://docs.oracle.com/cd/E40518_01/server.761/es_data_loading/toc.htm#Data%20Ingest%20logging

    As for performance of modifying records, please see this topic in the Endeca Server Developer's Guide. Some changes to the records are associated with performance costs associated with a partial or complete re-indexing of loaded records:

    http://docs.oracle.com/cd/E40518_01/server.761/es_dev/toc.htm#Updates%20to%20schema%20and%20configuration

    Hope this helps,

    Julia

    JM-Oracle
  • FredericoSB
    FredericoSB Member Posts: 419

    Hello Julia, thank you so much for your reply.

    Actually i had the loading guide to check the Data Ingest Web Service and i've tried to access the links they provide. It returns this:

    This XML file does not appear to have any style information associated with it. The document tree is shown below. (And it shows the document tree).

    I found the last part of your response very interesting but i don't know where i can find where my schema is and how can i see it and modify it. I just have a simple graph that tries to replace my values and it takes forever to do so. I've tried changing the values, even changing the data-domain with endeca-cmd but i always get the same performance.

    Any idea on how can i change the schema to see if I'm using any of the proprieties referred in the last document ?

    Regards,

    Frederico.

  • JM-Oracle
    JM-Oracle Member Posts: 198

    Hi Frederico,

    The URL below issues a request to the Endeca Server's Data Ingest Web Service and asks for its WSDL. You should do it from a browser, and it should know how to interpret and display the WSDL.

    http://localhost:<port>/endeca-server/ws/ingest/<dataDomain>?wsdl

    AS for the performance and modifications that you are doing, it does not look like you are modifying the schema, which are items in category 1 from the topic "Updates to schema and configuration". It looks like you are not doing that, and so you can ignore this link). Instead,

    your graph in Integrator replaces values. It may take a long time, if there are many attributes whose assignment values on records you want to change.

    Here is info on what the Data Ingest Web Service does, when replacing assignments on records:

    http://docs.oracle.com/cd/E40518_01/server.761/es_data_loading/toc.htm#Replacing%20record%20assignments

    Perhaps, you can use EQL too, to limit the number of records affected by the request in your graph in Integrator?

    Here is an equivalent info on how to use EQL in Integrator (and not directly in the web service request as shown in the link above), to select records for modifcation:

    http://docs.oracle.com/cd/E40518_01/integrator.311/integratoretl_users/toc.htm#Modify%20Records%20input

    Here are some additional examples of identifying records with EQL:

    http://docs.oracle.com/cd/E40518_01/server.761/es_data_loading/toc.htm#Identifying%20records%20with%20EQL

    Julia

    JM-Oracle
  • FredericoSB
    FredericoSB Member Posts: 419

    Hello Julia,

    When i go to display WSDL using that link, i get the document tree information as i referred in the previous post. Don't know if I'm accessing the correct URL but i used the information of the server and i always get the same type of structure and document as response from the URL.

    The purpose of such graph is to update 2 colums in all of my data, and such columns have value null for all the records. The idea is to run a Base Graph where the base attributes are stored in the data domain with Bulk Add/Replace Records and then to have different graphs that modify the records to update such columns.

    For example, we place 3.2 M rows, each one with 100 columns (for example) in the Data Domain with the Base Graph and then i run the Graph to modify 2 columns of those 100 columns using the Primary Key, so i have a modify records component with a SQL Query component coming in port 0, that returns 3.2 M rows with the primary key and 2 more columns with the updated values to replace in the Data Domain.

    The query has a good performance but the Modify Records don't. That is my problem, i tried to fix this but i haven't got anywhere.

    I even tried to set the EQL like: "PRIMARYKEY"=$input.PrimaryKey but it does the same, since i need to update all the records.

    Don't know if i explained well enough for you to understand why I'm doing such a thing but maybe you can advice me now that you know the purpose of my graph.

    Regards and many thanks for your help!

  • JM-Oracle
    JM-Oracle Member Posts: 198

    RE: not being able to see the WSDL, this is the problem with your browser. Or, possibly, with the secure not secure installation. Did you install the ES securely and need to use https instead of http? Does your browser have access to the secure installation?

    Alternatively, if you install soapUI tool (free), you can view the WSDL there, and also use this tool to check the health of the web service and see how queries are running directly.

    RE: your graph, it does sound like you are processing too many records at once. I am not an expert on Integrator and on how to build graphs more efficiently. Perhaps, a better approach is to reingest the records with values that are not null?

    Please see if anyone else from the list can reply to you regarding Integrator and efficient use of its graphs. Perpahs posting another question with Integrator in the subject will help get those replies.

  • FredericoSB
    FredericoSB Member Posts: 419

    Wasn't me that made the installation of Endeca. Can't give you much information about that since I actually don't know much about it.

    Regarding the graph, i limited the query to get less data from the query at once but the performance maintains regarding the number of records per second.

    It is a simple graph, don't know if it can be sub-divided in parts or phases and oracle guides advice developers to build such graphs in this way.

    Many thanks for your answers Julia and I'm going to wait for someone that can help me with the Integrator part. Still, you help me a lot.

    Actually, regarding the subject of the WSDL, whenever i tried to run the sample graphs (the ones with the attribute loading and stuff) i get an error regarding the webservice, it says it can access to the URL and if we try to access the URL it gives a 404 HTTP error.

    Regards and thanks!!

  • JM-Oracle
    JM-Oracle Member Posts: 198

    It is likely a hostname resolution problem, if you cannot access the web service when running the graphs.

    With this in mind, I recommend:

    - identify how the ES was installed, secure or not.

    - identify how you are connecting to the ES host machine and make sure this connection happens, from the machine running Integrator.http://docs.oracle.com/cd/E40518_01/server.761/es_admin/toc.htm#Hostname%20resolution%20in%20the%20Endeca%20Server

    - if you can connect and the graphs can run, look into performance.

    I suggest you post another question with regards to optimizing Integrator graphs.

    JM-Oracle
  • FredericoSB
    FredericoSB Member Posts: 419

    Hello Julia,

    First, thanks for the answers to fix my problem, I hope that it will do to someone that has that kind of problems.

    I've looked at the etc/hosts file and i've seen that the mapping from the machine-specific IP address to the full name of the machine is not on the first entry but I don't know if i can change that.

    The graphs run, that is not the problem, and i think neither is the performance of the components in the Graph but the performance of the Data Ingest Web Service while at using in the Modify Records Component. That is the problem of my graph.

    I am going to look at it again and open a new topic regarding the performance of the Data Ingest Web Service in the Modify Records Component.

    Many thanks for the help and have a good day!

    Regards,

    Frederico.

This discussion has been closed.