OAS - usage of custom Python scripts (creation, registration, usage in EXECUTE

Michal Zima

Description

One of the area, we would like to leverage in OAS in full extent, is the possibility of integration of OAS with Python environment, which is part of OAS suite (used internally for plenty of functionalities, mainly in DV - like Explain, Machine Learning Models, Advanced Analytics functions (Forecatst, Outliers etc...). What is especially of our interest is the possibility to create/write our own custom Python scripts, register  them into "platform"  and use it for creating own calculation (via EXECUTE_SCRIPT BI Server function) either in  OBIEE metadata or in Answers/DV calculations.

But here I have completely stucked here - since there is very little information about the process/steps/standards , which needs to be followed (nothing at all in standard OAS documentation, very little on public resources - just couple of youtube videos - some of them mentioned in Idea: https://cloudcustomerconnect.oracle.com/posts/a052ce91c3, but those dealing with very simple scenarios for Python script used in DV data flows). So I consider this as a "gap" from Oracle and I would appreciate, if this gap is filled (the sooner the better).

I have also created SR in Oracle support for this "lack of resources" issue :SR 3-25358627501 : Usage of custom python scripts in OAS (EVALUTE_SCRIPT etc..) - missing documentation/resources - but so far without any valuable feedback.

I am particularly trying to find answers/resources for following topics:

" What are the rules/principles for creation of custom python code/scrips, to be callable via EVALUTE_SCRIPT ( I know, because I was a little big digging - but it is trial-error approach - that there are some scripts in ??<<OAS Home>>/bi/bifoundation/advanced_analytics/python_scripts/obipy/examples - for example SayHello.py script, which can be served as a kind of example, but this

o what is exact "syntax" of final XML form for this script, which needs to be registered in script repository (<<OAS HOME>>/bi/bifoundation/advanced_analytics/script_repository)

o detailed explanation of arguments, which needs to be passed to Python script (and needs to be put into XML definition of script)

" What is the method of registering custom python script in script repository (via DV UI - which is uploaded somewhere and registered for user, who has uploaded the script or is it better to uploading XML definition into <<OAS HOME>>/bi/bifoundation/advanced_analytics/script_repository - thus the storage for Python script - which is then referenced in EVALUATE_SCRIPT call: EVALUATE_SCRIPT('filerepo://<<Python script>>.xml'))

" How to eventually  install additional packages into Python installation used by OAS (which is I guess, placed in <<OAS HOME>>/bi/modules/oracle.bi.dvml - but this is not documented at all) - if those are not included within this Python installation needed for custom python script ? Is this possible (and supported) 

So those are just basic stuff, which would be very handy to have somewhere described/documented to be able to leverage this feature/option available in OAS (I know, in OBIEE 12c it was about R language, in OAS Python has overruled, which for good reason).

I would be very glad, if Oracle  will cover this "grey area" in terms of documentation/knowledge sharing.

Use Case and Business Need

Beeing able to leverage integration of Python language in OAS at the full extent - creation/registration of custom Python script (callable via EXECUTE_SCRIPT BI Server function) and use them for extending predefined BI Server function. 

Original Idea Number: 2f449ffc32

Shriram Muthukrishnan

Hi Michael,

Here are my answers to your questions.

1) What is the syntax for final XML ?

This involves a few mandatory XML tags. These include  

• <outputs> - you need to predefine the name of the output columns and its data type 
• <options> - parameters that you wish to pass to the script
• <scriptname> , <scriptlabel> - giving a name and label to identify the custom script
• <target> - "python" will be the value since we need to run the script against the Python engine
• <type> - 'execute_script' 
• <scriptdescription> - to give a description on what the script is supposed to do
• <scriptcontent> - the tag which holds the Python code right from import statements to return statements

We have hosted couple of examples in Oracle Analytics Library (https://www.oracle.com/in/business-analytics/data-visualization/examples.html) on term frequency analysis and unpivoting. Looking into the details of the script will be helpful

2) Detailed explanation of arguments that needs to be passed.

Consider the following XML code

        <option>
            <name>topn</name>
            <displayName>topn</displayName>
            <value>30</value>
            <required>false</required>
            <ui-config />
        </option>

This is used to get an input from the user as to how many terms and its corresponding frequency the script show after execution. The arguments are taken in <option> tag with name, displayname, value, required and ui-config tags. These tags are common for any parameters

3) what is the method for registering custom scripts in OAS?

The most easiest way is to via the UI. Choose the option Create -> Custom Script -> upload your XML. This should show a preview of what is the inputs/outputs/arguments.

Click on OK to upload the script and you should be done.

4) Install additional packages?

The Python engine comes with a set of the most commonly used packages. In case you want to install other packages, you can follow this blog on how to do it. It gives a step by step tutorial http://oracledataviz.blogspot.com/2020/07/custom-scripts-in-oracle-analytics.html

Finally , we are working on enabling the custom script option in OAC as well. It is being currently worked upon by our development teams and should be released in a few OAC releases.

Let me know if you hit any issues.

Thanks,

Shriram 

 

Michal Zima

Hi Shiram,

thanks for you answer, I appreciate it, but I consider this as a really "very first kickoff". What you are describing is fine for some small playing and small POC, but I am missing serious and enterprise wide documentation/knowledge base - your post simply raises another questions :

1) Your description of XML format does not seems to be complete - what about some "inputs" ? I assume, there must be also <inputs> tag, where I define inputs (source variables) to Python scripts, to be passed from "outside" - or is it handled solely within <options> tag ?

2) You described registration of Python script via UI (I understand in OAC you don't have any other option), but when you upload Python script via UI, you have to take care also for permission to this "object" (if somebody else except the "creator" - person, who uploaded script - wants to use it).  So my question again leads to other form of "registration" of Python script on "on-premise" OAS environment - thus uploading script into cript repository folder on the server (<<OAS HOME>>/bi/bifoundation/advanced_analytics/script_repository) . Is this method supported on OAS or should we solely use registration via DV UI (and assigning permissions to uploaded script) ?

3) For installation of additional Python packages, you have pointed meto blog, which describes the process for simple Oracle Analytics Desktop (Windows) environment. But we want to use it on our "enterprise" OAS platform, running on Linux and thus the process should be described for this deployment.

As I wrote, I am considering you response as a kind of "first kickoff" (and really thanks for it), but to be able to use it seriously, it needs to be described in very much structured abd deatiled way (I am sure, you are aware of it).

Anyway I will start to "experiment" with Python script on our OAS "sandbox" and will hope, that in the meantime, you will fill this big documentation/knowledge gap .

Kind regards

 Michal

DWelch

I also agree that there is a huge documentation gap. Given a non opensource piece of software, the company that created it should provide syntactical documentation to help get started with creating scripts because anything that accepts XML will be looking for certain tags. I shouldn't have to piece meal my XML tags by looking at the built-in scripts and comparing them to their respective python scripts. For any enterprise use case this a terrible bottleneck in efficiency.