Skip navigation

Thanks to the OMC team I received my own OMC trial environments to set up some experiments. Looking through the OMC I saw some familiar components such as synthetic tests, and here and there some components that are used in  Oracle RUEI.

The possibilities are huge in OMC, which I will discuss in a later stage, but something I wanted to try out was if  I could create a mechanism to detect if a hackers collective was trying to break into a web applications by using some sort of a password attack.

 

My ingredients:

  • An Oracle Java Cloud Service containig a WebLogic 12c domain, hosting Web applications
  • An Oracle Management Cloud subscription, with the following components:
    • Application Performance management
    • Log Analytics
    • IT Analytics
    • Infrastructure Monitoring

 

Setup the basic needs

Before you can use the OMC some basic steps need to be done. These steps contain:

  • Install the APM agent
  • Install the Cloud agent
  • Enable and register the agents on my JCS environment to the OMC

 

Install the APM Agent

Of course, there is no agent software package, so first of all the software needs to be downloaded. The basic script can be downloaded from you OMC environment:

The script you can place on the servers of your JCS instance, in my case: the database, WebLogic and Oracle Traffic Director

 

After unzipped, the agent download can begin:

Cloud agent:

Java APM Agent:

The registration keys you can obtain in OMC, in the Administration TAB.

 

Then you enter the stage locations and install the agents

./AgentInstall.sh AGENT_TYPE=apm_java_as_agent AGENT_REGISTRATION_KEY=***************************** AGENT_BASE_DIR=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent  -staged
./AgentInstall.sh AGENT_TYPE=cloud_agent AGENT_REGISTRATION_KEY=************************* AGENT_BASE_DIR=/u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent  -staged

 

Adding the entities

Oracle provides JSON files for every type of environment which you can use to add your environment specifics to OMC, my example for JCS:

{
    "entities":
[
{
        "name":"QJCS01_server_1",
        "type":"omc_weblogic_j2eeserver",
        "displayName":"QJCS01 Managed Server 1 ",
        "timezoneRegion":"CET",
        "properties":{
                "host_name":
                        {"displayName":"Weblogic Host","value":"qjcs01-wls-1.compute-gse00003036.oraclecloud.internal"},
                "domain_home":
                        {"displayName":"Domain Home","value":"/u01/data/domains/QJCS01_domain"},
                "listen_port":
                        {"displayName":"Listen Port","value":"9073"},
                "listen_port_enabled":
                        {"displayName":"Listen Port Enabled","value":"true"},
                "ssl_listen_port":
                        {"displayName":"SSL Listen Port","value":"9074"},
"server_names":
{"displayName":"Server Names","value":"QJCS01_server_1"}
        },
        "associations":[
                { "assocType":"omc_monitored_by",
                  "sourceEntityName":"QJCS01_d_server_1",
                  "sourceEntityType":"omc_weblogic_j2eeserver",
                  "destEntityName":"QJCS01_domain",
                  "destEntityType":"omc_weblogic_domain"}
        ]
}
]

Together with a JSON credential file you can add all to OMC:

u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/agent_inst/bin/omcli add_entity agent /u01/app/oracle/tools/paas/state/homes/oracle/omc_cloud_agent/my_entities/qjcs01_domain.json -credential_file cred.json

 

I repeated these steps for my Database and Traffic Director, using their specific JSON files.

 

After adding the entities, you need to provision the APM agent using the script from your APM stage directory:

./ProvisionApmJavaAsAgent.sh -d /u01/data/domains/QJCS01_domain -no-wallet

 

And add  the APM jars to the domain, in the startWebLogic.sh( and restart the WebLogic domain)

 

JAVA_OPTIONS="${JAVA_OPTIONS} -javaagent:${DOMAIN_HOME}/apmagent/lib/system/ApmAgentInstrumentation.jar"
SAVE_JAVA_OPTIONS="${JAVA_OPTIONS}"

 

If all goes OK, you can see your agents being registered in OMC:

 

Now the basic steps are finished. As you click through the OMC, loads of information is already generated from your JCS instance

 

Log Analytics - detect a pattern

 

Now a simple use case: I wanted to discover if users try either unauthenticated(HTTP 401) or unauthorized(HTTP403) access a webapplication. I deployed a simple web application, and some users with different roles, to be able to test with it.

Some users had more permissions than others, so I could test between them.

Second, I wanted a huge load of performing these actions:

  • Accessing the webpage, try to login and do some action ( legal or illegal ).
  • Or try to login with a wrong password

 

For this I created a simple JMeter script to access the webpage and login, and the action within the session, which was an task to close an office, which was only permitted with someone with the managers role

 

I let this script run continuously, to generate the data I needed

 

 

Using  log analytics

A first step to make use of log analytics is that I analyzed the access logs, which gave a clear view of the loads of HTTP 401 and 403 errors.

Now these can happen on every website, and there should be nothing to worry about, but in this case,  a large volume of these errors passed, so this cannot be a mistake or a human error,

I clicked on the log analytics, selected the WebLogic domain which runs in the cloud, and selected in the pie chart the access logs

Then, In the left tab, the field Security Result

 

 

Note that denied count is very high. Next step was to save this search, and very cool was that I could create an alert out of it.

 

And I recieved a mailt with this specific alert, and one at the time the JMeter test had stopped, as that the alert had been cleared

 

Now this is a very first basic step I used OMC to detect hostile actions, so next time I will dive more deeper into all the great features!

This years Community organized by Oracle for it's partners took place in Split, Croatia, a very nice Mediterranean area, which we encountered during the city tour on Tuesday evening.

However, we did not came just for the fun, but to meet and greet other partners, share and absorb knowledge which is evident for companies to serve their customers, explore new technologies and methods, and have a sneak peak in another partners " kitchen", in a week of 5 days program: 1-3 General sessions, 4-5 Handson Workshops, Partner Awards and some networking events. Partners attended from all over the world: EMEA, US, Latin America, Down Under

 

The forum, formerly known as Oracle Fusion Middleware Partner forum has been “ lifted and shifted “ to the Cloud – PaaS the last years, so the focus of the content during this week was all about Oracle's Cloud products, but someone who pays good attention, can also extract the deeper content out of it, to what is useful for ones personally.

For me, I had a double role: of course attend and do knowledge sharing and absorbing for the company I'm into, but also tell something about the successful Oracle Process Cloud implementation my company did in 2016. A lot of these success stories, about developing technologies based on the cloud we're held on Monday during the so-called ACE Sessions.

 

As the forum already indicates, it was all about PaaS, and the presented content was done by VPs, directors and Product Managers from Oracle such as Vikas Anand, Robert Wunderlich and Jean-Marc Gottero. But also the partners had some interesting presentations, presenting about their solutions in all kinds of areas in the Oracle Cloud. The presentations handled about the following topics:

  • Agile DevOps
    • Handled about the DevOps Agility and methodology, and how the Oracle Developer Cloud fulfill a role in bringing  " DevOps"  uptp speed for a software company.
  • API APIPCS and API Management
    • A very interesting subject all about API and the Cloud Services, where all the benefits of APIs and management were brought, such as better security and protection, monitoring, discovery, the new Apiary platform. Also an overview of the existing and coming features; the API firewall caught my attention, and also the monitoring capacities, where I can see great capabilities in combination with  OMC (author sidenote).
  • ICS
    • A session about best practices around implementing integration patterns using ICS, with a clear mind about when to use ICS, and to see that ICS has an overlap with many other PaaS platforms, which is to be expected from an integration point of view Interesting was the topic about exposing databases using REST
  • IoT
    • Apart from the role IoT is already playing nowadays, it is also interesting to see how the combination and integration with cloud fits in, such as the Asset Monitoring Cloud Service where connected devices can be monitored by their location, performance, health and utilization. A very good use case about a companies production floor where every asset is being watched so that any disturbance in the production line can be detected in an early stage.
  • JCS
    • "WebLogic Server in the cloud" has become more mature with all the features you also use when running on premise, but a  lot of work is aready done for you. An overview of the tooling, the DevOps integration and methodologies are embraced as expected by the JCS. Important to know that a lot of the on premise multitenancy features are also available in JCS, and tools and methods to transfer your on premise WebLogic server to the cloud using the DPCT tool.
  • OMC
    • The Oracle Management Cloud has a lot of cool features such as Application Performance monitoring, Log Analytics for trouble shooting, performance analysis, and some other great features like end customer experiences by performing synthetic tests which can absorbe recorded actions users do ( these are some great features from RUEI)
  • PCS
    • PCS - Qualogys sucessfull implementation in the Netherlands. In thr months before I had some discussions with Jurgen and the PCS team but finally we decided I would be the spokesmen to tell  a short overview of what we have been doing

Our customer is a lower/ midsize municipality in the Netherlands and had a lack of insight in personnel manpower, and because of this the onboarding of new personell got stuck. New personnel got  registered in  different systems using different methods, even on an Excel spreadsheet. The Management information was very poor, no relationship between budgets and manpower, and there we're differences between systems regarding financial budgets and accountability.

 

To overcome this we build this solution using the Oracle Process Cloud, working togehther with Qualogy's Forceview HRM solution in the cloud

The PCS team constructed one process to:

  • Register all staff data into one system
  • One uniform way of informing the organization when there are changes
  • Good  quality and coherent management information

 

And all done by:

  • Having no delay in rolling it out
  • Having direct contact with the business about wishes and requirements
  • Using Oracle Process Cloud Service and Qualogy ’s Forceview

 

Screenshot of the PCS environment:

 

 

 

I also told this in a video interview which will soon be published on Oracle's Youtube channel, I keep you updated about that!

In this blog post I will setup a Java Cloud Service, which is Oracles PAAS service for delivering WebLogic Application Server, . Setting up a Java Cloud Service infrastructure means the following:

  • Setup a Database Cloud Service
  • Setup a Java Cloud Service including a Loadbalance feature. This will result in a basic Oracle Traffic Domain and a WebLogic Domain for your applications to be deployed on.

 

 

 

Setup a Java Cloud Service

 

To setup  a Java Cloud Service, you will setup a complete basic WebLogic Application Server domain. For this you need to set up:

 

Database Cloud Service

 

In my cloud console I first followed the process of setting up a database Cloud Service

 

  • Clicking on the Database Cloud Service Console, create a service

  • Next,you fill in some requested details

  • Last screen some advanced details. Don't forget to tick create Storage Container

    • Service Name: Service Name of the database
    • Key: You can generate a keypair in the console, or if you have done so, select you public key
    • Backup Destination: Should be both Cloud and Local for Java Cloud Service
    • Cloud Storage Container: Must be in the right naming format: Storage-<your identity domain/<storage container name ( name is free to choose>)
    • PDB leave it to the default

 

After a while, you have a database up and running .

Accessing the database can be done in a view ways:

- Through   DB/EM Console; just the familiar way of doing it

To access these console, you need to enable the predefined access rules

 

The Java Cloud Service

Second, the Java Cloud can be setup, which is a relative easy part. From your Cloud Dashboard you can click the  Java Cloud Service Icon, open the Service console

and Create Service.

 

 

 

Also inhere a very straight forward process:

Finally, when all details are filled in:

Think about accessing consoles, otherwise they're inaccessible, but later we will have to secure them more with OTD

Also enable a local loadbalancer, it will cost more but gives more control

After creation, you get 2 domains:

  • Your application domain
  • The Oracle Traffic Director Domain

 

Here's where the basic setup ends. Afterwards you can access your environments as if it's in you companies infrastructure. I deployed an MDB testapplication which generates lots of JMS Messages and CPU cycles.

 

Later on, in part 2 I will dive more into the multitenancy story, to share resources and do isolation.

Preface and Scope

A lot of customers run their business applications on Oracle Fusion Middleware technology. This can be for example a Web Portal which consists of different combined Oracle Fusion Middleware components. These environments produce lots of metadata content, diagnostic information and logging. To avoid bad performant environments by carrying all this kind of data, customers need to have a clear vision and strategy of how they will handle housekeeping, and how to implement this housekeeping in the different Oracle FMW components, each on their own level.

To ensure a stable and good performant environment it is important to be in control and have a proper housekeeping plan.

 

So, what does housekeeping means in relation to OFMW components?

  • Database related:
    • Purging/cleaning ancient OFWM Instance, process and meta data from the OFMW repositories
  • Log related, text or XML format
    • Logfile rotation, retention and cleaning
  • Diagnostic framework related
    • Cleaning diagnostic data of all OFMW environments
  • Implement housekeeping mechanisms on these various levels.
  • Monitor housekeeping in light of Capacity planning and growth trend.

So it is very important that housekeeping strategy and guidelines must be defined because of the following:

  • OFMW products rely strong on their metadata; it must be get information very fast to steer certain processes, but also generates all kinds of historic data which becomes ballast after a period.

To guarantee performance this data must not grow unnecessarily so has to be cleaned

  • Logs and diagnostic data might be useful for some period for analysis and historic perspective but need to be removed from the OFMW and stored somewhere else (Oracle Enterprise Manager Cloud control, log server ).
  • It depends on the stage of the environment to determine how long metadata, process data and logging should be kept; a development or test environment might require a longer retention period to study the outcome of tests or solve bugs in the code.

 

Housekeeping strategy Oracle Fusion Middleware components

Oracle Fusion Middleware generic strategies

This strategy is applicable for every OFMW base environment, either it’s an Oracle Service Bus or WebCenter Portal environment, they all rely on their underlying framework, namely:

  • Java Server Output logging
  • WebLogic Logging Framework
  • WebLogic Diagnostics Framework(WLDF)
  • Oracle Diagnostics Logging  Framework(ODL)
  • Dynamic Monitoring System(DMS)

Every framework need to have a guideline and strategy which covers:

  • Level of logging /diagnostics
    • Depending of the type of environment, the appropriate level must be set to meet the requirements of how deep diagnosis must be able to be done.
  • Retention and archiving guidelines
    • Every company or institute has its own internal guidelines of archiving data, sometimes regulated by law., so retention and archiving must be in line with the Companies policy.
  • Cleaning methodologies and mechanisms.

To be more specific, housekeeping handles:

  • WebLogic Server and domain logging (produced by Admin and managed servers).
  • Oracle Diagnostics Logging (Extended logging for specific Fusion Middleware components.
  • WebLogic Diagnostics Framework.
  • Node Manager Logging.
  • Oracle HTTP access en server logging.

 

WebLogic Server and Domain Logging

A typical WebLogic domain produces a vast amount of logging from all what’s happening inside.Before setting a proper housekeeping, the following needs to be determined

  • Which Rotation frequencies WebLogic Server and Domain Logs
  • Maximum per Logfile Size
  • Logfile retention
  • Levels of Logging(INFO, NOTICE, etc). Only relevant information is needed.

These values differ per stage of environment. Except for production, other stages may need to keep logfiles for a longer period to do analysis, or having another level to produce more logging.

 

Table  Logging Retention

 

Component

Platform component

Stage

Size per log file & Retain Limit

Retention in days

Remarks

WLS Server and domain logging

WebLogic Server

ALL

10MB -100

30

Log level: INFO

WLS Access logging

WebLogic Server

ALL

-

-

Keep it off; will be done in OHS

GC Logging

Java VM

DEV/TEST/ACC

10MB

14

Specific information about JVM Garbage collect times

GC Logging

Java VM

PRD

10MB

7

Specific information about JVM Garbage collect times

STDERR and STDOUT

Java-O/S output

ALL

10MB

30

Use logrotate from Linux

Oracle Diagnostics Logging

OFMW infrastructure

DEV/TEST/ACC

10MB

30

Rotation Policy:

Size based

Loglevel:INFO

 

 

 

 

 

 

Oracle Diagnostics Logging

OFMW infrastructure

PRD

10MB

30

Rotation Policy:

Time and Size based

Loglevel:INFO

Oracle HTTP Server Logs

HTTP

DEV/TEST/ACC

 

31

For OHS, use format ODL, use odl_logration, rotation_type T

Oracle HTTP Server Logs

HTTP

PRD

 

7

For OHS, use format ODL, use odl_logration, rotation_type T

Nodemanager

WebLogic Server

ALL

See remarks

See remarks

LogCount 1

LogLimit 100

LogLevel INFO

FileCount  30

FileMinSize 500

RotationType SIZE

NumberOfFilesLimited true

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Explanations:

WebLogic

If you use Node Manager to start a Managed Server, messages that would otherwise be output to stdout or stderr are written to a single log file for that server instance, SERVER_NAME.out; for example, DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.out, where DOMAIN_NAME is the name of the directory in which you located the domain and SERVER_NAME is the name of the server. WebLogic does not rotate STDERR and STDOUT files. Use the Linux logrotate.conf

Every WebLogic Server writes access logs, however in many scenarios there will be some webserver in front acting as a reverse proxy, such as Apache or Oracle HTTP Server, or even Oracle’s full loadbalancer solution Oracle Traffic Director. In these cases, switch access logging on WebLogic level off and leave thata to the webserver. This will minimize extra overhead. If you still want to turn on accesslogging for deeper diagnose, switch it off for at least the AdminServer.

 

 

Oracle Diagnostics Logging

Oracle Diagnostics Logging handles logging different using:

1. Runtime logging levels more product specific logging (SOA, OSB, WCP)

2. Log handlers handle rotation frequency and log file size:

odl-handler

trace handlers

 

Nodemanager

For operational management logging needs to be enabled by setting the Logfile rotation NativeVersionEnabled=false.

Oracle HTTP Logging

 

 

In case your company uses Oracle HTTP or Apache Server writes , you may need to consider in housekeeping the server logfiles.

These files can be housekeep by using one of the two retention mechanisms:

  • odl_rotatelogs which comes default with the installation of OHS
  • rotatelogs, the Apache Version

Use the rotation_type T (time-based), and set the rotation_policy parameter to:

 

frequency(in sec) retentionTime(in sec) startTime(in YYYY-MM-DDThh:mm:ss)

Example

(F)43200:(RT)604800 2017-01-06T10:53:29, the error log is rotated every 43200 seconds (12 hours), rotated log files are retained for maximum of 604800 seconds (7 days)

For details see Table  Logging Retention

 

 

WebLogic Diagnostic framework

 

 

There is a WLDF module for Oracle Fusion Middleware, called the Module-FMWDFW, installed consisting the following components:

Predefined Watches and Diagnostic Dumps to detect, diagnose and resolve problems Use WLS and SOA MBeans and DMS Metrics.

Detect critical failures and collect diagnostic dumps containing relevant diagnostic information like logs, metrics, server images,

ADR (Automatic Diagnostic Repository)for creating incidents WLDF for OFMW is has a default record volume staat on low

 

This module has a so-called record volume. Based on type of stage this level needs to be increased or lowered.

 

Stage

Record Volume

Harvester Sample period

Capture Diagnostic Image

WLDF Data Archive

WLDF EventsDataRetirementPolicy

HarvestedDataRetirementPolicy

DEV

Medium

5000

Yes

File

retirement-time 10

retirement-period 24         retirement-age 168

TEST

Medium

5000

Yes

File

retirement-time 10

retirement-period 24         retirement-age 168

ACC

Medium

5000

Yes

File

retirement-time 10

retirement-period 24         retirement-age 168

PRD

Low

5000

Yes

File

retirement-time 10

retirement-period 24         retirement-age 72

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Oracle Fusion Middleware specific housekeeping

Oracle Service Bus Alert data

Oracle Service Bus has to manage its own alert and diagnostics data. This data consists:

  • Pipeline Alerts (file based)
  • SLA Alerts ( file based )
  • Statistic reports (  File based )
  • Metrics Reports ( JMS Provider and database )

 

  • Oracle Service Bus
    • Alert data Pipeline:
      • OSB services generate pipeline data in every step of a routing or transformation in a service. Based on development, exceptions can be logged into alerts
      • SLA Alerts
        • Additional, individual Business Services can be instrumented with SLA Alerts to monitor performance and availability from OSB Business Services to external components (SOA or JCA to database, AQ or legacy)
      • OSB Metric data
        • OSB Monitoring framework keeps statistics of metrics of services
      • OSB Reporting data
        • Using the internal JMS reporter, developers can instrument services to write exceptions and other logging to a database

Now, all these metrics are not enabled by default, and the development party decides which tools to use. Admins need to configure the operational handling and housekeeping

 

Operational settings per service

 

Not every OSB service, either Proxy or Business Service needs to be monitored, but choose the most critical ones, you would like to monitor, those which are key services to all other operations. That can be 10 or maybe more, that’s upto your company to be decided.

 

Table OSB Service monitoring settings

 

Operational Settings

Default Value Service

State

Enabled

Monitoring

  1. Default Disabled à Per service to determine if it must be enabled

Aggregation Interval

10 minutes

SLA Alerts

Enabled

Pipeline Alerts

Enabled at Normal level or higher

Reports

Enabled at Normal level or higher

Logs

Enabled at Debug level or higher

Execution Tracing

Disabled

Message Tracing

Disabled

Offline Endpoint URIs

Disabled

Throttling State

Disabled

Maximum Concurrency

0

Throttling Queue

0

Message Expiration

0

 

 

 

For OSB a separate data retirement policy needs to be created, apart from the default one (see WebLogic Diagnostic framework for more information).

This specific OSB volume consists  pipeline and SLA alert data, and need regular housekeeping.

The values of this policy must be:

 

Retirement Age: 120 hrs. (PRD )  480 hrs (DEV/TEST/ACC)

Retirement Time: Midnight

Retirement Period: Every 24 hour

These parameter values are the same as the default WLDF data collector.

 

OSB Report data

Developers can use the OSB Reporting Provider to generate extra logging and metrics on specific services. This OSB metric data is being written a database, specifically in WLI_QS_REPORT_ATTRIBUTE, WLI_QS_REPORT_DATA tables.

Housekeeping can be done through:

  • Manual via OSB Console (not preferable)
  • EM Cloud control job or database job to delete the data, which is the most preferable to do.

 

Purge all data older than 1 month.

Stage

Retention

DEV/TEST/ACC

60 days

PRD

30 days

 

 

SOA Suite Housekeeping

SOA Suite keeps track of all process instance flow and tracking data, flow states and deployed artifact data and stores this in the so-called dehydration store, error hospital MDS (SOA and OWSM).

Before speaking of housekeeping, the following prerequisites need to be verified and set

 

  1. Audit levels per DEV-TEST-ACC-PRD
  2.    Check Capture Composite Instance State à on
  3.    Check Data Display Options According to:
    1. Instance fetching
    2. Display restriction

 

Table Audit Level

 

  Stage

Level

DEV

Development

TEST

Development

ACC

Production

PRD

Production

 

 

 

 

 

 

 

One very important thing keeping in mind while discussing SOA Suite housekeeping, is taking care of the SOA Suite Repository. So, a solid database growth strategy must be developed, in cooperation with the database admins. Keep in mind and discuss the following with your DBA:

  • Size/profile of the Database
  • Space and Resources usage and Database Performance
  • Growth Management
  • Space Management

A DBA has the knowledge and tools to do these things, but with strong input of an Oracle Fusion Middleware Administrator. See Database Profile / Size Suggestions, for recommended sizes.

 

The value of the audit level determines how many audit data is generated, so it is very important to have some strategy around this. In production environment, which can generate loads of message per day, it’s not advisable to set the Audit level too high but keep it to Production.

 

SOA Suite purging

Purging a SOA Suite database means: cleaning retired instance, process and metadata without influencing running processes. All this data is stored in tables in some database schema’s, where the most important one is the soainfra schema.

Auto Purge

The SOA Suite 12c installation comes with an auto purge option which can be enabled using the Fusion Middleware console. When enabled, the soa.delete_instances is scheduled as a database job.

To set up, use the following option:

 

Where to review the default auto purge settings; check the default schedule and other possibilities. Note the retention period of 7 days, this must be changed to the actual retention.

 

 

 

 

Purging must be implemented on 2 levels

  • SOA Instance Process data (CUBE, Composite, Mediator, Audit, DLV_MESSAGE)
  • MDS Deployment Labels

Now, purging must be calculated based on the growth of the database tables, so these need to be monitored.

The following instances with their states will be purged:

  • Completed
  • Faulted
  • Terminated by user
  • Stale
  • Unknown

Not purged will be composite instances that are in the following states:

 

  • Running (in-flight)
  • Suspended
  • Pending Recovery

 

One golden rule is:

Retention period = Longest Running Composite

 

But a good starting point is to see below table

 

 

 

Ways of purging

Purge levels and retentions

 

Purging must be implemented on 2 levels

  • SOA Instance Process data (CUBE, Composite, Mediator, Audit, DLV_MESSAGE)
  • MDS Deployment Labels

Now, purging must be calculated based on the growth of the database tables, so these need to be monitored.

The following instances with their states will be purged:

  • Completed
  • Faulted
  • Terminated by user
  • Stale
  • Unknown

Not purged will be composite instances that are in the following states:

 

  • Running (in-flight)
  • Suspended
  • Pending Recovery

 

One golden rule is:

Retention period = Longest Running Composite

 

But a good starting point is to see below table

Component

Retention and keep time

Stage

Remarks

Instance Tracking

60 days

Development, TEST

Composites, Cubes, Mediators, Workflow

Metadata

60 days

Development, TEST

OWSM metadata

Metatalabels

120 days

Development, TEST

MDS Deployment Artifacts ( WSDL’s, XSD’s)

Instance Tracking

60 days

ACC

Composites, Cubes, Mediators, Workflow

Metadata

60 days

ACC

OWSM metadata

Metatalabels

120 days

ACC

MDS Deployment Artifacts (WSDL’s, XSD’s)

Instance Tracking

30 days

PRD

Composites, Cubes, Mediators, Workflow

Metadata

30 days

PRD

OWSM metadata

Metatalabels

60 days

PRD

MDS Deployment Artifacts (WSDL’s, XSD’s)

 

 

 

 

 

So, monitoring composite running length is evident to come to a final strategy, although strategy must be evaluated every month. After a while of measuring and analysis a good view of what it needs to be van be developed, based on one monthly analysis, the real growth trend can be calculated based on this formula, but also keep in mind the Overall company policy on retention of data:

 

Daily-inflow-composite-space = (SOA Schema Size / Period)

 

Inflow data is all data that is being dehydrated to the FMW database which consist:

  • Instance Flow tracking data
  • Adapter report data
  • Error Hospital data
  • Fault alerts

The SOA schema size can be identified in SMALL, MEDIUM and LARGE as displayed in this table. A composite consists of different components such as BPEL, Mediator, Human Workflow. All these components are included in this inflow composite space.

 

 

Database Profile / Size Suggestions

Based om the Daily composite inflow calculation the following is applicable

 

Database Profile

Composite Space Persisted Daily

Minimum Retention of Space

Small

< 10 GB

< 100 GB

Medium

10-30 GB

100-300 GB

Large

> 30 GB

> 300GB

 

 

Above calculations are examples and based on the following reference documentation:

http://docs.oracle.com/middleware/12212/soasuite/administer/GUID-E285CE03-1BE9-4319-AB1D-E34441BF577C.htm#SOAAG97846

However, database must be monitored on real size values and this might influence the size and purging strategy. For example, a load stress test environment will generate loads of data. Based on this a real strategy for production can be predicted.

 

 

Purge Executions

Purge execution can be done in 2 ways. The auto-purge option uses the soa.delete_instances_auto procedure.

Purge ExecutionInitiating procedureProcedureWhen to use
Singlesoa.delete_instances_autosoa.delete_instancesSmall amount of instance deletion
Parallel soa.delete_instances_autosoa.delete_instancesLarge amount of instance deletion

 

 

 

Starting, Single is sufficient enough. However, if purge takes more time, the parallel is the one to be used. Note the following with parallel purging:

  • Must be done during non-business hours
  • For multiple executions, the database server must have enough CPU
  • Indexes must be dropped and re-added

 

 

 

 

2.3.2.2.3. Purge settings

Below table handles the preferred purge setting. Note that this is a starting point and needs to be evaluated as an administrator should always take care of managing database growth of FMW.

Purge setting

Value

Remarks

Frequency

Daily

Purge type is Single/Parallel

Retain

30 *)

Purge type is Single/Parallel

Maximum flows to purge

Use the default

Purge type is Single/Parallel

Batch Size

Use the default

Purge type is Single/Parallel

Degree of Parallel

Use the default

Purge type is Parallel

ignoreState

false

Mandatory false! **)

maxRuntime

Use the default

In case of many instance, increase

 

 

 

*) Retainment is a starting point

**) When set to true, all instances will be deleted, regardless of their state

 

The above table is applicable for PRD, DEV/TEST/ACC will have their own settings.

 

 

 

Some Hints and tips

 

 

For SOA Suite database maintenance, there are some hints to consider, such as:

  • For longer term Database Management, regarding to housekeeping, partitions must be created to accelerate data access and purging
  • Purging schedules are built into SOA infra layer
    • Schedules can be created and managed from FMW Console

 

  • Picking a SOA DB profile enables to get the performance features of Oracle DB for SOA related storage
  • Global hash feature helps with faster querying and helps with EM responsiveness and instance tracking performance
  • The global hash indexes avoid full table scans; this is usable after the purge and improves performance
  • Recreating of indexes periodically helps for better performance

As a follow up to my 2 minute Tech Tip https://youtu.be/14gEIXhCwbk taken by @BobRhubart about SOA Suite monitoring I'd like to give you some more context than that fuzzy performance of me during Oracle Open World 2016.

Monitoring your SOA Suite is a tough job because of the complexity of the environment. Usually Adminstrators stick to some CPU, memory, disk and basic WebLogic Monitoring, some have monitoring in place for some more advanced stuff such as JVM, JMS, JDBC, but when it comes to "Who monitors the Error Hospital on Business Faults?" or "Who is looking after SOA housekeeping?", than no one knows what i'm talking about.

 

And, in many cases, the necessary SOA Management pack hasn't been licensed, or switched on to make use of all the nice features you can get out of this pack. This pack transforms your local Fusion Middleware Console into an EM dashboard, can monitor the Error Hospital, trace on Instance flows and even monitor the Dehydration Store Performance the SOA Suite repository. So, who needs the local Fusion Middleware Console(EM)?

 

Impression of the screens in Enterprise Manager Cloud Control

However, if you are not licensed to use the SOA Suite management pack, than you are not permitted to use al these nice features.

 

As I told in the video, Oracle Enterprise Manager Cloud control has a feature called metric extensions. These are extensions if collected metrics which are delivered in a certain pack are not sufficient enough or maybe are not there at all. I did some research for a customer who used the SOA Suite but had no management pack for it.

Using the metric extension the customer was able to collect all the metric data from all the SOA environments and do the same stuff with as if it was a metric from a monitoring template. So setting thresholds, collection schemes, alert rules and notifications, and trending analysis also were possible.

The more difficult task in this case was that I had to gather and tweak  some useful SQL queries to get the information out of the SOA repository; in fact a lot of the truth about a SOA environment lies in it's database.

 

Way of working

 

I created a view extensions. Within EM, select Enterprise--> Monitoring --> Metric Extensions

 

Name Metric Extension ME$SOA_INFRA_MDS_COUNT; for having operational statistics about Instances Running, Closed, Stale, etc

 

I used the following query, as I'am not an SQL Guru, found it somewhere ( thanks to the creator ) to build my extensions

Next step is to create columns for displaying the values I would like to see, in this case the number if instances over the past 2 days and their state:

 

 

 

 

There was no need to have some alert for this, it was pure informational.

Next is to test uf the this gives the desired result:

When finished, you need to save the extension as a deployable target, publish it, and either deploy it to an individual target or attach it to a template.

 

The following metric extensions are used now:

ME$SOA_INFRA_BPEL_STATE

This monitors the BPEL states in the composite and alerts if it gets aborted ( need to set a date limit, todo action )

 

 

ME$SOA_INFRA_BUSS_FAULTS_NON_RECOVER

Monitors on non recoverable Business Faults

 

ME$SOA_INFRA_MED_STATE

Monitors om States of Mediators, same as BPEL States

ME$SOA_INFRA_RUNNING_INSTANCE_FAULTS

Monitors on open instances if faults occur

 

ME$SOA_REPOS_SIZE

Gives a daily report of the size and growth of the SOA Suite repository

 

 

There are lots more to think of so this will be extended with more useful information.

Here another short tech tip

Another issue I encountered is not yet identified at MOS, not in the context I encountered it. The error was written to the logs every minute:

<Error> <oracle.wsm.resources.policymanager> <WSM-02084> <Access denied. Permission "oracle.wsm.security.PolicyManagerPermission" is required to access the wsm policy manager "UsageTracker" method "recordUsage".>

 

The only MOS reports on this was partially applicable for my situation: Bug 23279489 : WSM-02084 ERROR WHILE STARTING WSM/SOA SERVER IN DOMAIN PARTITION SETUP

 

The SOA Suite Domain was setup using a separate SOA and separate WSM Cluster, however the MOS note speaks about a created partition to be deleted... however domain partition is not yet supported for SOA 12.2.1 so that surprised me.

 

So here's what brought me the solution.

  1. Login to Fusion Middleware Control, expand the hamburger menu and right click on the domain, select Security and the Application Policies
  2. Select the ws-pm application stripe and edit the policy for policy.Acessor

4. Click on Add and search for the resource UsageTracker and method record Usage, click continue

5. Grant Invoke and All permission

6. After done click ok and restart your domain.

 

Afterwards, the errors in the AdminServer and OWSM managed server logs are dissapeared.

Just a small blogpost about an issue I encountered while setting up during the rollout of a high available SOA Suite evironment version 12.2.1.

The domain consisted of:

- AdminServer

- 2 SOA Managed Servers in a cluster

- 2 OWSM Servers in a cluster.

 

While accessing the 12c Fusion Middleware Console as the WebLogic Administrator, click on the soa-infra dashboard gave me this view:

 

 

A second error that occurred was the an OWSM error in the logs, WSM-02084. I was mislead, thinking this was related. However, on MOS I found that view permission problems were identified as a bug in 12.2.1.

 

To overcome this issue, the following line must be  added to the JAVA_OPTIONS of the AdminServer. This can be done in the setDomainEnv.sh, or better option in 12c, the setUserOverrides.sh, which all our environments call from a cental location

 

-DEMAS_DISC_ADD_EXTERNAL_FARM=true

 

 

So in total the string looks like

# custom settings for each Server

if [ "${SERVER_NAME}" = "AdminServer" ]

then

   echo "Customizing USER_MEM_ARGS for SERVER_NAME ${SERVER_NAME}"

   export USER_MEM_ARGS="-Xms1g -Xmx2g"

   export JAVA_OPTIONS="$JAVA_OPTIONS -Djava.awt.headless=true -DEMAS_DISC_ADD_EXTERNAL_FARM=true"

fi

 

After restarting the AdminServer, the issue was gone

 

 

 

 

 

See more at https://support.oracle.com/epmos/faces/BugDisplay?parent=DOCUMENT&sourceId=2148752.1&id=22070511

Some of the old stuff from my previous whitepaper and some new insights about Oracle FMW on Exalogic. Lost to consider!

 

 

Exalogic as a converged engineered system has various layers to consider when we want to apply all the possiblle enhancements on every possible layer in the Oracle FMW Stack

 

In this whitepaper we will start with the following components:

  • WebLogic Server
  • Coherence
  • Database traffic ( JDBC)
  • Message Traffic ( HTTP, REST, SOAP,JMS)
  • Traffic Director

 

Area

Parameter

Setting

Remarks

WLS

Exalogic Enhancement

Tickbox enabled

Set in WLS console

WLS

I/O Enhancement

-Dweblogic.ScatteredReadsEnabled=true

-Dweblogic.GatheredWritesEnabled=true

-Dweblogic.replication.enableLazyDeserialization=true

Domain Env file

WLS

Exalogic Enhancement

# Enable Java Exalogic optimizations
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

Domain Env file

WLS

ListenAdress IPOIB

WLS AdminServer ListenAdress

WLS Console

WLS

CPU

  1. KernelMBean.AddWorkManagerThreadsByCpuCount mbean

FMW Console

WLS

WLS

Admin Channel

t3, http

WLS Console

WLS

Client Channels

t3, http

WLS Console

JDBC

 

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=<>)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=<DB Service>)))

JDBC URL

JDBC

JDBC optimization

  • oracle.jdbc.enableJavaNetFastPath=true

FMW Console

Coherence

WKA

-Dtangosol.coherence.wka1=(IPoIB adress localhost)

-Dtangosol.coherence.wka2=(IPoIB adress1)

-Dtangosol.coherence.localhost=(IPoIB adress2)



In-JVM coherence and separate Coherence cluster

SOA BPEL

QualityOfService

CacheEnabled

FMW Console

SOA Infra

SOA BPEL Coherence Dehydration

-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=<value>

-Dbpel.cache.cubeInstance.sizeLimit=<value>

-Dbpel.cache.invokeMessage.sizeLimit=<value>

-Dbpel.cache.deliveryMessage.sizeLimit=<value>

-Dbpel.cache.deliverySubscription.sizeLimit=<value>

Domain Env File

SOA Infra

SOA Coherence Optimizations

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa
-Dsoa.oracle.home=${SOA_ORACLE_HOME}
-Dsoa.instance.home=${DOMAIN_HOME}
-Dtangosol.coherence.log=jdk
-Dtangosol.coherence.transport.reliable=imb

-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl
-Dweblogic.transaction.blocking.commit=true
-Dweblogic.transaction.blocking.rollback=true
export EXTRA_JAVA_PROPERTIES

 

 

 

 

Domain Env File

JMS

Message Compression

StoreMessageCompressionEnabled=true

WLS Conole

JMS

Message Compression

PagingMessageCompressionEnabled

WLS Console

JMS

TLOG and persistency

JDBC Store ( Over IpoIB)

WLS Console

JMS

Lockless requests

  1. ServerMBean.useConcurrentQueueForRequestManager=true

FMW Console

JSP

JSP Factory caching

jsp_factory_caching=true

 

 

WebLogic Admin channel over Infiniband private net

 

The private InfiniBand fabric including WebLogic Clusters and Coherence clusters, requires a set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for cluster communication. These IP addresses are associated with the BOND0 interface.

 

Within Exalogic it is possible to use a private vnet that can be reserved for administrative purposes such as accessing the AdminServer and its consoles, internal cluster communication and so on.

This can be easily achieved by setting all listenadresses to the IP that is bound to Infiniband

The same we do for all managed servers and their listen adresses

Trouble is, that we can’t access the AdminServer url over IPOIB, For this we create separate channels

Also for the managed servers, which have clients access we create channels to connect to the outside world 

 

There are several channels to identify:

  • Several channels for EoIB interfaces, such as:
    • AdminServer Network Channel
    • HTTP and T3 Network Channels

These are needed for internal resources like AdminServers or managed servers internal communications to communicate with the outside world.

You must create two network channels for the Administration Server. The network channels are necessary for routing HTTP traffic and T3 traffic (TCP-based protocol used by WebLogic Server) coming in from an external data center via Ethernet over InfiniBand (EoIB). You must create the following network channels:

HTTP Client channel for AdminServer

  1. Log in the WLS console and Click Lock & Edit in the Change Center.
  2. In the left pane of the Console, expand Environment, and then Servers.
  3. In the Servers tab, click AdminServer(admin).
  4. Select Protocols → Channels. Click New.
  5. Enter AdminHTTPClient as the name of the new network channel and select http as the protocol, then click Next.
  6. Enter the following information in the Network Channel Addressing page:
    • Listen address: < the EOIB IP adress >This address is the floating IP assigned to the Administration Server using the BOND<n> interface.
    • Listen port: 7001
  7. Click Next, and select Enabled on the Network Channel Properties page. and Click Finish.
  8. Activate these changes,

 

T3 Client channels

Repeat the same steps as above but choose T3 as a protocol. Repeat these steps also for all managed servers

 

When started, you see the several interfaces which are listened on

Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.11.26:7001 for protocols iiop, t3, ldap, snmp, http.>

<Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "AdminChannel" is now listening on 10.16.160.21:7001 for protocols t3, http.>

And you still can use the normal AdminServer adress to logon, either hostname or Admin VIP configured.

When you configure Oracle Traffic Director, this should be configured using the IPoIB, in every other setup you choose the EoIB, but then you miss all IB enhancements.

  • Expand Farm_domain_name, SOA, and then right click soa-infra (server_name) (either one of the servers).
  • Click SOA Administration, and then BPEL Properties.
  • Select More BPEL Configuration properties.
  • Enter CacheEnabled for the property QualityOfService property.
  • Select Apply.
  • In the Domain Structure window, expand the Environment node.
  • Click Servers.
  • Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as hyperlinks) in the Name column of the table.
  • The settings page for the selected server appears.
  • Click Lock & Edit.
  • Click the Server Start tab.
  • In the Arguments field, add the following for all SOA managed servers

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES
 

 

 

  EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dbpel.cache.localStorage=true

  -Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g"
export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure, analyse, test and decide!

Using Embedded Coherence ( Several FMW products, SOA suite, IDM)

 

With optimizations in SOA Suite the process state can now reside in Oracle Coherence - which is an in-memory data grid. Coherence uses direct memory acces for replicating data.

We configure coherence using Unicast, therefor we should disable all multicast entries, located in the setDomainEnv

Futher more we should set all WKA adresses to use the IPoIB interface, by setting the virtual hostname that is set for the IP adress. This is typically the VIP adress used for SOA internal traffic ( attached to IB interface)

-Dtangosol.coherence.wka1=SOAHOST1-PRIV-V1

-Dtangosol.coherence.wka2=SOAHOST2-PRIV-V1

-Dtangosol.coherence.localhost=SOAHOST1-PRIV-V1

SOAHOST1-PRIV-V1 is the virtual host name that maps to the virtual IP where WLS_SOA1 is listening (in SOAHOST1), and so on.

 

Configuring Coherence Caches for Dehydrations

Dehydrations take place in the SOA repository database, but you can use Coherence to do offload this kind of work. If you configure the BPEL engine with the CacheEnabled property, the engine runs many fewer reads against database. Because many reads require locks and version checks, eliminating them improves BPEL engine performance. Configuring Oracle Coherence for dehydration requires the following steps:

Enabling the property

To enable the cache enabled property in SOA, follow theses steps:

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control
  2. Expand Farm_domain_name, SOA, and then right click soa-infra (server_name) (either one of the servers).
  3. Click SOA Administration, and then BPEL Properties.
  4. Select More BPEL Configuration properties.
  5. Enter CacheEnabled for the property QualityOfService property.
  6. Select Apply.

 

 

Next, set the required server properties for using in-process coherence cache for dehydration.

To enable the cache enabled property in SOA:

  1. In the Domain Structure window, expand the Environment node.
  2. Click Servers.
  3. Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as hyperlinks) in the Name column of the table.
  4. The settings page for the selected server appears.
  5. Click Lock & Edit.
  6. Click the Server Start tab.
  7. In the Arguments field, add the following for all SOA managed servers

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

Update SOA JVM settings

To optimize behavior with local storage caches and improve performance, update the SOA servers' start parameters to also include the following flags:

-Djava.net.preferIPv4Stack=true
-Xlargepages:exitOnFailure=true
-Doracle.xdkjava.exalogic.optimization=true


All tuning settings together in setDomainEnv or UserEnv file

Maybe itś obvious, but every setting you optimize must have a reason, so do not set all these setting just like that. Do it one by one with a group of parameters, and test it with it. The outcome of your analysis will give you a guideline of what is applicable for you and what is not. To be set in the setDomainEnv, or better use a separate parameter file and reference it to the setDomainEnv.

  • Java Optimizations:

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa
-Dsoa.oracle.home=${SOA_ORACLE_HOME}
-Dsoa.instance.home=${DOMAIN_HOME}
-Dtangosol.coherence.clusteraddress=227.7.7.9
-Dtangosol.coherence.clusterport=9778
-Dtangosol.coherence.log=jdk
-Dtangosol.coherence.transport.reliable=imb
 

       → Infiniband enabling! IMB driver mist be enabled for that. Only for Linux x86- 64 bit platforms.


-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl
-Dweblogic.transaction.blocking.commit=true
-Dweblogic.transaction.blocking.rollback=true
-Djavax.net.ssl.trustStore=<location of your trust store>"
export EXTRA_JAVA_PROPERTIES

 

  • BPEL Optimizations. The main parameter is the bpel.cache.localStorage; set it to true and then tune the other paramters to your needs, like cache sizes and worker threads.

  EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g"
export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure, analyse, test and decide!

Cluster replications

This applies only for stateful applications such as B2B Console, or SOA/BPM Composer

To enable session replication enhancements for the SOA_Cluster:

  1. Shut down the servers in the SOA_Cluster.
  2. To set replication ports for a managed server, such as WLS_SOA1:
    1. Under Domain Structure, click Environment and Servers. Click on the SOA managed server, Configuration tab, Cluster tab.
    2. In the Replication Ports field, enter a range of ports for configuring multiple replication channels. For example, replication channels for managed servers in SOA_Cluster can listen on ports starting from 8006 to 8009. To specify this range of ports, enter 8006-8009.

 

Create a custom network channel for each managed server in the cluster : 

Log on to the Admin Console In the Servers table, click the SOA managed server instance.

  1. Select Protocols, and then Channels, and click New.
  2. Listen Address: SOAHOST1-PRIV-V1 ( (IPoIB Adress )
  3. Listen port: 8006 Remove the default external listen port.
  4. Click Next, and in the Network Channel Properties page, select Enabled and Outbound Enabled, then click Finish.
  5. Under the Network Channels, select ReplicationChannel, the network channel you created for the WLS_SOA1 managed server.

 

  • In the Replication Channel field, ensure that ReplicationChannel is set as the name of the channel to be used for replication traffic.
  • In the Advanced section, select the Enable One Way RMI for Replication option., and click Save.
  • Activate the changes and restart the managed servers.
  • Add the system property -Djava.net.preferIPv4Stack=true to the start parameters for the WebLogic servers
  • Configure multiple services to share the same store if they will commonly be invoked within the same transaction.
  • Do no use the XA driver for the JDBC Store because the JMS infrastructure of WebLogic itselve is fully equipped for XA
  • JMS often executes direct database operations to invoke its store services within the same transaction,so using a JDBC data source with Logging Last Resource (LLR) might be a good option; it will do two-phase commit, in a single local transaction DB operation, this improves the overall transaction performance.

 

  • MessageCompressionOptions: Along with the option above, choose one of the compression option in. Note that compression is done by gzip, so you can choose for the best compression:
    • GZIP_DEFAULT_COMPRESSION
    • GZIP_BEST_COMPRESSION
    • GZIP_BEST_SPEED
    • LZF

 

JMS also makes use of the domain wide Exalogic optimizations by using concurrent queues for pending messages. A buffer queue is used for incoming requests which increases throughput aby scheduling these requests and prevent locks. 

 

Other Considerations

These will discuss some other considerations you could think of, either by determining if it’s applicable to your situation, either it may come out of some load or stresstest scenario and analysis.

Hugepages and Transparent HugePages

Default in Exalogic bare metal compute nodes. Hugepages allocation can be good for applications which have to allocate large memory blocks.

In your Exalogic VM infstrastructure design, decide which VM will be used for such type of applications like WebCenter Portal, and if they need it. In some situation it can be necessary to reserve a computenode for these environments instead of doing dynamic allocation, and you might consider to use Transparent HugePages for these.






Oracle Traffic Director enhancements

 

In almost every Enterprise Deployment Guide from a Fusion Middleware Product, Oracle uses the Oracle WebTier as a controlling layer for you HTTP calls.

In Exalogic, a software loadbalancer is added, as you might know, called Oracle Traffic Director. Depending on the approach of how you design your Cloud infrastructure with one of these components

  • Network topology
  • Segregations of duties and and task rolled out over several departments, or done by one department overall? Who will take care of what?( Cloud Infrastructure team, Middleware Team, all by one DevOps team)
  • Other solutions than provided by Oracle ( 3rd party WebTiers, hardware loadbalancers)
  • Failover groups
  • Origin Servers
  • Various listeners (HTTP, TCP)

SOA Suite

The SOA servers listen on both EoIB and IPoIB interfaces for in and external traffic. Internal could be:

  • Cluster traffic
  • Internal apllication traffic ( RMI, T3 ) An example could be an configured route for mediator traffic, or business services to SOA composites.
  • External traffic for in and outbound services.

 

The default channel uses an IPoIB listen address. This is for optimized server-to-server as well as for Coherence dehydration and deployment optimizations.

Besides the IPoIB interfaces, the SOA Managed Servers also listen on EoIB virtual IP addresses so that they can be accessed externally for the following purposes:

  • RMI/JMX/T3 traffic
  • HTTP traffic used for remote deployment.
  • External JMS producers and consumers.
  • Can use separate WebLogic channels for T3 and HTTP for traffic isolation.

 

OSB

OSB can follow the same pattern as SOA Suite. OSB should be the central bus for routing, transforming and exchanging messagetaffic between different front and backend applications

OSB also uses coherence for result caching. We now use the in JVM mechanism, which means internal coherence libraries will be used.

There for, Coherence WKA addresses need to be configured in the server start parameters. These WKA addresses should be configured with the IPoIB address. These addresses should be accessible only internal within the OSB WebLogic domain by all managed and AdminServer.

 

 

1  Oracle SOA Suite Basic tuning

This section has been divided in several subsections to identify each layer which participates in the entire stack, and will go through step by step to identify the best optimized settings.

The subsections are

 

  • Operating System/Storage /Network settings
  • Java Virtual Machine settings
  • Oracle WebLogic generic settings
  • Oracle SOA Suite 11g settings:
    • SOA Infrastructure engines
    • SOA Services

 

 

It is very important that individual performance analysis are as similar to each other as possible, to ensure that it is actually meaningful to compare and draw conclusions from the results.

Ensure that the environment is warmed up and has reached its steady state. Measurements and anlysis must be taken during the steady state.

Performance can differ markedly between cold and warm environments because of issues like the impact of optimizations that are done after a number of repetitions of a particular piece of code, the optimizations done by the database after a number of queries have been executed, the settling of objects in the various parts of the JVM heap over time.

 

Not only consider CPU and Memory % but:

  • Memory stats (inc GC)
  • Connection pool stats
  • JMS queue stats
  • Transaction stats
  • Data in and output

 

These areas are typically to look at in these types of environments:

 

  • CPU
  • Memory
  • Locks
  • Bandwidth
  • Storage
  • Resource pools

 

Always consider before implementing :

  • What am I changing
  • Has the bottleneck gone away?
  • Is there a new bottleneck?

 

Way of working:

  • Bottom up
    • Work from the CPU upwards
  • Treat SOA Suite as any other Java application at O/S level
    • Key suspects such as Garbage Collection and locking   

 

 

 

 

 

Improvement Areas : Performance

 

Files/directories recommendations

  • Writing logs to local storage will direct bottlenecks from network connections to shared storage.
  • BUT transaction logs and JMS persistence stores must remain in shared storage, so that failover can work

 

 

Java VM Settings

The JVM heap should be at least 6G for the SOA managed servers. Allow 50% overhead process memory from heap size. For 6G heap allow the java process 10G of memory. Monitor/tune heap size, TLA size etc. during load testing.

 

AdminServer tuning

The AdminServer plays an important role in the processing of SOA transaction flows. The following components need attention:

 

AdminServer JVM Arguments

For JVM, use the following:

  • Set the XgcPrio to pausetime
  • Set the transaction rollback blocking to true
  • Disable Managed Server notifications

 

 

Additional Fusion Middleware EM console and DMS settings:

  • Set the discovery cache age to 28800000
  • Set the discovery wait time to 30000
  • Set the cache results to true
  • Increase the dumpinterval to 10800 with a maximum of 75 MB
  • Enable Large Repository
  • Increase the collector configuration settings for DMS:
    • Use an interval of 300 secs with a remove cycle of 3
    • And use an interval of 120 secs with a remove cycle of 2, set this to default
  • Restrict the amount of displayed instance information to 12 hours

 

Implement SOA ping

Apply Patch 17571359 which provides a HTTP URL for load balancers that returns SOA server status in the format: http://[host]:[port]/soa-infra/services/isSoaServerReady. It will return response code 503 (Service Unavailable) when SOA Server is not ready to accept requests and 200(OK) when SOA Platform is running and accepting requests.

 

JDBC related settings

  • Connection Pool

       JDBC connection pool should be set so that it doesn’t need to grow after it is initialized. This can be done by:

    • Set initial capacity the same as maximum capacity.
    • Be aware that datasources always have enough free connections
    • Statement caching can eliminate potential performance impacts caused by repeated cursor creation and repeated statement parsing and creation. Statement caching also reduces the performance impact of communication between the application server and the database server
    • Disable unnecessary connection testing and profiling.

 

  • Large Payload handling

Disable wrap datatype in case of large SOA payloads being inserted through JCA.

 

SOA Suite transactional improvements

During a longer period of time, it becomes clear which transactions run longer than initially expected. One should always monitor long running transactions to detect possible problem areas.

Based on that analysis, the following needs attention:

 

SOA Database:                                             Distributed lock timeout

WebLogic domain wide transacations:     JTA – identify increasing amount of application errors and rollbacks

SOA Engine beans:                                      Increase Timeout

BPEL:                                                            Wait time for synchronous invocations

 

 

In-Flight processes

Make use of this option when using synchronous operations, but limit persistency.

  • InMemoryOptimisation (transient processes)
  • Reduce audit threshold
  • Reduce completion persist level
  • Increase audit trail size threshold
  • Increase large document threshold

 

SOA Suite BPEL engine

  • Minimize the amount of audit on transactions
  • Increase the dispatcher threads to a value ( after gaining performance statitistics)
  • Persist one-way invocations in a asynchronous manner

                    

SOA Suite Mediator engine

  • Prefer to use DVM lookups for XSL transformations ( this is to reduce I/O operations)
  • Use and tune deferred routing rule mechanism

 

 

Improvement areas: Stability

 

Instance Purging/archiving

Scripts are available for instance archiving and purging. A better option is to use of database partitions which can be managed by point in time to archive and drop instance data.

See http://docs.oracle.com/cd/E28280_01/admin.1111/e10226/soaadmin_partition.htm for details. Oracle DBA’s should be able to assist in moving to partitioned database from current non-partitioned one.

 

MDS update/refresh

  • Whenever there is a change in endpoint from end systems, the MDS has to be updated. End point info should be in config plan, which allows new values at the time of deployment.
  • If you change endpoint info in MDS post deployment, then you can clear MDS cache using EM from the MDSAppRuntime
  • MDS should be cleared on a regular basis – old metadata labels(> 3 months ) should be cleaned

 

Instance Patching

Instance patchingis useful when

  • A new version of process can be deployed and running instances can either continue on old version
  • Or migrate to new version either automatically (if versions are compatible) .

 

Zero Downtime Patching

Apply patch to Admin server first. If the patch does not contain WebLogic cluster fixes that would cause an inconsistency in the cluster the patch may be applied without shutting down the whole cluster.

Since OPatch works on the SOA and oracle_common directories in middleware home the cluster will also need at least two middleware home directories. Shutdown all the members using one of the middleware home directories, apply the patch(es) and then restart the server instances. Repeat those steps on the members using the other middleware home.

 

Oracle Service Bus Basic Tuning

This section has been divided in several subsections to identify each layer which participates in the entire stack, and will go through step by step to identify the best optimized settings. In fact it is the same approach as in the Oracle SOA Suite, although from a historical perspective, OSB in tighter integrated with WebLogic.

 

The subsections are:

  • Java Virtual Machine settings
  • Oracle WebLogic generic settings
  • Oracle Service Bus 11g settings:
    • Service Bus Infrastructure engines
    • OSB Proxy and Business Services

Key areas are:

  • JVM
  • JDBC
  • JMS
  • Caching /Coherence

 

Improvement areas: Performance Tuning

 

JVM tuning

Always keep in mind that OSB is typically focusing on throughput, short live objects so try to size the JVM not too small or too big. GC should not take longer than 1 sec.

Size of JVM is usually 4 to 6 GB.

 

Logging on WebLogic

Disable access logging for managed and AdminServer, leave that to Oracle HTTP Server.

 

JMS

Consider if persistency is needed for any JMS. If it’s not, disable it. This should be known by development team but operations team can advise and discuss.

On non-Exalogic hardware, use file based persistency.

 

Router Runtime Cache

To reduce the amount of time spent on compiling the proxy pipeline set this value to 80% of the amount of deployed proxy services.

 

 

Service Result caching

Use this options in certain OSB Business services, in combination with the underlying coherence framework.

Consider to use either in JVM coherence or configure coherence in its own JVM. For the last option you need more resources ( CPU, RAM, Storage)

 

Static data

Cache the static data that need to be configured in your application. You need to consider:

  • Amount of data need to be cached
  • Which caching api to use – recommendation is to use Coherence

   

 

JCA Adapter tuning

Consider every polling interval used ad some adapter (FTP, file, DB) if applicable

 

activationInstances

For parallel processing use this parameter. The most optimum value should be determined on the amount of messages per hour, usually this parameter is 6 or 8.

 

Monitoring and alerting

Keep this for the most services at global level, only for key services, define a separate monitoring approach.

 

For several proxy and business services which arre marked as critical, configure and use pipeline and / or SLA alerting

 

Improvement areas: Stability

 

Logging, monitoring, tracing

  • Log as minimum as necessary
  • Monitor important areas with an SLA alert
  • Never enable tracing in production

 

Purging alerts

OSB SLA and pipeline alerts need to be purged on a regular base. Create a custom WLDF archive for the alerts and use the following purging strategy:

 

Type

Retention

Purge time

Pipeline and SLA Alerts

14 days ( 336 hours )

Every day at 00:00