Some of the old stuff from my previous whitepaper and some new insights about Oracle FMW on Exalogic. Lost to consider!

 

 

Exalogic as a converged engineered system has various layers to consider when we want to apply all the possiblle enhancements on every possible layer in the Oracle FMW Stack

 

In this whitepaper we will start with the following components:

  • WebLogic Server
  • Coherence
  • Database traffic ( JDBC)
  • Message Traffic ( HTTP, REST, SOAP,JMS)
  • Traffic Director

 

Area

Parameter

Setting

Remarks

WLS

Exalogic Enhancement

Tickbox enabled

Set in WLS console

WLS

I/O Enhancement

-Dweblogic.ScatteredReadsEnabled=true

-Dweblogic.GatheredWritesEnabled=true

-Dweblogic.replication.enableLazyDeserialization=true

Domain Env file

WLS

Exalogic Enhancement

# Enable Java Exalogic optimizations
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

Domain Env file

WLS

ListenAdress IPOIB

WLS AdminServer ListenAdress

WLS Console

WLS

CPU

  1. KernelMBean.AddWorkManagerThreadsByCpuCount mbean

FMW Console

WLS

WLS

Admin Channel

t3, http

WLS Console

WLS

Client Channels

t3, http

WLS Console

JDBC

 

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=<>)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=<DB Service>)))

JDBC URL

JDBC

JDBC optimization

  • oracle.jdbc.enableJavaNetFastPath=true

FMW Console

Coherence

WKA

-Dtangosol.coherence.wka1=(IPoIB adress localhost)

-Dtangosol.coherence.wka2=(IPoIB adress1)

-Dtangosol.coherence.localhost=(IPoIB adress2)



In-JVM coherence and separate Coherence cluster

SOA BPEL

QualityOfService

CacheEnabled

FMW Console

SOA Infra

SOA BPEL Coherence Dehydration

-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=<value>

-Dbpel.cache.cubeInstance.sizeLimit=<value>

-Dbpel.cache.invokeMessage.sizeLimit=<value>

-Dbpel.cache.deliveryMessage.sizeLimit=<value>

-Dbpel.cache.deliverySubscription.sizeLimit=<value>

Domain Env File

SOA Infra

SOA Coherence Optimizations

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa
-Dsoa.oracle.home=${SOA_ORACLE_HOME}
-Dsoa.instance.home=${DOMAIN_HOME}
-Dtangosol.coherence.log=jdk
-Dtangosol.coherence.transport.reliable=imb

-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl
-Dweblogic.transaction.blocking.commit=true
-Dweblogic.transaction.blocking.rollback=true
export EXTRA_JAVA_PROPERTIES

 

 

 

 

Domain Env File

JMS

Message Compression

StoreMessageCompressionEnabled=true

WLS Conole

JMS

Message Compression

PagingMessageCompressionEnabled

WLS Console

JMS

TLOG and persistency

JDBC Store ( Over IpoIB)

WLS Console

JMS

Lockless requests

  1. ServerMBean.useConcurrentQueueForRequestManager=true

FMW Console

JSP

JSP Factory caching

jsp_factory_caching=true

 

 

WebLogic Admin channel over Infiniband private net

 

The private InfiniBand fabric including WebLogic Clusters and Coherence clusters, requires a set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for cluster communication. These IP addresses are associated with the BOND0 interface.

 

Within Exalogic it is possible to use a private vnet that can be reserved for administrative purposes such as accessing the AdminServer and its consoles, internal cluster communication and so on.

This can be easily achieved by setting all listenadresses to the IP that is bound to Infiniband

The same we do for all managed servers and their listen adresses

Trouble is, that we can’t access the AdminServer url over IPOIB, For this we create separate channels

Also for the managed servers, which have clients access we create channels to connect to the outside world 

 

There are several channels to identify:

  • Several channels for EoIB interfaces, such as:
    • AdminServer Network Channel
    • HTTP and T3 Network Channels

These are needed for internal resources like AdminServers or managed servers internal communications to communicate with the outside world.

You must create two network channels for the Administration Server. The network channels are necessary for routing HTTP traffic and T3 traffic (TCP-based protocol used by WebLogic Server) coming in from an external data center via Ethernet over InfiniBand (EoIB). You must create the following network channels:

HTTP Client channel for AdminServer

  1. Log in the WLS console and Click Lock & Edit in the Change Center.
  2. In the left pane of the Console, expand Environment, and then Servers.
  3. In the Servers tab, click AdminServer(admin).
  4. Select Protocols → Channels. Click New.
  5. Enter AdminHTTPClient as the name of the new network channel and select http as the protocol, then click Next.
  6. Enter the following information in the Network Channel Addressing page:
    • Listen address: < the EOIB IP adress >This address is the floating IP assigned to the Administration Server using the BOND<n> interface.
    • Listen port: 7001
  7. Click Next, and select Enabled on the Network Channel Properties page. and Click Finish.
  8. Activate these changes,

 

T3 Client channels

Repeat the same steps as above but choose T3 as a protocol. Repeat these steps also for all managed servers

 

When started, you see the several interfaces which are listened on

Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.11.26:7001 for protocols iiop, t3, ldap, snmp, http.>

<Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "AdminChannel" is now listening on 10.16.160.21:7001 for protocols t3, http.>

And you still can use the normal AdminServer adress to logon, either hostname or Admin VIP configured.

When you configure Oracle Traffic Director, this should be configured using the IPoIB, in every other setup you choose the EoIB, but then you miss all IB enhancements.

  • Expand Farm_domain_name, SOA, and then right click soa-infra (server_name) (either one of the servers).
  • Click SOA Administration, and then BPEL Properties.
  • Select More BPEL Configuration properties.
  • Enter CacheEnabled for the property QualityOfService property.
  • Select Apply.
  • In the Domain Structure window, expand the Environment node.
  • Click Servers.
  • Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as hyperlinks) in the Name column of the table.
  • The settings page for the selected server appears.
  • Click Lock & Edit.
  • Click the Server Start tab.
  • In the Arguments field, add the following for all SOA managed servers

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES
 

 

 

  EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dbpel.cache.localStorage=true

  -Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g"
export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure, analyse, test and decide!

Using Embedded Coherence ( Several FMW products, SOA suite, IDM)

 

With optimizations in SOA Suite the process state can now reside in Oracle Coherence - which is an in-memory data grid. Coherence uses direct memory acces for replicating data.

We configure coherence using Unicast, therefor we should disable all multicast entries, located in the setDomainEnv

Futher more we should set all WKA adresses to use the IPoIB interface, by setting the virtual hostname that is set for the IP adress. This is typically the VIP adress used for SOA internal traffic ( attached to IB interface)

-Dtangosol.coherence.wka1=SOAHOST1-PRIV-V1

-Dtangosol.coherence.wka2=SOAHOST2-PRIV-V1

-Dtangosol.coherence.localhost=SOAHOST1-PRIV-V1

SOAHOST1-PRIV-V1 is the virtual host name that maps to the virtual IP where WLS_SOA1 is listening (in SOAHOST1), and so on.

 

Configuring Coherence Caches for Dehydrations

Dehydrations take place in the SOA repository database, but you can use Coherence to do offload this kind of work. If you configure the BPEL engine with the CacheEnabled property, the engine runs many fewer reads against database. Because many reads require locks and version checks, eliminating them improves BPEL engine performance. Configuring Oracle Coherence for dehydration requires the following steps:

Enabling the property

To enable the cache enabled property in SOA, follow theses steps:

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control
  2. Expand Farm_domain_name, SOA, and then right click soa-infra (server_name) (either one of the servers).
  3. Click SOA Administration, and then BPEL Properties.
  4. Select More BPEL Configuration properties.
  5. Enter CacheEnabled for the property QualityOfService property.
  6. Select Apply.

 

 

Next, set the required server properties for using in-process coherence cache for dehydration.

To enable the cache enabled property in SOA:

  1. In the Domain Structure window, expand the Environment node.
  2. Click Servers.
  3. Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as hyperlinks) in the Name column of the table.
  4. The settings page for the selected server appears.
  5. Click Lock & Edit.
  6. Click the Server Start tab.
  7. In the Arguments field, add the following for all SOA managed servers

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

Update SOA JVM settings

To optimize behavior with local storage caches and improve performance, update the SOA servers' start parameters to also include the following flags:

-Djava.net.preferIPv4Stack=true
-Xlargepages:exitOnFailure=true
-Doracle.xdkjava.exalogic.optimization=true


All tuning settings together in setDomainEnv or UserEnv file

Maybe itś obvious, but every setting you optimize must have a reason, so do not set all these setting just like that. Do it one by one with a group of parameters, and test it with it. The outcome of your analysis will give you a guideline of what is applicable for you and what is not. To be set in the setDomainEnv, or better use a separate parameter file and reference it to the setDomainEnv.

  • Java Optimizations:

 

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa
-Dsoa.oracle.home=${SOA_ORACLE_HOME}
-Dsoa.instance.home=${DOMAIN_HOME}
-Dtangosol.coherence.clusteraddress=227.7.7.9
-Dtangosol.coherence.clusterport=9778
-Dtangosol.coherence.log=jdk
-Dtangosol.coherence.transport.reliable=imb
 

       → Infiniband enabling! IMB driver mist be enabled for that. Only for Linux x86- 64 bit platforms.


-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl
-Dweblogic.transaction.blocking.commit=true
-Dweblogic.transaction.blocking.rollback=true
-Djavax.net.ssl.trustStore=<location of your trust store>"
export EXTRA_JAVA_PROPERTIES

 

  • BPEL Optimizations. The main parameter is the bpel.cache.localStorage; set it to true and then tune the other paramters to your needs, like cache sizes and worker threads.

  EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g"
export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure, analyse, test and decide!

Cluster replications

This applies only for stateful applications such as B2B Console, or SOA/BPM Composer

To enable session replication enhancements for the SOA_Cluster:

  1. Shut down the servers in the SOA_Cluster.
  2. To set replication ports for a managed server, such as WLS_SOA1:
    1. Under Domain Structure, click Environment and Servers. Click on the SOA managed server, Configuration tab, Cluster tab.
    2. In the Replication Ports field, enter a range of ports for configuring multiple replication channels. For example, replication channels for managed servers in SOA_Cluster can listen on ports starting from 8006 to 8009. To specify this range of ports, enter 8006-8009.

 

Create a custom network channel for each managed server in the cluster : 

Log on to the Admin Console In the Servers table, click the SOA managed server instance.

  1. Select Protocols, and then Channels, and click New.
  2. Listen Address: SOAHOST1-PRIV-V1 ( (IPoIB Adress )
  3. Listen port: 8006 Remove the default external listen port.
  4. Click Next, and in the Network Channel Properties page, select Enabled and Outbound Enabled, then click Finish.
  5. Under the Network Channels, select ReplicationChannel, the network channel you created for the WLS_SOA1 managed server.

 

  • In the Replication Channel field, ensure that ReplicationChannel is set as the name of the channel to be used for replication traffic.
  • In the Advanced section, select the Enable One Way RMI for Replication option., and click Save.
  • Activate the changes and restart the managed servers.
  • Add the system property -Djava.net.preferIPv4Stack=true to the start parameters for the WebLogic servers
  • Configure multiple services to share the same store if they will commonly be invoked within the same transaction.
  • Do no use the XA driver for the JDBC Store because the JMS infrastructure of WebLogic itselve is fully equipped for XA
  • JMS often executes direct database operations to invoke its store services within the same transaction,so using a JDBC data source with Logging Last Resource (LLR) might be a good option; it will do two-phase commit, in a single local transaction DB operation, this improves the overall transaction performance.

 

  • MessageCompressionOptions: Along with the option above, choose one of the compression option in. Note that compression is done by gzip, so you can choose for the best compression:
    • GZIP_DEFAULT_COMPRESSION
    • GZIP_BEST_COMPRESSION
    • GZIP_BEST_SPEED
    • LZF

 

JMS also makes use of the domain wide Exalogic optimizations by using concurrent queues for pending messages. A buffer queue is used for incoming requests which increases throughput aby scheduling these requests and prevent locks. 

 

Other Considerations

These will discuss some other considerations you could think of, either by determining if it’s applicable to your situation, either it may come out of some load or stresstest scenario and analysis.

Hugepages and Transparent HugePages

Default in Exalogic bare metal compute nodes. Hugepages allocation can be good for applications which have to allocate large memory blocks.

In your Exalogic VM infstrastructure design, decide which VM will be used for such type of applications like WebCenter Portal, and if they need it. In some situation it can be necessary to reserve a computenode for these environments instead of doing dynamic allocation, and you might consider to use Transparent HugePages for these.






Oracle Traffic Director enhancements

 

In almost every Enterprise Deployment Guide from a Fusion Middleware Product, Oracle uses the Oracle WebTier as a controlling layer for you HTTP calls.

In Exalogic, a software loadbalancer is added, as you might know, called Oracle Traffic Director. Depending on the approach of how you design your Cloud infrastructure with one of these components

  • Network topology
  • Segregations of duties and and task rolled out over several departments, or done by one department overall? Who will take care of what?( Cloud Infrastructure team, Middleware Team, all by one DevOps team)
  • Other solutions than provided by Oracle ( 3rd party WebTiers, hardware loadbalancers)
  • Failover groups
  • Origin Servers
  • Various listeners (HTTP, TCP)

SOA Suite

The SOA servers listen on both EoIB and IPoIB interfaces for in and external traffic. Internal could be:

  • Cluster traffic
  • Internal apllication traffic ( RMI, T3 ) An example could be an configured route for mediator traffic, or business services to SOA composites.
  • External traffic for in and outbound services.

 

The default channel uses an IPoIB listen address. This is for optimized server-to-server as well as for Coherence dehydration and deployment optimizations.

Besides the IPoIB interfaces, the SOA Managed Servers also listen on EoIB virtual IP addresses so that they can be accessed externally for the following purposes:

  • RMI/JMX/T3 traffic
  • HTTP traffic used for remote deployment.
  • External JMS producers and consumers.
  • Can use separate WebLogic channels for T3 and HTTP for traffic isolation.

 

OSB

OSB can follow the same pattern as SOA Suite. OSB should be the central bus for routing, transforming and exchanging messagetaffic between different front and backend applications

OSB also uses coherence for result caching. We now use the in JVM mechanism, which means internal coherence libraries will be used.

There for, Coherence WKA addresses need to be configured in the server start parameters. These WKA addresses should be configured with the IPoIB address. These addresses should be accessible only internal within the OSB WebLogic domain by all managed and AdminServer.