In a 2 managed clustered weblogic environment with OSM 7.0.3 and an external load balancer to balance the incoming traffic it is noticed from the managed servers that whichever server is scanning for orders and that is understood by the server's logs:
####<Oct 16, 2012 2:54:33 PM EEST> <Info> <oms> <> <osm_ms01> <Timer-9> <oms-internal> <> <fab6ae59fd53672b:704b5627:13a64686216:-8000-0000000000000010> <1350388473505> <BEA-000000> <cluster.ClusteredHandlerFactory: Querying for high activity orders across the cluster>
is the server that will serve a new order.
Is there a way to achieve a perfect load balance? In a test case of 200 orders all orders where processes by one node and it is the one that scans for new orders.
We configured the external load balancer to split the traffic..But nothing!! Is there an internal mechanism that gathers all orders that are send to multiple servers and executes them in the server that is currently scanning for orders ???
Is there in any manual or Oracle Support Document/note on how is decided in a multiple-clustered environment which server will execute orders???
Here's some general information on load balancing:
1. With OSM order affinity, the managed server instance that receives the order (precisely, creates the OSM order) has sole ownership of the order. Other than specific circumstances, the ownership is not transferred, and thus processing of that order stays with that instance till completion.
2. The OSM web service (createOrder API) has no load balancing mechanism internally if HTTP is used as the transport. So if you only send orders to one managed server, that instance will create and thus own all these orders. In contrast, if you use JMS as the transport, it is load-balanced by the JMS distributed destination (provided you are not sending instead to member queues of the distribution destination).
Now, assuming you are using HTTP, you need to ensure that the Load Balancer is really round-robining on the 2 managed servers among HTTP messages of order submissions. Monitor your TCP pipes to verify.
A problem we've seen, is if you are using SoapUI with pre-emptive authentication disabled, the SOAP request without pre-emptive authentication will be rejected, causing a re-send. Because of LB, all orders ended up in one managed server, as the reject-then-accept SOAP message sequence becomes cyclic with odd-even round-robin. So, enable pre-emptive authentication to avoid that.
Btw, is your cartridge handling high-activity orders? If not, I have a suspicion that your pasted log message may be a red-herring.
High-activity orders have a large number of tasks, and usually a large number of sub-processes and tasks that must be executed concurrently. High activity order processing is enabled by default -- see com.mslv.oms.handler.cluster.ClusteredHandlerFactory.HighActivityOrder.CollectionCycle.Enabled in oms-config.xml.
There are a few levels where you can monitor traffic. I assume you are not seeing any traffic in weblogic console already? For me, just to troubleshoot whether the traffic is coming in at all from the load balancer, I'd monitor at the TCP level, e.g. using tcpmon.