This discussion is archived
4 Replies Latest reply: Jan 8, 2013 8:23 PM by 840063 RSS

Coherence Common issues with Nodes Started through WLS Node Manager

694204 Newbie
Currently Being Moderated
I have an issue with Coherence Common which I am hoping someone can shed some light on. I have set up a custom NamespaceContentHandler which processes some custom xml configuration in an extensible cache configuration and this uses reflection to create and run some custom tasks when the coherence node starts. This was all up and running in Eclipse and when the Node started the Handler code was invoked and the Events were raised and using the Processing Pattern loaded some data into the cluster node. This all worked fine.

However I now have the requirement to integrate this code into a WLS domain and have the Coherence Servers & Cluster configured in the WLS console. After setting this up I am now facing a problem that was not there in the standalone JVM eclipse environment. It seems that my custom cluster configuration file is being processed by 2 seperate threads when started from Weblogic Node Manager. So in the coherence logs in the weblogic domain $DOMAIN_HOME/server_coherence/SERVER/SERVER.out I can see errors relating to redefinition of the ExtensibleEnvironment. Log extract as below:-

Oracle Coherence Version 3.6.0.4 Build 19111

Grid Edition: Production mode

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2012-03-06 11:19:01.489/1.077 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-cluster-facade-999.0.0.0-SNAPSHOT.jar!/esl-extensible-cache-config.xml"

Using the Incubator Extensible Environment for Coherence Cache Configuration

Copyright (c) 2010, Oracle Corporation. All Rights Reserved.


2012-03-06 11:19:01,997 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Started ConfigurationHandler
2012-03-06 11:19:02,042 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Finished ConfigurationHandler()
2012-03-06 11:19:02,042 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Starting onStartScope()
2012-03-06 11:19:02,047 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Finishing onStartScope()

2012-03-06 11:19:02.062/1.650 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-cluster-facade-999.0.0.0-SNAPSHOT.jar!/esl-cluster-cache-config.xml"
2012-03-06 11:19:02.078/1.666 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-cluster-facade-999.0.0.0-SNAPSHOT.jar!/esl-processing-cache-config.xml"
2012-03-06 11:19:02.091/1.679 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-processing-pattern-1.3.4.23238.jar!/coherence-processingpattern-pof-cache-config.xml"
2012-03-06 11:19:02.092/1.680 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-common-1.7.3.20019.jar!/coherence-common-cache-config.xml"
2012-03-06 11:19:02.123/1.711 Oracle Coherence GE 3.6.0.4 <Warning> (thread=Main Thread, member=n/a): Resource for [interface com.oracle.coherence.patterns.processing.internal.task.TaskProcessorDefinitionManager] is already registered as [com.oracle.coherence.environment.extensible.ExtensibleEnvironment$SingletonResourceResolver@82c8cae]. Skipping requested registration. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment

2012-03-06 11:19:02,143 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Task Name [FixedGeographyCacheLoader]
2012-03-06 11:19:02,143 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Implementation [com.maersk.esl.cache.task.fixedgeo.FixedGeographyReloadTask]
2012-03-06 11:19:02.157/1.745 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Extensible Environment XML configuration has been reset. Will now restart it. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment
2012-03-06 11:19:02.157/1.745 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Extensible Environment XML configuration has been reset. Will now restart it. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment
2012-03-06 11:19:02.165/1.753 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/maerskwas/domains/devWlsDomain/local-lib/coherence-cluster-facade-999.0.0.0-SNAPSHOT.jar!/esl-extensible-cache-config.xml"
2012-03-06 11:19:02.165/1.772 Oracle Coherence GE 3.6.0.4 <Error> (thread=Main Thread, member=n/a): An attempt to recursively load and process a Coherence Cache Configuration has occurred. This is usually caused by accessing a NamedCache or Service (through the CacheFactory) from a NamespaceContentHandler, ElementContentHandler and/or AttributeContentHandler. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment
2012-03-06 11:19:02.184/1.773 Oracle Coherence GE 3.6.0.4 <Error> (thread=Main Thread, member=n/a): An attempt to recursively load and process a Coherence Cache Configuration has occurred. This is usually caused by accessing a NamedCache or Service (through the CacheFactory) from a NamespaceContentHandler, ElementContentHandler and/or AttributeContentHandler. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment
2012-03-06 11:19:02.186/1.774 Oracle Coherence GE 3.6.0.4 <Error> (thread=Main Thread, member=n/a): Stack Trace
com.oracle.coherence.environment.extensible.ExtensibleEnvironment.setConfig(ExtensibleEnvironment.java:392)
com.tangosol.net.DefaultConfigurableCacheFactory.<init>(DefaultConfigurableCacheFactory.java:186)

When I look at the Log4j entries from our own logging it shows that the NamespaceContentHandler is invoked twice by the MAIN thread and a BACKGROUND thread which explains the above. However the reason why this happens is not apparent. The log4j entries are as follows:-

2012-03-06 11:19:01,997 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Started ConfigurationHandler
2012-03-06 11:19:02,042 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Finished ConfigurationHandler()
2012-03-06 11:19:02,042 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Starting onStartScope()
2012-03-06 11:19:02,047 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Finishing onStartScope()
2012-03-06 11:19:02,143 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Task Name [FixedGeographyCacheLoader]
2012-03-06 11:19:02,143 [Main Thread] DEBUG c.m.e.c.c.ConfigurationHandler - Implementation [com.maersk.esl.cache.task.fixedgeo.FixedGeographyReloadTask]
2012-03-06 11:19:02,200 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Started ConfigurationHandler
2012-03-06 11:19:02,200 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Finished ConfigurationHandler()
2012-03-06 11:19:02,200 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Starting onStartScope()
2012-03-06 11:19:02,200 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Finishing onStartScope()
2012-03-06 11:19:02,205 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Task Name [FixedGeographyCacheLoader]
2012-03-06 11:19:02,205 [Environment.Background.Executor:Thread-1] DEBUG c.m.e.c.c.ConfigurationHandler - Implementation [com.maersk.esl.cache.task.fixedgeo.FixedGeographyReloadTask]

Has anyone any idea how Coherence Nodes are working with the WLS Node Manager invocation process. As it is launched as a simple JVM in the same way as in the eclipse version.

I am lost as to how I can stop this happening. Can I somehow switch off thsi Background Executor Thread.

TIA
Martin
  • 1. Re: Coherence Common issues with Nodes Started through WLS Node Manager
    694204 Newbie
    Currently Being Moderated
    Upon further investigation what I found was that basically this is caused by difference of of the Cache Server class used when running standalone Coherence and when running in a WLS environment.
    My Eclipse project was using the default com.tangosol.net.DefaultCacheServer in its run configuration.

    However when Coherence Servers are started from within WLS using Node Manager the actual Main run class is weblogic.nodemanager.server.provider.WeblogicCacheServer.

    When I put this in my eclipse environment I ended up with the same behaviour with 2 threads processing the cache config and causing the Extensible Environment conflicts.

    Has anyone worked with the coherence incubator projects inside this WLS configuration who can help me out. It seems that is the way that Coherence in a WLS domain is supposed
    to be managed allowing WLST control of Coherence servers in the same manner as WLS servers. But it seems that this is incompatible with the coherence incubator projects !?.

    TIA
    Martin
  • 2. Re: Coherence Common issues with Nodes Started through WLS Node Manager
    923522 Newbie
    Currently Being Moderated
    After raising a support request the issue has been reproduced on both WLS 10.3.4 and WLS 10.3.5. So it will need a patch from Oracle it would seem. Interesting that such a major issue has been around across two point releases so I guess not many people can be using NodeManager coherence or at least not with the incubator code !
  • 3. Re: Coherence Common issues with Nodes Started through WLS Node Manager
    Steve-Oracle Newbie
    Currently Being Moderated
    I helped on your support case.

    It was actually to demonstrate the problem using your testcase with the standard com.tangosol.net.DefaultCacheServer if you passed the top level configuration file as an argument to the DefaultCacheServer, e.g.

    java -cp ..... com.tangosol.net.DefaultCacheServer your-extensible-cache-config.xml

    Or likewise with a simple Java class that uses the uses the ConfigurableCacheFactory as follows:

    import com.tangosol.net.CacheFactory;
    import com.tangosol.net.DefaultConfigurableCacheFactory;
    import com.tangosol.net.DefaultCacheServer;

    public class BasicCacheServer extends DefaultCacheServer {
         
         private static final String CACHE_CONFIG_PROPERTY = "tangosol.coherence.cacheconfig";

         public BasicCacheServer(String configFile) {

              super((DefaultConfigurableCacheFactory) CacheFactory
                        .getCacheFactoryBuilder().getConfigurableCacheFactory(
                                  configFile, getContextClassLoader()));
         }
         
         public static void main(String[] args) {
              String cacheConfig = System.getProperty(CACHE_CONFIG_PROPERTY,
                        DefaultConfigurableCacheFactory.FILE_CFG_CACHE);

              final BasicCacheServer cacheServer;
         cacheServer = new BasicCacheServer(cacheConfig);
              cacheServer.startAndMonitor(DEFAULT_WAIT_MILLIS);
         }
         
    }

    and specify the your top-level cache config file using the tangosol.coherence.cacheconfig system property. Unfortunately its not clear to me with the way things are packaged why this leads to the error you saw from the Incubator code, however potentially the WLS class the node manager launches can be altered to initialise the cache differently, hence avoiding this problem.
  • 4. Re: Coherence Common issues with Nodes Started through WLS Node Manager
    840063 Newbie
    Currently Being Moderated
    Is there a fix for this? I am also running into this issue. If startup a coherence server using weblogic.nodemanager.server.provider.WeblogicCacheServer I get the following error and the CacheLoaders that I have configured do not get called.

    2013-01-09 03:55:22.177/2.287 Oracle Coherence GE 3.7.1.5 <Info> (thread=Main Thread, member=n/a): Extensible Environment XML configuration has been reset. Will now restart it. Class:com.oracle.coherence.environment.extensible.ExtensibleEnvironment Method:setConfig

    If I start the Coherence server using com.tangosol.net.DefaultCacheServer everything is fine. No errors messages and CacheLoaders work fine.

    There was a mention of changing the class nodemanager uses to start the Coherence Server - where can I change this?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points