You can accomplish that grouping one or more caches in different services. A Coherence cluster manages a group of JVMs, in which each JVM manages one or more services. Each cache must belong to a service. If a cache "X" belongs to service "A", and a cache "Y" belongs to a service "B", means that cache "X" will live only in the JVMs that contains the service "A". You can create the same structure with OSB and external cache servers.
Dave Felcey, Coherence product manager, wrote an interesting blog that explains this scenario in more details: https://blogs.oracle.com/felcey/entry/coherence_clustering_principles