This discussion is archived
10 Replies Latest reply: Jan 9, 2012 4:32 AM by 599962 RSS

WebLogic Threads Settings and DataSource Connections

599962 Newbie
Currently Being Moderated
Hello,

we have been running our PRD Oracle SOA Suite with a 2 nodes cluster , active active.
We lately face errors like the one

<Jan 2, 2012 5:55:40 AM UTC> <Error> <oracle.soa.bpel.engine> <BEA-000000> <
java.lang.OutOfMemoryError: unable to create new native thread

Ihave investigated a lot and I know that it is most likely related to threads consumption and the related O/S threads stack which might be exhausted in the O/S remaining memory, if we abstract the JVM one.
My question though is another one.
Where could I set/modify the number of threads (min, max ) and how the relation of the threads with the defined connections in the Data Sources is done.

I have read many docs from Oracle but I could not find a relevant one which contains all this info.

Your feedback is appreciated.

Kind Regards,
Loukas
  • 1. Re: WebLogic Threads Settings and DataSource Connections
    Pierluigi Vernetto Pro
    Currently Being Moderated
    before you start playing with threads, I would suggest you to enable the JVM option HeapDumpOnOutOfMemory and run a heap analysis using any tool like Eclipse MAT. But perhaps you have already done it.


    http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

    Anyway in general thread control is achieved with Work managers

    http://weblogic-wonders.com/weblogic/2010/11/21/understanding-the-work-managers/

    (I guess you already know that too)

    I would assume that getting a DB connection from the pool is achieved synchronously by the same Execute Thread, but I might be wrong...
  • 2. Re: WebLogic Threads Settings and DataSource Connections
    PHCharbonneau Explorer
    Currently Being Moderated
    Hi Loukas,

    Can you please let me know which JVM vendor you are using e.g. Sun / Oracle HotSpot, Oracle Jrockit or IBM VM?

    java.lang.OutOfMemoryError: unable to create new native thread

    This means that your JVM is running out of native memory / C-Heap which is required when attempting to create a new Java Thread. This does not necessarily means that your OS is out of physical / virtual memory unless you are using a 64-bit VM.

    There are many scenarios which can trigger this problem . One common problem is using a 32-bit VM along a Java Heap which is too big (e.g. 2.5 GB) vs. # of Threads created by your VM / Thread Pool settings.
    For a 32-bit VM, the rule is the bigger your Java Heap, smaller is your C-Heap capacity and lower is your # of Java Threads that can be created by your VM.

    Can you please provide the information below:

    - JVM vendor and settings (Xms & Xmx, HotSpot vs. JRockit vs IBM, 32-bit vs. 64-bit)
    - When problem is observed please generated a JVM Thread Dump so we can have a look at all your created and active Java Threads. Please also capture the total process size in MB and available physical / virtual memory of your OS (Solaris, Windows, AIX etc.)

    Thanks.
    P-H
    http://javaeesupportpatterns.blogspot.com/
  • 3. Re: WebLogic Threads Settings and DataSource Connections
    599962 Newbie
    Currently Being Moderated
    Hello P-H,

    we are using a 32bit SUN JVM Hotspot on x86_64 server, on RedHat EL 5.3. The total physical memory on the server is 16GB.
    The Xms & Xmx for each wls server on this physical machine:

    soa_server1: -Xms2304m -Xmx2304m
    AdminServer:-Xms768m -Xmx1536m
    bam_server1:-Xms768m -Xmx1536m

    One dummy question: what u mean by C-Heap? Ok I got. You meant the Native Heap (C-Heap)
    The problem has occured 2 times around 03:00 in the morning. We are trying to find a way to have automatic thread dump generation.
    Unfortunately we lack the knowledge in this regards and consultancy is out of question.
    I have read a lot of ways to achieve it but I do not know the most effective one.

    Brgds,
    Loukas

    Edited by: rouglas4 on Jan 4, 2012 10:58 AM
  • 4. Re: WebLogic Threads Settings and DataSource Connections
    PHCharbonneau Explorer
    Currently Being Moderated
    Hi Loukas and thanks for the provided information,

    Since you also use the HotSpot VM, can you please provide me with your PermGen space settings as well e.g. PermSize / MaxPermSize e.g. 256 MB? 512 MB?

    Given the capacity that you have from your physical server (16 GB) and Linux OS and fact that you are using a 32-bit VM, I'm definitely suspecting that you are simply running out of VM Native Heap (C-Heap).

    The problem is that Weblogic is using a self tuning Thread Pool approach (since WLS 9.2) which means the # of created Threads may change and grow over time (grow / shrink) depending of your load or platform situation, slowdown conditions etc.. You need to ensure that you have enough Native Heap capacity to handle these dynamic Thread creations under any application scenario (happy and non happy paths).

    That being said, your SOA Server 1 Java Heap is big for a 32-bit VM (2304 MB). Typically I do not recommend to my clients to go over 2 GB (2048 MB) for a single HotSpot VM with let's say 512 MB of PermGen space since this is leaving smaller space for your native Heap so you have to find a proper balance otherwise you need to either scale up your Weblogic domain vertically (add more JVM's / managed servers) or switch to a 64-bit VM.

    My recommendations to you:

    - Determine how much Java Heap you really need (enable verbose GC or monitoring tool of your choice) and determine if you can reduce your current settings to 2048 MB, this will re-allocate a 256 MB space segment along with memory addresses for your Native Heap, providing you extra capacity for Thread creations and Native VM operations.

    Regarding Thread Dump generation, a simple approach is to generate one every 5 minutes using Linux / UNIX cron (kill -3 <Java Pid>). This is really non intrusive and will at least provide you with # and source of Threads at least 5 minutes before / after your OOM occurence.

    One final point, If the # of threads created is too big e.g. few hundreds, then please consider using Weblogic Work Managers and Max Thread Constraints. This will prevent Weblogic from creating too many Threads from your application requests and overload your Native Heap capacity.

    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/
  • 5. Re: WebLogic Threads Settings and DataSource Connections
    599962 Newbie
    Currently Being Moderated
    Hi P-H,
    unfortunately I have even more PermGen:

    -XX:PermSize=768m -XX:MaxPermSize=768m
    I followed the rule for 1/3 of the Heap Max/Min.

    So you mean that Native Memory used by the JVM for Code Optimization and for loading the classes and libraries is not enough?
    Would the following give me the remaining space left for native heap: Native Space = ( ProcessSize – MaxHeapSize – MaxPermSize)

    and how could I find the Process Size in the top command on linux? I am doing it like this:

    pssoa ->15294

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    15294 oracle11 25 0 3529m 3.0g 43m S 1 19.3 94:51.67 java

    so it should be that the process size is 19,3 % of 16GB of the total physical memory?

    **I forgot to mention that I do not have GC verbose but the following settings:
    -Xms2304m -Xmx2304m -XX:PermSize=768m -XX:MaxPermSize=768m -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/opt/oracle11/Middleware/user_projects/domains/soa_domain/servers/soa_server1/logs/gc_soa1.log

    I will change it to verbose with the first opportunity. I also use Java Visual VM to monitor and I see it reaches very close to the 2.3 GB of the heap size.
    Maybe it make sense to send you a snapshot from the gc log file which also includes a Full GC:

    172863.170: [GC [PSYoungGen: 732501K->36016K(740928K)] 2217205K->1525030K(2313792K), 0.0755300 secs] [Times: user=0.15 sys=0.00, real=0.08 secs]
    172865.243: [GC [PSYoungGen: 740912K->3059K(707968K)] 2229926K->1509895K(2280832K), 0.0312270 secs] [Times: user=0.11 sys=0.00, real=0.04 secs]
    173060.846: [GC [PSYoungGen: 707955K->18376K(744320K)] 2214791K->1562389K(2317184K), 0.0773040 secs] [Times: user=0.15 sys=0.00, real=0.08 secs]
    173260.845: [GC [PSYoungGen: 719496K->14552K(743232K)] 2263509K->1564750K(2316096K), 0.0534370 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    173260.899: [Full GC [PSYoungGen: 14552K->0K(743232K)] [PSOldGen: 1550197K->631422K(1572864K)] 1564750K->631422K(2316096K) [PSPermGen: 499493K->476490K(786432K)], 3.3323150 secs] [Times: user=3.32 sys=0.00,
    real=3.33 secs]
    173509.561: [GC [PSYoungGen: 701120K->19747K(746240K)] 1332542K->651169K(2319104K), 0.0825430 secs] [Times: user=0.15 sys=0.00, real=0.09 secs]
    173797.352: [GC [PSYoungGen: 724323K->24477K(744768K)] 1355745K->657463K(2317632K), 0.0589950 secs] [Times: user=0.11 sys=0.01, real=0.06 secs]
    174037.630: [GC [PSYoungGen: 729019K->19997K(748480K)] 1362005K->659886K(2321344K), 0.0557790 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    174230.409: [GC [PSYoungGen: 728875K->19092K(747136K)] 1368764K->663836K(2320000K), 0.0668570 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    174500.516: [GC [PSYoungGen: 728276K->18843K(750592K)] 1373020K->667076K(2323456K), 0.0570470 secs] [Times: user=0.10 sys=0.00, real=0.06 secs]
    174759.014: [GC [PSYoungGen: 732571K->21141K(749568K)] 1380804K->671330K(2322432K), 0.0566830 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    175024.432: [GC [PSYoungGen: 734869K->22096K(753024K)] 1385058K->677457K(2325888K), 0.0607560 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    175203.445: [GC [PSYoungGen: 740624K->14870K(751936K)] 1395985K->675065K(2324800K), 0.0549310 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    175425.870: [GC [PSYoungGen: 733398K->18844K(753984K)] 1393593K->680930K(2326848K), 0.0559440 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    175704.209: [GC [PSYoungGen: 739484K->24479K(753088K)] 1401570K->690390K(2325952K), 0.0611810 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    175943.573: [GC [PSYoungGen: 745119K->20204K(754752K)] 1411030K->693145K(2327616K), 0.0601360 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    176211.366: [GC [PSYoungGen: 742252K->17741K(753728K)] 1415193K->695124K(2326592K), 0.0450120 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
    176340.841: [GC [PSYoungGen: 739789K->13277K(754880K)] 1417172K->692259K(2327744K), 0.0516820 secs] [Times: user=0.10 sys=0.01, real=0.05 secs]
    176555.007: [GC [PSYoungGen: 736989K->19851K(755072K)] 1415971K->700457K(2327936K), 0.0436280 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
    176738.683: [GC [PSYoungGen: 743563K->15278K(756480K)] 1424169K->701251K(2329344K), 0.0510010 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    176758.010: [GC [PSYoungGen: 741550K->3079K(729408K)] 1427523K->691998K(2302272K), 0.0177030 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
    176758.951: [GC [PSYoungGen: 729351K->332K(751552K)] 1418270K->691476K(2324416K), 0.0143520 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
    176759.974: [GC [PSYoungGen: 720204K->384K(720320K)] 1411348K->691668K(2293184K), 0.0103710 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
    176769.043: [GC [PSYoungGen: 720256K->1582K(751808K)] 1411540K->692982K(2324672K), 0.0120730 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]
    176770.072: [GC [PSYoungGen: 719150K->200K(717824K)] 1410550K->692615K(2290688K), 0.0101910 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
    176797.086: [GC [PSYoungGen: 717768K->22081K(748864K)] 1410183K->714516K(2321728K), 0.0262450 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
    176798.629: [GC [PSYoungGen: 736449K->352K(750400K)] 1428884K->712871K(2323264K), 0.0291570 secs] [Times: user=0.11 sys=0.00, real=0.03 secs]
    176799.596: [GC [PSYoungGen: 714720K->160K(751040K)] 1427239K->712783K(2323904K), 0.0114260 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
    176846.794: [GC [PSYoungGen: 715360K->4029K(750592K)] 1427983K->716668K(2323456K), 0.0192550 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
    177062.494: [GC [PSYoungGen: 719229K->15450K(752768K)] 1431868K->728431K(2325632K), 0.0540310 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    177286.233: [GC [PSYoungGen: 734234K->18731K(737536K)] 1447215K->732928K(2310400K), 0.0557730 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    177609.632: [GC [PSYoungGen: 737515K->26792K(745088K)] 1451712K->742930K(2317952K), 0.0563500 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    177864.054: [GC [PSYoungGen: 737448K->19892K(730560K)] 1453586K->744225K(2303424K), 0.0612050 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    178132.776: [GC [PSYoungGen: 730548K->18341K(749120K)] 1454881K->745839K(2321984K), 0.0576360 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    178397.126: [GC [PSYoungGen: 729509K->24076K(735296K)] 1457007K->753232K(2308160K), 0.0603540 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    178673.510: [GC [PSYoungGen: 735244K->20026K(748224K)] 1464400K->755999K(2321088K), 0.0608590 secs] [Times: user=0.13 sys=0.01, real=0.06 secs]
    178756.583: [GC [PSYoungGen: 729530K->14123K(747712K)] 1465503K->753207K(2320576K), 0.0501920 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    178785.886: [GC [PSYoungGen: 723627K->7790K(751040K)] 1462711K->751922K(2323904K), 0.0450590 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
    178933.405: [GC [PSYoungGen: 722670K->14618K(750272K)] 1466802K->760298K(2323136K), 0.0512480 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    179095.324: [GC [PSYoungGen: 729498K->15784K(754560K)] 1475178K->763527K(2327424K), 0.0660340 secs] [Times: user=0.10 sys=0.01, real=0.06 secs]
    179295.351: [GC [PSYoungGen: 737000K->21960K(753088K)] 1484743K->771194K(2325952K), 0.0559870 secs] [Times: user=0.12 sys=0.00, real=0.05 secs]
    179520.146: [GC [PSYoungGen: 743176K->20462K(753344K)] 1492410K->777532K(2326208K), 0.0626870 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    179804.786: [GC [PSYoungGen: 740974K->18926K(753472K)] 1498044K->780707K(2326336K), 0.0643720 secs] [Times: user=0.08 sys=0.00, real=0.07 secs]
    179976.170: [GC [PSYoungGen: 739438K->14462K(755392K)] 1501219K->777768K(2328256K), 0.0541500 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
    180175.216: [GC [PSYoungGen: 737470K->21412K(754048K)] 1500776K->786328K(2326912K), 0.0558660 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    180360.592: [GC [PSYoungGen: 744420K->16221K(756288K)] 1509336K->788910K(2329152K), 0.0560110 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    180602.350: [GC [PSYoungGen: 740957K->19309K(754880K)] 1513646K->794725K(2327744K), 0.0586460 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    180742.361: [GC [PSYoungGen: 744045K->13371K(757376K)] 1519461K->792741K(2330240K), 0.0526240 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    180989.546: [GC [PSYoungGen: 740987K->22672K(750336K)] 1520357K->803652K(2323200K), 0.0596360 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    181239.247: [GC [PSYoungGen: 750288K->21320K(755712K)] 1531268K->808769K(2328576K), 0.0604200 secs] [Times: user=0.13 sys=0.01, real=0.06 secs]
    181442.020: [GC [PSYoungGen: 746952K->18413K(756032K)] 1534401K->810721K(2328896K), 0.0584520 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    181670.994: [GC [PSYoungGen: 744045K->16896K(758080K)] 1536353K->813426K(2330944K), 0.0578350 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    181820.892: [GC [PSYoungGen: 745536K->14644K(756992K)] 1542066K->812729K(2329856K), 0.0542340 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    182017.090: [GC [PSYoungGen: 743284K->19813K(758784K)] 1541369K->819602K(2331648K), 0.0609370 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    182190.713: [GC [PSYoungGen: 750821K->16901K(758656K)] 1550610K->822559K(2331520K), 0.0607650 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    182342.243: [GC [PSYoungGen: 747901K->15530K(760256K)] 1553559K->824962K(2333120K), 0.0543110 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    182497.174: [GC [PSYoungGen: 749034K->14097K(759680K)] 1558466K->826597K(2332544K), 0.0584360 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    182728.826: [GC [PSYoungGen: 747601K->18455K(760960K)] 1560101K->832416K(2333824K), 0.0584340 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    182828.746: [GC [PSYoungGen: 753623K->18529K(760640K)] 1567584K->835130K(2333504K), 0.0569770 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    182929.492: [GC [PSYoungGen: 753697K->19910K(760832K)] 1570298K->845289K(2333696K), 0.0514600 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
    183078.455: [GC [PSYoungGen: 755526K->14604K(761024K)] 1580905K->849513K(2333888K), 0.0622210 secs] [Times: user=0.14 sys=0.00, real=0.07 secs]
    183356.799: [GC [PSYoungGen: 750220K->19573K(761216K)] 1585129K->857019K(2334080K), 0.0619630 secs] [Times: user=0.11 sys=0.00, real=0.07 secs]
    183552.428: [GC [PSYoungGen: 755701K->16213K(761280K)] 1593147K->855558K(2334144K), 0.0610980 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    183664.043: [GC [PSYoungGen: 752341K->24636K(753344K)] 1591686K->870106K(2326208K), 0.0447730 secs] [Times: user=0.12 sys=0.00, real=0.05 secs]
    183665.642: [GC [PSYoungGen: 753340K->19211K(757568K)] 1598810K->877196K(2330432K), 0.0314950 secs] [Times: user=0.11 sys=0.00, real=0.03 secs]
    183848.953: [GC [PSYoungGen: 747915K->15974K(758848K)] 1605900K->910927K(2331712K), 0.0670070 secs] [Times: user=0.16 sys=0.00, real=0.06 secs]
    184150.151: [GC [PSYoungGen: 746790K->19652K(758400K)] 1641743K->918013K(2331264K), 0.0478680 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
    184500.743: [GC [PSYoungGen: 750468K->27131K(752256K)] 1648829K->928755K(2325120K), 0.0735960 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    184757.704: [GC [PSYoungGen: 752251K->28336K(753472K)] 1653875K->932726K(2326336K), 0.0665640 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    185062.454: [GC [PSYoungGen: 753456K->21305K(754624K)] 1657846K->936639K(2327488K), 0.0650190 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    185431.938: [GC [PSYoungGen: 743225K->22919K(753728K)] 1658559K->940469K(2326592K), 0.0623440 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    185685.443: [GC [PSYoungGen: 744839K->23275K(755136K)] 1662389K->942751K(2328000K), 0.0623070 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    185973.127: [GC [PSYoungGen: 746923K->20068K(754944K)] 1666399K->946132K(2327808K), 0.0614850 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    186345.372: [GC [PSYoungGen: 743716K->23743K(755840K)] 1669780K->952555K(2328704K), 0.0582290 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    186592.117: [GC [PSYoungGen: 749119K->22847K(755904K)] 1677931K->954500K(2328768K), 0.0570940 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    186818.589: [GC [PSYoungGen: 748221K->16704K(756288K)] 1679874K->955248K(2329152K), 0.0585710 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    187175.297: [GC [PSYoungGen: 742784K->22755K(756224K)] 1681328K->963223K(2329088K), 0.0639720 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    187465.831: [GC [PSYoungGen: 748835K->25353K(755136K)] 1689303K->967590K(2328000K), 0.0666820 secs] [Times: user=0.12 sys=0.01, real=0.06 secs]
    187775.141: [GC [PSYoungGen: 750601K->20930K(755840K)] 1692838K->970886K(2328704K), 0.0634260 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    188090.887: [GC [PSYoungGen: 746178K->21017K(757376K)] 1696134K->973498K(2330240K), 0.0610100 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    188381.676: [GC [PSYoungGen: 748633K->27046K(754688K)] 1701114K->982069K(2327552K), 0.0651250 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    188699.880: [GC [PSYoungGen: 754662K->20688K(756672K)] 1709685K->985011K(2329536K), 0.0710230 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    188874.106: [GC [PSYoungGen: 747024K->14212K(756096K)] 1711347K->980420K(2328960K), 0.0533980 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    189099.700: [GC [PSYoungGen: 740548K->20314K(756928K)] 1706756K->988045K(2329792K), 0.0612750 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    189386.756: [GC [PSYoungGen: 746970K->21879K(756160K)] 1714701K->994866K(2329024K), 0.0592670 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    189776.935: [GC [PSYoungGen: 748486K->23131K(757504K)] 1721473K->1000988K(2330368K), 0.0628120 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    190049.181: [GC [PSYoungGen: 751643K->25024K(753600K)] 1729500K->1004610K(2326464K), 0.0636810 secs] [Times: user=0.12 sys=0.01, real=0.06 secs]
    190346.053: [GC [PSYoungGen: 753536K->22610K(757312K)] 1733122K->1009329K(2330176K), 0.0518250 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    190607.002: [GC [PSYoungGen: 750354K->19608K(756864K)] 1737073K->1010774K(2329728K), 0.0617870 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    190863.501: [GC [PSYoungGen: 747291K->19780K(758080K)] 1738457K->1013483K(2330944K), 0.0542480 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    191167.603: [GC [PSYoungGen: 749124K->27712K(757120K)] 1742827K->1026086K(2329984K), 0.0801490 secs] [Times: user=0.14 sys=0.01, real=0.08 secs]
    191437.558: [GC [PSYoungGen: 757056K->17337K(755840K)] 1755430K->1024744K(2328704K), 0.0562960 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    191753.680: [GC [PSYoungGen: 743865K->24194K(750784K)] 1751272K->1032860K(2323648K), 0.0619210 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    192084.908: [GC [PSYoungGen: 750722K->23247K(756608K)] 1759388K->1037767K(2329472K), 0.0640770 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    192219.906: [GC [PSYoungGen: 749711K->17614K(756288K)] 1764231K->1036481K(2329152K), 0.0585380 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    192257.388: [GC [PSYoungGen: 744078K->9925K(754368K)] 1762945K->1034247K(2327232K), 0.0494430 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
    192651.017: [GC [PSYoungGen: 734533K->26479K(751104K)] 1758855K->1053097K(2323968K), 0.0664500 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    192888.949: [GC [PSYoungGen: 751087K->19520K(755136K)] 1777705K->1051007K(2328000K), 0.0613230 secs] [Times: user=0.12 sys=0.01, real=0.06 secs]
    193038.298: [GC [PSYoungGen: 742528K->14952K(754304K)] 1774015K->1050505K(2327168K), 0.0558750 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    193250.873: [GC [PSYoungGen: 737960K->16648K(755712K)] 1773513K->1055074K(2328576K), 0.0588400 secs] [Times: user=0.11 sys=0.00, real=0.06 secs]
    193477.482: [GC [PSYoungGen: 741128K->16286K(755200K)] 1779554K->1056900K(2328064K), 0.0587210 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    193692.917: [GC [PSYoungGen: 740766K->22970K(755776K)] 1781380K->1065113K(2328640K), 0.0615360 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    194089.979: [GC [PSYoungGen: 748474K->26159K(751680K)] 1790617K->1075330K(2324544K), 0.0709660 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    194304.937: [GC [PSYoungGen: 751663K->17900K(755392K)] 1800834K->1070835K(2328256K), 0.0614510 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    194463.802: [GC [PSYoungGen: 741548K->31038K(754688K)] 1794483K->1087130K(2327552K), 0.0503720 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
    194465.309: [GC [PSYoungGen: 754686K->160K(743296K)] 1810778K->1078849K(2316160K), 0.0287160 secs] [Times: user=0.11 sys=0.00, real=0.03 secs]
    194617.418: [GC [PSYoungGen: 711712K->14598K(748992K)] 1790401K->1130171K(2321856K), 0.0582240 secs] [Times: user=0.16 sys=0.00, real=0.06 secs]
    194772.112: [GC [PSYoungGen: 726150K->17256K(751360K)] 1841723K->1135578K(2324224K), 0.0585810 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    194969.203: [GC [PSYoungGen: 732136K->16927K(749952K)] 1850458K->1139782K(2322816K), 0.0621840 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    195240.859: [GC [PSYoungGen: 731807K->19045K(754048K)] 1854662K->1144321K(2326912K), 0.0640860 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
    195424.009: [GC [PSYoungGen: 739365K->19195K(752704K)] 1864641K->1146835K(2325568K), 0.0574680 secs] [Times: user=0.12 sys=0.01, real=0.05 secs]
    195628.228: [GC [PSYoungGen: 739515K->19259K(756288K)] 1867155K->1152218K(2329152K), 0.0612200 secs] [Times: user=0.13 sys=0.00, real=0.06 secs]
    195892.431: [GC [PSYoungGen: 744315K->18878K(755200K)] 1877274K->1156439K(2328064K), 0.0632670 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    196136.574: [GC [PSYoungGen: 743934K->16848K(758080K)] 1881495K->1156539K(2330944K), 0.0574940 secs] [Times: user=0.12 sys=0.00, real=0.05 secs]
    196280.667: [GC [PSYoungGen: 745872K->18971K(757376K)] 1885563K->1160313K(2330240K), 0.0410340 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
    196465.595: [GC [PSYoungGen: 747995K->16891K(759488K)] 1889337K->1165997K(2332352K), 0.0626150 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
    196655.447: [GC [PSYoungGen: 748859K->15915K(758912K)] 1897965K->1167751K(2331776K), 0.0536770 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
    196885.059: [GC [PSYoungGen: 747872K->18994K(760448K)] 1899708K->1172964K(2333312K), 0.0601520 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
    197057.846: [GC [PSYoungGen: 752882K->16674K(759872K)] 1906852K->1174415K(2332736K), 0.0596260 secs]




    brgds,
    Loukas

    Edited by: rouglas4 on Jan 4, 2012 3:41 PM
  • 6. Re: WebLogic Threads Settings and DataSource Connections
    PHCharbonneau Explorer
    Currently Being Moderated
    Hi Loukas,

    Yes, Native Heap capacity for a 32-bit VM is typically (4 GB limit - Xmx - MaxPermSize). As you can see, bigger is the Max Heap size & Max PermGen space, smaller is the Native capacity and smaller the # of Java Threads that can be created by this VM. Typically, once your process size is closing in to 3.5 GB, you will be running dangerously close to ouf of memory addresses and exposed to OurOfMemoryError for native operations.

    PermGen space is applicable for HotSpot VM only and is used mainly for class metadata caching and some other optimizations. Native heap is required for Java Threads creation and other native VM operations.

    In your case, you have both a big Java Heap (2304 MB) and PermGen space (768 MB) which is leaving very small room for Native Heap and Thread creation. This is your problem.
    The GC data output is very useful and yes please include a Full GC, memory spaces post Full GC is what you need to determine how you can tune your Java Heap in a safely manner. Verbose GC and post Full GC will also give you the PermGen space footprint.

    Please look at options and opportunities of reducing both Max Java Heap and PermGen by 256 MB.

    Thanks.
    P-H
  • 7. Re: WebLogic Threads Settings and DataSource Connections
    599962 Newbie
    Currently Being Moderated
    Hi P-H,

    your explanaion is really helpful. We will plan to descrease the 2 paramters next week, since we are in a freeze right now (no changes allowed on PRD).
    The memory allocation of our 2 soa processes (on 2 different physical machines - today we started again the 2nd since it was down for functional issues with EDN) have more or less the same size:

    soa_server1 3542868
    soa_server2 3460392

    So we are close to the 3.5GB you mentioned but fortunately the load is spread now and the heap size is really not used more than 1GB.

    No, another question about the thread monitoring. I know that most of the info can be found in the generated thread dumps , as soon as we will put it in cron with kill -3.
    Is there any other way to monitor and maybe set alarms about the thread count, before we examine the option of the work managers?

    During the investigations for this issue we got also info that the issue could be related to the stack size the Linux OS allocates for the threads. This can be found by ulimit -s and in our case is 10MB. But I want to ask you, if this meant to be physical memory which is remaining if we calculate (total physical memory - processes memory - I/O-Caching memory used by Linux). That being said , I understood that each thread in the JVM maps an address space (memory) on the rest physical RAM and it can come to a stuck in case the max number of threads is reached in the physical memory outside the JVMs.
    You feedback on this would also be very welcome.

    Last but not least a question about the logging. By activating the kill -3 everything will be directed to the .out file . In our case we have the default like soa_server1.out which is also collecting info about the status of the service. Willn't be difficult to split this file whenever we will have an issue in order to abstract the lines for the dump?
    Can we define a dedicated file for the dump?

    When I check in the logging tab of the soa_server1 I see that the following are not selected:

    Redirect stdout logging enabled
    Redirect stderr logging enabled

    and stdout Format is standard. Should we take some actions in this regards to have more informative out files?

    Thanks in advance,
    Loukas
  • 8. Re: WebLogic Threads Settings and DataSource Connections
    907419 Newbie
    Currently Being Moderated
    Hi Loukas,

    You may use the following flag to define the no. of threads in a weblogic instance or else you can look into an option of configuring work manager:
    -Dweblogic.threadpool.MinPoolSize=xx(numeric)

    Relation of threads with Datasource:

    Well so far i have seen the below cases where the above 2 are inter-related:

    1) it would be a good idea to have thread count close to the connection pool size, though it depends how is the load in your case and you may not want to have too many threads as well.

    2) "Pinned to Thread" option in datasource and this is a good option and improves the system performance and you may want to look into detail for this option.

    let me know if this helps, Rest i believe you already have a solution for the error which you got.

    -Rohit
  • 9. Re: WebLogic Threads Settings and DataSource Connections
    RenévanWijk Oracle ACE
    Currently Being Moderated
    Some explanation about the inner workings of the JVM can be found here: http://middlewaremagic.com/weblogic/?p=6930

    JVM and operating system tuning can be found here: http://middlewaremagic.com/weblogic/?p=7083 (look in the system section)
    More on operating system tuning can be found here: http://middlewaremagic.com/weblogic/?p=7737

    Maybe the posts give you some idea on how to proceed.
  • 10. Re: WebLogic Threads Settings and DataSource Connections
    599962 Newbie
    Currently Being Moderated
    Hello Rene,
    I am aware of your web site and I had already checked the second link.
    Now I am going to check the other 2 to get a better overview.
    Many thanks.
    Loukas

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points