Using zonestat to Monitor Oracle Solaris Zones

Version 4

    by Marc Nesello

     

    Introduction

    In this article I want to give a short overview of the monitoring tool zonestat, which can be used to monitor the zones in Oracle Solaris 11. This article will discuss the monitoring of CPU resources and memory resources of native Solaris zones. Monitoring zones is important as this will help you to find out which zones need more resources and which zones barely utilize the resource assigned to them. That way you can ensure that the resources of the server are distributed well among the zones.

     

    You can either use zonestat in the global zone to get an overview of all zones on the server or you can use zonestat from within the non-global zone to observe the current zone. The command zonestat offers many reports to choose from. For example, reports about processes, memory, CPUs, network and shared memory. This article discusses two reports that monitor the usage of the CPU and memory. These reports are called processor-set and memory.

     

    When using the zonestat or libzonestat commands, ensure that the service system/zones-monitoring:default is running in the global zone, so that the zonestat or libzonestat commands are able to monitor the zones.

     

    Monitoring Zones with Dedicated CPUs

    Imagine you have a native zone called myzone in Oracle Solaris 11 which is configured with the dedicated-cpu resource with 1 to 2 CPUs. This means that Oracle Solaris 11 creates a resource pool specifically for the zone and assigns 1 to 2 CPUs to this pool.

     

    You can get an overview about the usage of CPU resources by using the report processor-set. Just call zonestat -r processor-set 60 1. This will display the report processor-set once after one minute. If you call zonestat from the global zone, the report will look like this:

     

    -bash-4.1$ zonestat -r processor-set 60 1

    Collecting data for first interval...

    Interval: 1, Duration: 0:01:00

    PROCESSOR_SET                   TYPE  ONLINE/CPUS     MIN/MAX              

    pset_default            default-pset          2/2         1/-

                                    ZONE USED %USED   CAP  %CAP SHRS  %SHR %SHRU

                                 [total]  0.12 6.30% -     -      - -     -

                                [system]  0.00 0.38% -     -      - -     -

                                  global  0.11 5.92% -     -      - -     -

     

    PROCESSOR_SET                   TYPE  ONLINE/CPUS     MIN/MAX

    myzone                 dedicated-cpu          2/2         1/2

                                    ZONE  USED %USED CAP  %CAP   SHRS %SHR %SHRU

                                 [total]  1.00 50.1% -     -      - -     -

                                [system]  0.00 0.00% -     -      - -     -

                                  myzone  1.00 50.1% -     -      - -     -

     

     

    The first block shows statistics of the default processor set, which is only used by the global zone in this case. The second block shows statistics of the processor-set assigned to the zone myzone.

     

    In the second block you can see the minimum, maximum, and currently assigned CPUs to this processor set, which are 2. Then three lines starting with [total], [system], and myzone follow, and they display the CPU usage (in percentage and number of CPUs). The line starting with [system] usually describes the usage of this resource (CPU) by other zones or the global zone. But, because the dedicated-cpu resource is used, the processor set is exclusively assigned to the zone. No other zones or the global zone use this processor set, and [system] is 0 in this case. After the line beginning with [system] there is a line for each zone using this resource. Again, this processor set is used only by this zone, so only the zone (myzone) is displayed here, showing that it uses 1.00 CPUs or almost 50.1% percent of the CPUs in the processor set. The [total] line is the sum of the [system] and myzone lines.

     

    You also can call zonestat in the nonglobal zone. In this case the report only shows statistics of the processor set that is used by the zone.

     

    Monitoring Zones with Capped CPU

    If myzone has not dedicated CPUs, but a CPU cap, zonestat is also a way to observe the cap.

    You can use the same command zonestat -r processor-set 60 1 for this. Running it from a global zone, the output will look like this:

     

    -bash-4.1$ zonestat -r processor-set 60 1

    Collecting data for first interval...

    Interval: 1, Duration: 0:01:00

    PROCESSOR_SET                   TYPE  ONLINE/CPUS     MIN/MAX

    pset_default            default-pset          4/4         1/-

                                    ZONE  USED %USED CAP  %CAP   SHRS %SHR %SHRU

                                 [total]  1.20 30.1% -     -      - -     -

                                [system]  0.01 0.33% -     -      - -     -

                                  myzone  0.99 24.9% 1.50 66.5%      -     - -

                                  global  0.14 3.63% -     -      - -     -

                               solmarc10  0.04 1.19% 1.50 3.19%      -     - -

     

    From the global zone all zones are displayed which use a certain processor set. In this case the global zone and the zones myzone and solmarc10 use the default processor set. You can observe the caps and CPU usage in the above output. When running this from within a non-global zone only the CPU cap of the current zone is displayed, and the CPU usage of all other zones is summarized in the line [system].

     

     

    Monitoring Zones with Capped Memory

    To observe memory resource usage the zonestat report memory can be used. In this case the zone myzone is configured with a physical memory cap of 2GB and the zone solmarc10 is configured with a physical memory cap of 1 GB. When calling zonestat -r memory 60 1 from the global zone, following output is shown:

     

    -bash-4.1$ zonestat -r memory 60 1

    Collecting data for first interval...

    Interval: 1, Duration: 0:01:00

    PHYSICAL-MEMORY              SYSTEM MEMORY

    mem_default                          15.7G

                                    ZONE  USED %USED CAP  %CAP

                                 [total] 4059M 25.1%     -     -

                                [system] 2570M 15.9%     -     -

                                  global 1149M 7.12%     -     -

                                  myzone  193M 1.20% 2048M 9.45%

                               solmarc10  146M 0.90% 1024M 14.2%

     

    VIRTUAL-MEMORY               SYSTEM MEMORY

    vm_default                           23.7G

                                    ZONE  USED %USED CAP  %CAP

                                 [total] 5933M 24.3%     -     -

                                [system] 2911M 11.9%     -     -

                                  global 2710M 11.1%     - -

                                  myzone  169M 0.69% 3072M 5.53%

                               solmarc10  141M 0.58% 1536M 9.23%

     

    LOCKED-MEMORY                SYSTEM MEMORY

    mem_default                          15.7G

                                    ZONE  USED %USED CAP  %CAP

                                 [total] 2727M 16.9%     -     -

                                [system] 2719M 16.8%     -     -

                                  global 7408K 0.04%     -     -

                                  myzone     0 0.00% -     -

                               solmarc10     0 0.00% -     -

     

    For each memory type a block is displayed. The first block shows statistics about physical memory. The column SYSTEM MEMORY shows the physical memory installed in the system. The second block is virtual memory, which means physical memory + swap files. In this case you can observe that both zones have a cap set. The third block is locked memory, for which no cap was set in the installed zones. Running this report from the non-global zone will only display memory usage of the current zone and sums the memory usage of other zones and the global zone in the line [system].

     

    There is also an API available to use the zonestat command functionality within programs. This is a great way for ISVs to monitor resource usage of the zone within their applications. This API is documented in the man page for the command libzonestat: http://docs.oracle.com/cd/E36784_01/html/E36873/libzonestat-3lib.html

     

    See Also

    To learn more about virtualization monitoring in Oracle Solaris Zones, you can view the complete white paper on the Oracle Technology Network: http://www.oracle.com/technetwork/server-storage/solaris11/documentation/virt-monitoring-solaris-zones-2660307.pdf

     

    About the Author

    Marc Nesello is a Senior Software Engineer at ISV Engineering in Germany. He is part of the team working with SAP, where he works with SAP on topics related with virtualization, Oracle Solaris monitoring or the Oracle Solaris Studio compiler.