Create (Boot) Order Using The Delegated Zone Restarter

Version 1

    Introduction

    On modern systems the chances are very high that they are running consolidated workloads. In addition applications are becoming more and more complex using multiple elements running in multiple virtual environments as one cohesive whole. However in such a situation boot order of these elements can also be important as you many want to wait with starting certain applications only if others are already running.

     

    To help address this Oracle Solaris 11.4 now introduces a new technology called Delegated Zones Restarter (DZR), which allows the administrator to set dependencies between Oracle Solaris Zones and even have the booting of one zone depend on an application running with another zone. This is now in part possible because of the introduction of Goal Services and now Oracle Solaris Zones are part of the Oracle Solaris Service Management Facility (SMF). For more specific information see the man page sec.zones(8).

     

    In short, with the goal services you can define when you deem a zone to be fully booted when the service milestone/goals is online. You can add dependencies to other services to the milestone/goals service, for example on your application, and once everything is running the SMF service representing the zone in the global zone is online too. Buy then making the SMF service of another zone dependent on the first zone, this second zone will only start booting once the application in the first zone is online.

     

    To be clear this is only for the boot order at boot time of the global zone. An administrator can always boot zones by hand if they want to.

     

    Note: This works for both Native Zones and Kernel Zones as long as they're running Oracle Solaris 11.4.

     

    Overview

    In this article we'll build this in steps. We'll be using the environment as set up - shown Figure 1 - in the OTN article on how to install Oracle Solaris Zones, so if you don't have an environment with zones installed yet you can use that as a guide how to build it. It has a zone called testzone and two webzones, and we can simulate that the application running in the testzone requires a web server to be running.

     

    OTN_Pictures_DZR-1.png

    Figure 1: The general setup.

     

    First we'll have the testzone simply dependent on an Apache Webserver service running in the global zone, then we'll show how to change that dependency to one of the webzones, with the presumption that the zone is running means the webserver is up. Next we'll show how include the dependency on the webserver in the webzone. And then finally we'll show how you can set the dependencies on the testzone such that it will boot if either webzone is running.

     

    Setting a Zone Dependency on a Service

    The core task of the Delegated Zone Restarter is to boot zones in the correct order at time of boot or reboot. This means that for this feature to take effect you will need to set autoboot to true in the zonecfg. So in this case we'll need to confirm that autoboot is set for the testzone so it looks at it's dependencies, and not set for the webzones because we want them to delay the boot until we boot them by hand.

     

    OTN_Pictures_DZR-2.png

    Figure 2: Making the testzone dependent on a service in the global zone.

     

    First we're going to show how you can make the booting of the zone dependent on one or more services in the global zone, as shown in Figure 2. First we need to check if the global zone has the Apache Webserver installed, it should buy default but always good to validate:

    root@global:~# pkg info apache-24
              Name: web/server/apache-24
           Summary: Apache Web Server V2.4
       Description: The Apache HTTP Server Version 2.4
          Category: Web Services/Application and Web Servers
             State: Installed
         Publisher: solaris
           Version: 2.4.33
            Branch: 11.4.0.0.1.9.0
    Packaging Date: June 18, 2018 at  6:41:14 PM
              Size: 29.06 MB
              FMRI: pkg://solaris/web/server/apache-24@2.4.33-11.4.0.0.1.9.0:20180618T184114Z
       Project URL: https://httpd.apache.org/
        Source URL: https://archive.apache.org/dist/httpd/httpd-2.4.33.tar.gz

     

    Next we can set the dependency:

    root@global:~# zonecfg -z testzone
    zonecfg:testzone> add smf-dependency
    zonecfg:testzone:smf-dependency> set fmri=svc:/network/http:apache24
    zonecfg:testzone:smf-dependency> end
    zonecfg:testzone> exit
    root@global:~# svcs -l testzone
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        online
    next_state   none
    state_time   Wed Jun 27 21:42:27 2018
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/network/http:apache24 (disabled)

     

    Note the dependency on http:apache24 that has now been added to the zone service.

     

    Now is we reboot the global zone, the testzone will not boot until we enable the http:apache24 service:

    root@global:~# reboot
    reboot: Halting 1 zone.
     
    ...
     
    root@global:~# zoneadm list -cv
      ID NAME             STATUS      PATH                         BRAND      IP    
       0 global           running     /                            solaris    shared
       - testzone         installed   /system/zones/testzone       solaris    excl  
       - webzone-1        installed   /system/zones/webzone-1      solaris    excl  
       - webzone-2        installed   /system/zones/webzone-2      solaris    excl  
    root@global:~# svcs -l testzone
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   none
    state_time   July 10, 2018 at  1:23:22 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/network/http:apache24 (disabled)
    root@global:~# svcadm enable http:apache24
    root@global:~# svcs -l testzone
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   online
    state_time   July 10, 2018 at  1:23:22 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/network/http:apache24 (online)

    ...

    root@global:~# svcs -l testzone
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        online
    next_state   none
    state_time   July 10, 2018 at  1:24:54 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/network/http:apache24 (online)

     

    As you can see, after the reboot initially the zone service is enabled but not online waiting for the http:apache24 service to come online. Once we enable this service the zone service next_state transitions to online and a little later the state is now online too.

     

    Note that you can set multiple dependencies for the single zone, something we'll use at the end of this article.

     

    Setting a Dependency on Another Zone

    In this step we replace the previous dependency with one on webzone-1 as shown in Figure 3.

     

    OTN_Pictures_DZR-3.png

    Figure 3: Making the testzone dependent on webzone-1.

     

    You can do this by simply setting a new FMRI:

    root@global:~# zonecfg -z testzone
    zonecfg:testzone> select smf-dependency 0
    zonecfg:testzone:smf-dependency> info
    smf-dependency 0:
            fmri: svc:/network/http:apache24
    zonecfg:testzone:smf-dependency> set name=web
    zonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-1
    zonecfg:testzone:smf-dependency> info
    smf-dependency 0:
            name: web
            fmri: svc:/system/zones/zone:webzone-1
    zonecfg:testzone> exit

     

    Note we're also giving the dependency the name web, this is mostly for later when we want to group dependencies. In general it's good make sure you set a name on the dependency when you first create it.

     

    Now we reboot the zone and then boot webzone-1 while checking for the status on testzone:

    root@global:~# reboot
    reboot: Halting 1 zone.

    ...
     
    root@global:~# zoneadm list -cv
      ID NAME             STATUS      PATH                         BRAND      IP    
       0 global           running     /                            solaris    shared
       - testzone         installed   /system/zones/testzone       solaris    excl  
       - webzone-1        installed   /system/zones/webzone-1      solaris    excl  
       - webzone-2        installed   /system/zones/webzone-2      solaris    excl  
    root@global:~# svcs -l testzone
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   none
    state_time   July 10, 2018 at  2:31:00 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/system/zones/zone:webzone-1 (disabled)
    root@global:~# zoneadm -z webzone-1 boot
    root@global:~# svcs -l testzone webzone-1
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   online
    state_time   July 10, 2018 at  2:31:00 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      true (temporary)
    state        online
    next_state   none
    state_time   July 10, 2018 at  3:11:06 AM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

    ...

    root@global:~# svcs -l testzone webzone-1
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        online
    next_state   none
    state_time   July 10, 2018 at  3:11:25 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      true (temporary)
    state        online
    next_state   none
    state_time   July 10, 2018 at  3:11:06 AM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

     

    So once the webzone-1 is online, the testzone starts booting.

     

    Appending the Goal Service

    As stated above the state of the service representing the zone in the global zone is linked to the goal service milestone/goals running inside the zone.

     

    OTN_Pictures_DZR-4.png

    Figure 4: Setting the goal service up to be dependent on a service.

     

    So, as shown in Figure 4, if you want make this dependent on specific services inside the zone you'll need to change the goals for this service:

    root@global:~# zlogin webzone-1
    [Connected to zone 'webzone-1' pts/1]
    Last login: Wed Jun 27 23:26:13 2018 on pts/4
    Oracle Corporation      SunOS 5.11      Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07  June 2018
    root@webzone-1:~# svcs -l apache24
    fmri         svc:/network/http:apache24
    name         Apache 2.4 HTTP server
    enabled      false
    state        disabled
    next_state   none
    state_time   July 10, 2018 at  3:16:23 AM PDT
    logfile      /var/svc/log/network-http:apache24.log
    restarter    svc:/system/svc/restarter:default
    manifest     /lib/svc/manifest/network/http-apache24.xml
    dependency   optional_all/error svc:/system/filesystem/autofs:default (online)
    dependency   require_all/none svc:/system/filesystem/local:default (online)
    dependency   require_all/error svc:/milestone/network:default (online)
    root@webzone-1:~# svcs -l goals
    fmri         svc:/milestone/goals:default
    name         goals services for the system
    enabled      true
    state        online
    next_state   none
    state_time   July 10, 2018 at  3:16:32 AM PDT
    logfile      /var/svc/log/milestone-goals:default.log
    restarter    svc:/system/svc/restarter:default
    manifest     /lib/svc/manifest/milestone/goals.xml
    dependency   require_all/restart svc:/milestone/multi-user-server (online)
    root@webzone-1:~# svcadm goals svc:/network/http:apache24
    root@webzone-1:~# svcs -l goals
    fmri         svc:/milestone/goals:default
    name         goals services for the system
    enabled      true
    state        online
    next_state   none
    state_time   July 10, 2018 at  3:16:32 AM PDT
    logfile      /var/svc/log/milestone-goals:default.log
    restarter    svc:/system/svc/restarter:default
    manifest     /lib/svc/manifest/milestone/goals.xml
    dependency   require_all/restart svc:/network/http:apache24 (disabled)
    root@webzone-1:~# exit
    logout

    [Connection to zone 'webzone-1' pts/1 closed]

     

    At this point the goal service milestone/goals is now set to only go online once the http:apache24 service is up and running.

     

    Now if we reboot the global zone again we can see that booting the testzone waits until the http:apache24 service is up (which we didn't enable yet):

    root@global:~# reboot
    reboot: Halting 2 zones.

    ...

    root@global:~# zoneadm list -cv
      ID NAME             STATUS      PATH                         BRAND      IP    
       0 global           running     /                            solaris    shared
       - testzone         installed   /system/zones/testzone       solaris    excl  
       - webzone-1        installed   /system/zones/webzone-1      solaris    excl  
       - webzone-2        installed   /system/zones/webzone-2      solaris    excl  
    root@global:~# zoneadm -z webzone-1 boot
    root@global:~# zlogin webzone-1
    [Connected to zone 'webzone-1' pts/1]
    Last login: Tue Jul 10 19:30:23 2018 on pts/1
    NOTE: system has 1 active defect; run 'fmadm list' for details.
    Oracle Corporation      SunOS 5.11      Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07  June 2018
    You have new mail.
    root@webzone-1:~# svcs -xv 
    svc:/network/http:apache24 (Apache 2.4 HTTP server)
    State: disabled since July 10, 2018 at  7:33:31 PM PDT
    Reason: Disabled by an administrator.
       See: http://support.oracle.com/msg/SMF-8000-05
       See: http://httpd.apache.org
       See: man -M /usr/apache2/2.4/man -s 8 httpd
       See: /var/svc/log/network-http:apache24.log
    Impact: 1 dependent service is not running:
            svc:/milestone/goals:default

    svc:/milestone/goals:default (goals services for the system)
    State: maintenance since July 10, 2018 at  7:33:49 PM PDT
    Reason: goal service has dependencies that cannot be satisfied without administrative intervention.
    Reason: Service svc:/network/http:apache24 is disabled.
       See: http://support.oracle.com/msg/SMF-8000-GE
      Path: svc:/milestone/goals:default
              svc:/network/http:apache24
       See: http://support.oracle.com/msg/(null)
       See: man -M /usr/share/man -s 7 smf
       See: man -M /usr/share/man -s 8 svcadm
       See: /var/svc/log/milestone-goals:default.log
    Impact: This service is not running.
    root@webzone-1:~# svcadm enable http:apache24
    root@webzone-1:~# exit
    logout

    [Connection to zone 'webzone-1' pts/1 closed]
    You have new mail in /var/mail/root
    root@global:~# svcs -l testzone webzone-1
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        online
    next_state   none
    state_time   July 10, 2018 at  7:36:03 PM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      true (temporary)
    state        online
    next_state   none
    state_time   July 10, 2018 at  7:35:44 PM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

     

    You can see that both zones were in the installed state, testzone because it's waiting for webzone-1's goal service and webzone-1 because it doesn't have autoboot enabled. Once we boot webzone-1 and log in we can see the system warns that the milestone/goals service dependencies are not being met because the http:apache24 service isn't enabled. Once we enable that, the milestone is met and the testzone boots.

     

    Setting Multiple Dependencies

    Now let's say, as shown in Figure 5, the testzone is dependent on any Apache Webserver it can see, and it doesn't matter which one is up first. In other words, it doesn't care which of the webzones is up as long as one of them is.

     

    OTN_Pictures_DZR-5.png

    Figure 5: Making the testzone dependent in either webzone-1 or webzone-2.

     

    To achieve this you only have to make some minor changes to the zone configuration:

    root@global:~# zonecfg -z testzone
    zonecfg:testzone> info
    zonename: testzone
    brand: solaris
    autoboot: true
    bootargs: -m verbose
    anet 0:
            linkname: net0
            configure-allowed-address: true
    smf-dependency 0:
            name: web
            fmri: svc:/system/zones/zone:webzone-1
    zonecfg:testzone> select smf-dependency 0
    zonecfg:testzone:smf-dependency> set grouping=require_any
    zonecfg:testzone:smf-dependency> info
    smf-dependency 0:
            name: web
            fmri: svc:/system/zones/zone:webzone-1
            grouping: require_any
    zonecfg:testzone:smf-dependency> end
    zonecfg:testzone> add smf-dependency
    zonecfg:testzone:smf-dependency> set name=web
    zonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-2
    zonecfg:testzone:smf-dependency> set grouping=require_any
    zonecfg:testzone:smf-dependency> info
    smf-dependency 1:
            name: web
            fmri: svc:/system/zones/zone:webzone-2
            grouping: require_any
    zonecfg:testzone:smf-dependency> end
    zonecfg:testzone> exit

     

    By adding a second dependency, and also calling this dependency web, and setting the grouping to require_any for both dependencies they will be grouped together as one combined dependency, and the zone should now boot when either zone is booted.

     

    Note that we're not changing the goal service milestone/goals inside webzone-2, we're assuming you can do this if you want to.

     

    Now we reboot, and enable/boot one of the two webzones:

    root@global:~# zoneadm list -cv
      ID NAME             STATUS      PATH                         BRAND      IP    
       0 global           running     /                            solaris    shared
       - testzone         installed   /system/zones/testzone       solaris    excl  
       - webzone-1        installed   /system/zones/webzone-1      solaris    excl  
       - webzone-2        installed   /system/zones/webzone-2      solaris    excl  
    root@global:~# svcs -l testzone webzone-1 webzone-2
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   none
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_any/none svc:/system/zones/zone:webzone-2 (disabled) svc:/system/zones/zone:webzone-1 (disabled)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      false
    state        disabled
    next_state   none
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

    fmri         svc:/system/zones/zone:webzone-2
    name         Solaris Zone
    enabled      false
    state        disabled
    next_state   none
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/webzone-2.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    root@global:~# zoneadm -z webzone-2 boot
    root@global:~# svcs -l testzone webzone-1 webzone-2
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        offline
    next_state   online
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      false
    state        disabled
    next_state   none
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

    fmri         svc:/system/zones/zone:webzone-2
    name         Solaris Zone
    enabled      true (temporary)
    state        online
    next_state   none
    state_time   July 11, 2018 at  1:36:52 AM PDT
    logfile      /var/log/zones/webzone-2.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

    ...

    root@global:~# svcs -l testzone webzone-1 webzone-2
    fmri         svc:/system/zones/zone:testzone
    name         Solaris Zone
    enabled      true
    state        online
    next_state   none
    state_time   July 11, 2018 at  1:37:10 AM PDT
    logfile      /var/log/zones/testzone.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    dependency   require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled)

    fmri         svc:/system/zones/zone:webzone-1
    name         Solaris Zone
    enabled      false
    state        disabled
    next_state   none
    state_time   July 11, 2018 at  1:08:59 AM PDT
    logfile      /var/log/zones/webzone-1.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml

    fmri         svc:/system/zones/zone:webzone-2
    name         Solaris Zone
    enabled      true (temporary)
    state        online
    next_state   none
    state_time   July 11, 2018 at  1:36:52 AM PDT
    logfile      /var/log/zones/webzone-2.messages
    restarter    svc:/system/zones:default
    manifest     /lib/svc/manifest/system/zones.xml
    root@global:~# zoneadm list -cv
      ID NAME             STATUS      PATH                         BRAND      IP    
       0 global           running     /                            solaris    shared
       1 webzone-2        running     /system/zones/webzone-2      solaris    excl  
       2 testzone         running     /system/zones/testzone       solaris    excl  
       - webzone-1        installed   /system/zones/webzone-1      solaris    excl 

     

    As you can see, after the reboot the testzone is waiting for either webzone. And once we boot webzone-2 it takes a little bit of time for it to complete it's milestones/goals goal service and once it does testzone boots.

     

    Note that if we hadn't set the grouping to require_any SMF would have defaulted to require_all, so if that's what you want you can just omit that from the definition of the dependencies.

     

    Conclusion

    This concludes the examples in this article, here are a few additional thoughts. First, these tools allow you to easily and quickly create inter-zone dependencies to fit most of your needs. And this is especially handy in cases where a lot of smaller services are layered and dependent on each other, or if one service coming online before others is an issue.

     

    Second, it's important to note that at this point this is single node only. So if you migrate a zone to another system and you reboot the system, if another zone is dependent on it this zone will not boot until the dependency is met. You can of course still boot the zone by hand.

     

    Last, if you give the dependencies different names, let's say you give two dependencies the name web and two others the name firewall, and you give them all grouping=require_any, this will a create a logical AND of two ORs. That is to say, the zone will boot if either of the dependencies in with the web name are met, and either of the dependencies with the firewall name are met. It could be that you only need one webserver to be up and one firewall.