This discussion is archived
11 Replies Latest reply: Jul 24, 2012 3:24 AM by 935139 RSS

Solaris 10 JET install and ZFS

JulianG Newbie
Currently Being Moderated
Hi - so following on from Solaris Volume Manager or Hardware RAID? - I'm trying to get my client templates switched to ZFS but it's failing with:


sudo ./make_client -f build1.zfs
Gathering network information..
Client: xxx.14.80.196 (xxx.14.80.0/255.255.252.0)
Server: xxx.14.80.199 (xxx.14.80.0/255.255.252.0, SunOS)
Solaris: client_prevalidate
Clean up /etc/ethers
Solaris: client_build
Creating sysidcfg
WARNING: no base_config_sysidcfg_timeserver specified using JumpStart server
Creating profile
Adding base_config specifics to client configuration
Adding zones specifics to client configuration
ZONES: Using JumpStart server @ xxx.14.80.199 for zones
Adding sbd specifics to client configuration
SBD: Setting Secure By Default to limited_net
Adding jass specifics to client configuration
Solaris: Configuring JumpStart boot for build1.zfs
Solaris: Configure bootparams build
Starting SMF services for JumpStart
Adding Ethernet number for build1 to /etc/ethers
cleaning up preexisting install client "build1"
removing build1 from bootparams
removing /tftpboot/inetboot.SUN4V.Solaris_10-1
svcprop: Pattern 'network/tftp/udp6:default/:properties/restarter/state' doesn't match any entities
enabling network/tftp/udp6 service
svcadm: Pattern 'network/tftp/udp6' doesn't match any instances
updating /etc/bootparams
copying boot file to /tftpboot/inetboot.SUN4V.Solaris_10-1
?
Force bootparams terminal type
-Restart bootparamd
Running '/opt/SUNWjet/bin/check_client build1.zfs'
Client: xxx.14.80.196 (xxx.14.80.0/255.255.252.0)
Server: xxx.14.80.199 (xxx.14.80.0/255.255.252.0, SunOS)
Checking product base_config/solaris
Checking product custom
Checking product zones
Product sbd does not support 'check_client'
Checking product jass
Checking product zfs
WARNING: ZFS: ZFS module selected, but not configured to to anything.
--------------------------------------------------------------
Check of client build1.zfs
-> Passed....


So what is "WARNING: ZFS: ZFS module selected, but not configured to to anything." referring to? I've amended my template and commented out all references to UFS so I now have this:


base_config_profile_zfs_disk="slot0.s0 slot1.s0"
base_config_profile_zfs_pool="rpool"
base_config_profile_zfs_be="BE1"
base_config_profile_zfs_size="auto"
base_config_profile_zfs_swap="65536"
base_config_profile_zfs_dump="auto"
base_config_profile_zfs_compress=""
base_config_profile_zfs_var="65536"


I see there is a zfs.conf file in /opt/SUNWjet/Products/zfs/zfs.conf do I need to edit that as well?

Thanks - J.
  • 1. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    So, a couple of things.

    Firstly, it's just a WARNING, so you can ignore it.

    What it is saying is that you have the JetZFS module in the template, but you have not configured it.

    You only need the "zfs" module if you want to configure anything over and above the rpool. The base_config_* variables will do all the ZFS stuff appropriate to the rpool.

    For tidiness, simply remove zfs from the base_config_products line. (and optionally the section of the template that contains all the variables that begin with "zfs".

    Don't edit the zfs.conf file in the Products/zfs directory, this is the base template that is imported into the client template file when you create a new template and specify that you want zfs.

    You might want to consider leaving the BE name blank, and it'll populate it with the O/S rev, which is useful.

    You might want to consider turning on compression. I've not seen any degradation, and CPUs are faster than disks.

    Hope this helps,
    Mike

    Edited by: mramcha on Jul 20, 2012 1:53 PM
  • 2. Re: Solaris 10 JET install and ZFS
    JulianG Newbie
    Currently Being Moderated
    Thankyou ... a couple more tries and it worked. I just need to get my head around this and how to replicate what we had when using SVM in the past, but the very few benefits that I've seen so far are going to make my life a lot easier I hope!!

    One quick question though is it possible to have this with ZFS:

    /logs = 64G quota
    /logs/zones = 120G quota

    So /logs + /logs/zones treated as two separate file systems with two different quotas being applied?

    Thanks - Julian.
  • 3. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    Yes.

    You need to create a dataset for each.

    so

    zfs create rpool/logs
    zfs set mountpoint=/logs rpool/logs
    zfs set quota=184g rpool/logs
    zfs create rpool/logs/zones
    zfs set quota=120g rpool/logs/zones


    Should do the trick. You might also want to set up a reservation as well.

    (EDIT)

    The above won't work as logs/zones is a sub dataset of logs. So you either increase the logs quota to 184g as edited above), or you create the zones dataset slightly differently:

    zfs create rpool/logs
    zfs set mountpoint=/logs rpool/logs
    zfs set quota=64g rpool/logs
    zfs create rpool/lzones
    zfs set mountpoint=/logs/zones rpool/lzones
    zfs set quota=120g pool/logs/zones

    Mike

    Edited by: mramcha on Jul 20, 2012 3:54 PM
  • 4. Re: Solaris 10 JET install and ZFS
    JulianG Newbie
    Currently Being Moderated
    Thanks Mike - been having a play around with this and a number of the JET settings don't appear to being picked up and can't see why. I edited my JET template as such:

    base_config_profile_zfs_disk="slot0.s0 slot1.s0"
    base_config_profile_zfs_pool="rpool"
    base_config_profile_zfs_be=""
    base_config_profile_zfs_size="104g"
    base_config_profile_zfs_swap="64g"
    base_config_profile_zfs_dump="auto"
    base_config_profile_zfs_compress=""
    base_config_profile_zfs_var="24576"

    This does create a 104G slice 0, 64G SWAP but shouldn't /var have a 24G quota set?

    bash-3.2# df -h |grep rpool
    rpool/ROOT/10 101G 1.9G 31G 6% /
    rpool/ROOT/10/var 101G 27M 31G 1% /var
    rpool/export 101G 32K 31G 1% /export
    rpool/export/home 101G 31K 31G 1% /export/home
    rpool 101G 106K 31G 1% /rpool

    bash-3.2# zfs list
    NAME USED AVAIL REFER MOUNTPOINT
    rpool 70.0G 31.4G 106K /rpool
    rpool/ROOT 1.96G 31.4G 31K legacy
    rpool/ROOT/10 1.96G 31.4G 1.94G /
    rpool/ROOT/10/var  27.0M  31.4G  27.0M  /var
    rpool/dump 2.00G 31.4G 2.00G -
    rpool/export 63K 31.4G 32K /export
    rpool/export/home 31K 31.4G 31K /export/home
    rpool/swap 66.0G 97.4G 16K -

    I also tried to create a new ZFS pool using the ZFS configuration at the bottom of the client template but this was also completely ignored:

    zfs_pools_zones="mirror slot0.s1 slot1.s1"
    zfs_zones_filesys_zones="mountpoint=/zones quota=auto"

    I basically want to keep my global and non-global zones separate from each other and then further create folders within /zones and apply quotas to these.

    I thought this config looked ok but running the make_client complains again that the ZFS module is not configured.

    ..

    Checking product zfs
    WARNING: ZFS: ZFS module selected, but not configured to to anything.
    --------------------------------------------------------------
    Check of client build1.zfs
    -> Passed....


    Thanks - Julian.
  • 5. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    Hi Julian,

    If you check the comments for the zfs_var variable, it simply says to set any value to create it. You can manually set the reservation/quota when done. (i.e. expected behaviour)

    For the additional pool called zones, there are a couple of things.
    Firstly, you need to populate zfs_pools to let JET know that you want to create an additional pool.

    zfs_pools="zones"
    zfs_pools_zones="mirror slot0.s1 slot1.s1"

    Then you need to do similarly for filesystems:
    zfs_zones_filesys="zones"
    zfs_zones_filesys_zones="mountpoint=/zones" (you shouldn't say quote=auto, give a real value if you want one)


    So, the above is the correct syntax, and your error message will go away, and it should just work.... except for the fact that I didn't implement the ability to specify slot numbers AND slice numbers, as it is not considered good practice to create multiple slices on a disk with different pools in each. Performance will suck. (If the transaction log tries to write to both slices at the same time, your poor disk head will be rattling all over the disk.)

    To accomplish what you want, I'd suggest simply adding a dataset to your rpool. Still keeps it as a separate dataset, and you can apply a quota and reservation for safety.

    Change/modify the following:

    base_config_profile_zfs_size="auto"

    zfs_rpool_filesys="zones"
    zfs_rpool_filesys_zones="mountpoint=/zones quota=50g reservation=50g"


    (alternatively, if you have more disks, you could specify a new pool as you have done above, but using say "mirror slot2 slot3".)

    Mike

    Edited by: mramcha on Jul 23, 2012 11:04 AM
  • 6. Re: Solaris 10 JET install and ZFS
    JulianG Newbie
    Currently Being Moderated
    OK thanks .... again. I seem to fix one problem and create two more! Is there a log file at all for when the ZFS steps are carried out? I now do get a /zones created but I've edited the client template as I'd like to make sure that /var + /export + /logs are created with some restrictions around them. I've tried a couple of variations on the config but still no joy. All I end up with is:


    zfs list
    NAME USED AVAIL REFER MOUNTPOINT
    rpool 270G 277G 106K /rpool
    rpool/ROOT 1.97G 277G 31K legacy
    rpool/ROOT/10 1.97G 277G 1.97G /
    rpool/dump 2.00G 277G 2.00G -
    rpool/export 63K 277G 32K /export
    rpool/export/home 31K 277G 31K /export/home
    rpool/swap 66.0G 343G 16K -
    rpool/zones 31K 200G 31K /zones

    So no /var or /logs and /export is not restricted to 32G.

    The make_client does not error at all either. The syntax seems to make sense....

    zfs_pools="rpool"
    zfs_rpool_filesys="var"
    zfs_rpool_filesys_var="quota=24g"

    zfs_rpool_filesys="logs"
    zfs_rpool_filesys_logs="mountpoint=/logs quota=32g"

    zfs_rpool_filesys="export"
    zfs_rpool_filesys_export="quota=24g"

    zfs_rpool_filesys="zones"
    zfs_rpool_filesys_zones="mountpoint=/zones quota=200g reservation=200g"


    I've tried adding a "mountpoint=/...." on /var + /logs + /export and still the same.

    The base ZFS config has remained the same:

    base_config_profile_zfs_disk="slot0.s0 slot1.s0"
    base_config_profile_zfs_pool="rpool"
    base_config_profile_zfs_be=""
    base_config_profile_zfs_size="auto"
    base_config_profile_zfs_swap="64g"
    base_config_profile_zfs_dump="auto"
    base_config_profile_zfs_compress=""

    Thanks - J.
  • 7. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    Hi Julian,

    You MUST create /var as part of the installation in base_config, as stuff gets put there really early during the install.

    The ZFS module allows you to create additional filesystems/volumes in the rpool, but does not let you modify the properties of existing datasets/volumes.

    So,

    you still need
    base_config_profile_zfs_var="yes" if you want a /var dataset.

    /export and /export/home are created by default as part of the installation. You can't modify that as part of the install.

    For your zones dataset, seems to be fine and as expected, however, the zfs_rpool_filesys needs to list ALL the filesystems you want to create. It should read zfs_rpool_filesys="logs zones". This makes JET look for variables of the form zfs_rpool_filesys_logs and zfs_rpool_filesys_zones. (The last variable is always picked up, in your case the zones entry. Remember, the template is a simple name=value set of variables. If you repeat the "name" part, it simply overwrites the value.)

    So you really want:
    zfs_rpool_filesys="logs zones"
    zfs_rpool_filesys_logs="mountpoint=/logs quota=32g"
    zfs_rpool_filesys_zones="mountpoint=/zones quota=200g reservation=200g"

    (incidentally, you don't need to put zfs_pools="rpool" as JET assumes this automatically.)

    So, if you want to alter the properties of /var and /export, the syntax you used would work, if the module was set up to allow you to do that. (It does not currently do it, but I may update it in the future to allow it).

    (Send me a direct e-mail and I can send you an updated script which should then work as expected, check my profile and you should be able to guess my e-mail address)

    Alternatively, I'd suggest writing a simple script and stick it into the /opt/SUNWjet/Clients/<clientname> directory with the following lines in them:

    varexportquotas:
    #!/bin/sh
    zfs set quota=24g rpool/export
    zfs set quota=24g rpool/ROOT/10/var

    and then running it in custom_scripts_1="varexportquotas"

    (Or you could simply type the above commands the first time you log in after the build. :-) )

    Mike

    Edited by: mramcha on Jul 23, 2012 1:39 PM

    Edited by: mramcha on Jul 23, 2012 1:45 PM
  • 8. Re: Solaris 10 JET install and ZFS
    JulianG Newbie
    Currently Being Moderated
    Thanks Mike - got there in the end! Have amended the base_config_profile_zfs* settings and done the rest by way of a custom script which was less painful than trying to get it in the client template. I've still got some full end to end builds to do as my JET setup also does O/S patching, installs some additional patches and creates zones but it's all looking good now.

    Thanks again for all your help.
  • 9. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    Great.

    Assume you are doing logs and zones in the template, and setting quotas for export and var in the script.

    Again, I can send you an updated zfs module script that will let you set quotas for existing volumes in the template if you want.

    Are you using the JetZONES module to build your zones as well?

    Mike
  • 10. Re: Solaris 10 JET install and ZFS
    JulianG Newbie
    Currently Being Moderated
    Hi Mike - sort of, my client template now does this:

    base_config_profile_zfs_disk="slot0.s0 slot1.s0"
    base_config_profile_zfs_pool="rpool"
    base_config_profile_zfs_be=""
    base_config_profile_zfs_size="auto"
    base_config_profile_zfs_swap="64g"
    base_config_profile_zfs_dump="auto"
    base_config_profile_zfs_compress=""
    base_config_profile_zfs_var="yes"

    My script then does this:
    zfs set quota=24g rpool/export/home
    zfs set quota=24g rpool/ROOT/10/var
    zfs create rpool/export/zones
    zfs create rpool/export/zones/build14
    zfs set quota=32g rpool/export/zones/build14
    zfs create rpool/logs
    zfs set quota=32g rpool/logs
    zfs set mountpoint=/logs rpool/logs

    And yes I use the JETzones module to create a single zone at present, plus I like the fact that this zone is restricted in how much disk space it can consume. Something we never had control of when using UFS.

    ZFS list output:

    bash-3.2# zfs list
    NAME USED AVAIL REFER MOUNTPOINT
    rpool 70.0G 477G 106K /rpool
    rpool/ROOT 1.97G 477G 31K legacy
    rpool/ROOT/10 1.97G 477G 1.94G /
    rpool/ROOT/10/var 27.2M 24.0G 27.2M /var
    rpool/dump 2.00G 477G 2.00G -
    rpool/export 23.5M 477G 33K /export
    rpool/export/home 31K 24.0G 31K /export/home
    rpool/export/zones 23.5M 477G 32K /export/zones
    rpool/export/zones/build14 23.4M 32.0G 23.4M /export/zones/build14
    rpool/logs 33K 32.0G 33K /logs
    rpool/swap 66.0G 543G 16K -


    Thanks - Julian.
  • 11. Re: Solaris 10 JET install and ZFS
    935139 Newbie
    Currently Being Moderated
    So, I've got an updated JetZFS module which will remove the need for the script altogether.

    You should be able to type:

    zfs_rpool_filesys="export/home ROOT/10/var export/zones export/zones/build14 logs"
    zfs_rpool_filesys_export_home="quota=24g"
    zfs_rpool_filesys_ROOT_10_var="quota=24g"
    zfs_rpool_filesys_export_zones_build14="quota=32g"
    zfs_rpool_filesys_logs="mountpoint=/logs quota=32g"

    Let me know if you want the updated file that lets you do this. (It'll be in the next release of JetZFS anyway)

    (but your script works for you anyway, so no real need... but for the next time.)

    Mike

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points