I have a set of five hard drives, 2 TB each. I would like to create two (or even more) different zpools over these hard drives, each with different settings (e.g. redundancy policy, encryption, etc.). How can I do that? I have found out on the Internet that you should use disk slices to achieve that goal, but I would need more specific instructions.
If you can accept some performance degradation creating zpool on slices rather than on entire disks,
Use "fdisk" to slice the disks the way you like, and specify slices (like c5t0d3p3 or c5d3p3) rather than entire disks (like c5t0d3 or c5d3) in "zpool create" command.
The only reason for different pools is for a different raid layout: mirror, raidz, raidz2, raidz3.
For encryption, compression, dedup those are all per dataset properties so there is no need to create a different pool to have different settings for those per dataset.
You could even consider that using the copies property gives you different redundancy requirements. Ideally you should pick your raid layout based on what you can afford to spend and run in terms of disks and what performance you are willing to accept to get what you require in space.
I would recommend that you never run multiple ZFS pools off the same local drive, unless it is for playing around where performance doesn't mater.
You do need separate pools if you want different redundancy policies but trying to configure different pools with different redundancies across disk slices is difficult to manage. Recovery is also difficult. ZFS storage pools were never intended to be created on p* devices and is not recommended.
The easiest configuration to set up, manage, and recover with 5 disks if you also need to allocate space for the root pool is like this:
rpool (2 mirrored disks)
data pool (2 mirrored disks)
We had a configuration recently where a rpool and our tools pool were built across disk slices of the same disks. The rpool disk failed, which also took out our tools pool, which meant our tools were unavailable until the root pool disk was replaced. If we had used the setup above, we could have replaced the root pool disk and taken no outage at all. After we replaced the root pool disk, I
reconfigured this system to a configuration like the one above, which is so much easier to manage and recover from. I also took
the opportunity to install Solaris 11.1, which allows me to configure smtp-notify to notify me when a disk fails. We had assumed
that some other reconfiguration made the tools unavailable, but it was a failed root pool disk all along. Live and learn.