ZFS Feature Flags

When ZFS was open-sourced by Sun Microsystems (now Oracle) in 2005, it contained a version number, allowing them to add underlying changes to the on-disk structure offering new features, while preventing corruption by old implementations that didn’t know how to handle the changes. This simple versioning scheme was strictly copied by the other implementations at the time (mostly FreeBSD). So when Sun added a feature, the version number got incremented, and other implementations copied the code as-is from OpenSolaris.

However after Oracle’s acquisition of Sun, in 2010, they closed the source of ZFS, not allowing others to copy new features. Thus, the ZFS developers from Illumos and FreeBSD formed a joint development effort to continue from the last open code, calling it OpenZFS.

In 2012 they decided that, as Oracle would continue developing new features, incrementing the version number, and probably in ways too difficult to implement them compatibly without access to the source code, a solution was needed.

They decided to increment the version number to 5000, in hopes Oracle will never get there, and create a new system for new feature usage and compatibility detection: feature flags.

About the author

Natalia Portillo was born in Canary Islands, a transgender girl who is an Independent and an Open-source Developer, a Computer Historian, an Emulator lover, a file system guru, an Apple Certified Macintosh Technician, and a .NET fan.

What are feature flags?

Feature flags are a list of features that can be created by any implementation of ZFS, and can be in three states: disabled, enabled or active. They also have a somewhat undescriptive but schematic

name: feature@:.

is the organization that created the feature, in inverse DNS format, for example, org.illumos. is the feature name.

When a feature is disabled, it means that the implementation supports it, but will not use it on the pool (for those that don’t know a ZFS pool, it is a collection of disks or other storage media that form a single storage entity).

When it is enabled, it means that the implementation will use it any time, either automatically or by manual user action. But it has not yet done so.

When it is active, it means that the implementation is using it, and changes are made to the underlying on-disk structure.

This way when Oracle’s implementation founds ZFS with version number 5000, it will not mount it. Moreover, when feature flags enabled ones (like FreeBSD since 9.2) found it will act accordingly to the feature and its state following these rules:

• If the feature is disabled, it will not be used at all, whether the implementation understands it or not.
• If the feature is enabled and the implementation does not understand it, it will not be used, but ZFS will still work.
• If the feature is enabled and the implementation understands it, it will be used when appropriate, moving its state to active.
• If the feature is active and the implementation does not understand it, it will indicate to the user accordingly and not mount it.
• If the feature is active and the implementation understands it, it will be used as appropriate.

One of the advantages of this system is that any implementation can create a new feature without breaking compatibility or interchangeability of the pool with other implementations, as long as it’s not being actively used.

Also, a feature flag can depend on another feature flag. Therefore, when you use one, its dependencies must also be used.

Feature flags explained:

com.delphix:extensible_dataset

This feature expands the underlying on-disk format to allow a more flexible use of its structures by other features. It does nothing by itself except allowing those other features. It becomes active as soon as any feature depending on it becomes active, and returns to enabled when all users of the depending features have been removed from the pool.

com.delphix:enabled_txg

This feature enables to record the exact moment on which a new feature is enabled. It is only used as a dependency for other features and once it becomes active, it never returns to enable.

com.delphix:async_destroy

When the user destroys a filesystem from the ZFS pool, all of the pool must be traversed to free the blocks it used. Sum a big filesystem and slow storage, and it will take hours. Add a reboot or power outage and the pool won’t be mounted until the operation is finished.

This feature adds on-disk data to do that operation asynchronously, in the background, so that the pool can be used while the blocks are being freed. Also if a reboot happens, the mount is not delayed, and the background operation is restarted where it left.

This feature is only active while a destroy operation is pending, and deactivates itself when it finishes.

com.delphix:empty_bpobj

This feature changes behavior with the references used by snapshots giving a performance increase, and a disk usage reduction in scenarios with a large number of snapshots. Once active, it only returns to enabled when everything created in the pool after activating it is destroyed.

com.joyent:filesystem_limits

This feature allows administrators to set how many filesystems or snapshots can be created under any point in the zpool. Once active, it cannot be deactivated.

org.illumos:lz4_compress

This feature allows setting the lz4 algorithms for compression. It’s about 50% faster on writing and 80% faster on reading while giving compressing files about 10% more. The boot pool can use this algorithm, and once this feature is enabled, it becomes active and will never return to enabled.

com.joyent:multi_vdev_crash_dump

This feature allows a dump device to be configured with a pool comprised of multiple vdevs, in any configuration.

com.delphix:spacemap_histogram

This feature enables storing more information about how free space is organized. Once enabled, it will become active when a new free space map is created. Thereafter, it will never return to the enabled state.

com.delphix:bookmarks

This feature allows creation of bookmarks that are like snapshots without holding the data they reference on the disk. They’re useful as ZFS send sources. It is active while there are bookmarks in the pool.

com.delphix:hole_birth

This feature improves performance for incremental sends, and receives objects with holes (sections of data that won’t get written onto the disk, e.g. blocks filled with zeros), storing information about when the hole was created. It becomes active once enabled.

com.delphix:embedded_data

This feature improves performance and space usage by storing any file that fits in 112 bytes (before or after compression) in the metadata instead of creating a block for it. It becomes active once enabled.

com.delphix:large_blocks

This feature allows using blocks bigger than 128KiB using the recordsize property. It becomes active once a record size is set to anything bigger than that. Moreover, it will only return to enabled once all filesystems that had the recordsize ever increased beyond that are destroyed.

org.illumos:sha512

This feature allows using a truncated version of the SHA-512 hash algorithm as the pool integrity checksum. This hash is about 50% faster on 64-bit hardware than SHA-256, and can be used for deduplication but cannot be used for boot pools. It becomes active once a new entry has been created while it was enabled, and will only return to enabled once all filesystems set to use it are destroyed.

org.illumos:skein

This feature enables using the Skein hash algorithm as the pool integrity checksum. This hash is about 80% faster on 64-bit hardware than SHA-256. It can be used for deduplication but cannot be used for boot pools. It is also salted to prevent hash collision attacks on systems with deduplication. It becomes active once a new entry has been created while it was enabled, and will only return to enabled once all filesystems set to use it are destroyed.

org.illumos:edonr

This feature enables using the Edon-R hash algorithm as the pool integrity checksum. This hash is about 350% faster on 64-bit hardware than SHA-256. It cannot be used for boot pools, and will not be used for deduplication unless verification is enabled. Furthermore, it is salted to prevent hash collision attacks on systems with deduplication. It becomes active once a new entry has been created while it was enabled, and will only return to enabled once all filesystems set to use it are destroyed.

Recommended features

What features you should enable is a very difficult thing to answer. However, there are some features whose usefulness is beyond doubt.

While you may think you will never destroy filesystems, there comes a day you have to do so. That is when you sit there thinking why you didn’t enable async_destroy before. Thus, just go and enable it, it will not harm you.

If you use ZFS compression, unless you are using gzip, enable lz4_compress and use it. It’s faster and better than ZFS’s default compression algorithm. Therefore, there is absolutely no reason not to use it.

embedded_data, hole_birth and empty_bpobj do not harm for being enabled. When needed, just enable them as they give several benefits.

If you ever ZFS send/receive, you should enable bookmarks also.

filesystem_limits, large_blocks and multi_vdev_crash_dump are very specific. Hence if you don’t know whether you require them, then be rest assured you don’t.

If you will physically connect the pool to a Linux system, you cannot use sha512 or skein, until ZFS on Linux 0.7.0 gets stable (at the time of this writing, it is not). If not, skein gives you enough performance benefits to be a must-enable, unless you’re not using SHA-256 (you are using it only if you have enabled it manually or you are using deduplication).

edonr is not yet supported on FreeBSD 11-CURRENT . So you can forget about it for now on anything but illumos.

If you want to know which feature flags are supported on your system, do man zpool-features in a terminal.

Examples

Before using feature flags, you should check whether your ZFS implementation supports them, and which ones are supported. The easiest way to do this is by typing the following command:

root # zpool upgrade -v

This command lists all the supported feature flags as well as the legacy ZFS versions. Then, you can check all the implementations where you use that pool (if it is in a removable media) and choose which ones to enable. It will also indicate which features are implemented in a read-only compatible way, meaning that when active, the pool can still be accessed to implementations that don’t support them, but read-only. If no feature flags are listed, it means that implementation supports none of them.

To enable a feature, you should use:

zpool set feature@=enabled

You can omit the organization name. Consequently, enabling the LZ4 compression in a pool called MyPool would be:

root # zpool set feature@lz4_compress=enabled MyPool

To see which feature flags are enabled in the pool, type:

root # zpool get all

Supported feature flags will be prepended with a feature@ and the organization name will be omitted. Unsupported feature flags will be prepended with a unsupported@, and depending on where they are marked as active or inactive , the implementation will not be able to use the pool (or use it in a read-only fashion), or just ignore the unsupported feature, respectively.