qemu 2.12 does not support if=scsi

kdave — Fri, 08 Jun 2018 00:00:00 +0200

After update of qemu to version 2.12, my testing vms stopped to just warn about the if=scsi (with a bit more cryptic message), and did not want to start.

qemu-system-x86_64: -drive file=root,if=scsi,media=disk,cache=none,index=0,format=raw: machine type does not support if=scsi,bus=0,unit=0

The if=scsi shortcut was handy, the maze of qemu command line options may take some time to comprehend but then you can do wonders.

The release notes of version 2.12 do mention that the support is gone and that other options should be used instead, also mentions which ones. But it does not tell how exactly. As this should be a simple syntax transformation from one option to another, an example would save a lot of time.

I was not able to find any ready-to-use example in a few minutes so had to experiment a bit (this saved me documentation reading time).

The setup I use is a file-backed root image for a virtual machine, nothing really fancy. The file name is root, sparse file, raw ie. no qcow2, no caching.

Here you are:

-device virtio-scsi-pci,id=scsi
-drive file=root,id=root-img,if=none,format=raw,cache=none
-device scsi-hd,drive=root-img

First you need to define the bus, otherwise the 3rd line will complain that there’s no SCSI. Second line is to define the file backed drive and the third one puts that together.

Using SCSI might not be the best idea for a qemu VM as the emulated driver is buggy and crashes, so I’d recommend to use virtio, but for a almost read-only root image it’s fine. Also the device names are predictable.

Linux crypto: testing blake2s

kdave — Sat, 27 Apr 2019 00:00:00 +0200

The BLAKE2 algorithm is out for some time, yet there’s no port to linux crypto API. The recent push of WireGuard would add it to linux kernel, but using a different API (zinc) than the existing one. I haven’t explored zinc, I assume there will be a way to use it from inside kernel, but this would need another layer to switch the APIs according to the algorithm.

As btrfs is going to use more hashing algos, we are in the process of selection. One of the contenders is BLAKE2, though everybody would have a different suggestion. In order to test it, a port is needed. Which basically is a glue code between the linux crypto API and the BLAKE2 implementation.

I’m not going to reimplement crypto or anything, so the the default implementation is going to be the reference one found in blake2 git.

Note: due to the maximum space of 32 bytes available in the btrfs metadata blocks, the version is BLAKE2s.

The BLAKE2s sources

From the repository https://github.com/BLAKE2/BLAKE2:

ref/blake2s-ref.c
ref/blake2.h
ref/blake2-impl.h

Briefly skimming over the sources, there’s nothing that’ll cause trouble. Standard C code, some definitions. Adapting for linux kernel would need to replace the stdint.h types (uint8_t etc) with the uXX (u8 etc) and change path to includes. For simplicity, let’s remove the ifdefs for C++ and MSVC too.

Add the new algorithm definition (CRA)

Though it’s possible to prepare a module for an out-of-tree build (see below), let’s do it inside the linux.git/crypto/ path for now. There’s also plenty of sources to copy & paste. I’ve used crc32c_generic.c, and it turned out to be useful.

The crypto hash description is defined in struct shash_alg, contains the technical description like type of the algorithm, length of the context and callbacks for various phases of the hash calculation (init, update, final), and identification of the algorithm and the implementation. The default implementations are C and use the string “-generic” in the name.

The crc32c module came with a few stub callbacks (checksum_init etc), that will only call into the blake2 functions and the definition can stay. Simple search and replace from crc32c to blake2s will do most of the conversion.

Add the glue code for crypto API

Now we have the blake2s.c with the reference implementation, crypto algorithm definition. The glue code connects the API with the implementation. We need 2 helper structures that hold the context once the user starts digest calculation. The private blake2s context is embedded into one of them. The intermediate results will be stored there.

And the rest is quite easy, each callback will call into the respective blake2s function, passing the context, data and lengths. One thing that I copied from the examples is the key initialization that’s in blake2s_cra_init, that appears to be global and copied to the context each time a new one is initialized.

Here the choice of using crc32c.c helped as there were the stub callback with the right parameters, calling the blake2s functions that can retain their original signature. This makes later updates easier. All the functions are static so the compiler will most probably optimize the code that there will be no unnecessary overhead.

Well and that’s it. Let’s try to compile it and insert the module:

linux.git$ make crypto/blake2s.o
...
  CC [M]  crypto/blake2s.o
linux.git$ make crypto/blake2s.ko
...
  CC      crypto/blake2s.mod.o
  LD [M]  crypto/blake2s.ko
...

Check that it’s been properly loaded

linux.git/crypto$ sudo insmod ./blake2s.ko

and registered

linux.git$ cat /proc/crypto
name         : blake2s
driver       : blake2s-generic
module       : blake2s
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 32
...

The selftest says it passed, but there no such thing so far. There are test values provided in blake2 git so it would be nice to have too (tm). But otherwise it looks good.

To do actual test, we’d need something inside kernel to utilize the new hash. One option is to implement a module that will do that or use the userspace library libkcapi that can forward the requests from userspace to the available kernel implementations.

Test it with libkcapi

The libkcapi project at http://www.chronox.de/libkcapi.html provides an API that uses the AF_ALG socket type to exchange data with kernel. The library provides a command line tool that we can use right away and don’t need to code anything.

$ kcapi-dgst -c blake2s --hex < /dev/null
48a8997da407876b3d79c0d92325ad3b89cbb754d86ab71aee047ad345fd2c49

The test vectors provided by blake2 confirm that this is hash of empty string with the default key (0x000102..1f).

Out-of-tree build

Sources in the linux.git require one additional line to Makefile, build it unconditionally as a module. Proper submission to linux kernel would need the Kconfig option.

obj-m += blake2s.o

The standalone build needs a Makefile with a few targets that use the existing build of kernel. Note that you’d need a running kernel with the same built sources. This is usually provided by the kernel-*-devel packages. Otherwise, if you build kernels from git, you know what to do, right?

KDIR ?= /lib/modules/`uname -r`/build
obj-m := blake2s.o

default:
        $(MAKE) -C $(KDIR) M=$$PWD

clean:
        $(MAKE) -C $(KDIR) M=$$PWD clean

modules_install:
        $(MAKE) -C $(KDIR) M=$$PWD modules_install

After running make, the kernel module is ready for use.

What next?

Send it upstream. Well, after some work of course.

update the coding style of the blake2 sources
add Kconfig
write self-tests
optionally add the optimized implementations

All the files can be found here:

Makefile: out-of-tree build
blake2.h: copied and updated for linux
blake2-impl.h: copied and updated for linux
blake2s.c: copied and updated for linux

References

Btrfs hilights in 5.2

kdave — Thu, 25 Jul 2019 00:00:00 +0200

A bit more detailed overview of a btrfs update that I find interesting, see the pull request for the rest.

Read-time, write-time corruption detection

The tree-checker is a recent addition to the sanity checks, a functionality that verifies a subset of metadata structures. The capabilities and strength is limited by the context and bits of information available at some point, but still there’s enough to get an additional value.

The context here means a b-tree leaf that packs several items into a block (ie. the nodesize, 16KiB by default). The individual items’ members’ values can be verified against allowed values, the order of the keys of all items listed in the leaf header can be checked etc. This is just for a rough idea what happens.

With that in the works, there are two occasions that can utilize the extended checking:

read-time – the first time a block is read fresh from disk
write-time – when the final contents of a block is in memory and going to be written to disk

It’s clear that the read-time is merely a detector of problems that already happened, so there’s not much to do besides warning and returning an error (EUCLEAN). Turning the filesystem to read-only to prevent further problems is another option and inevitable on some occasions.

The write side check aims to catch silent errors that could make it to the permanent storage. The reasons why this happens are two fold, but the main idea is to catch memory bit flips. You’d be surprised how often this happens, that would be for a separate article entirely.

The new checks in 5.2 improve:

device item (item that describes the physical device of the filesystem) – check that items have the right key type, device id is within allowed range and that usage and total sizes are within limits of the physical device
inode item (item for inodes, ie. files, directories or special files)– the objectid (that would be the inode number) is in the range, generation is consistent with the global one and the basic sanity of mode, flags/attributes and link count
block group profiles – check that only a single type is set in the bit mask that represents block group profile type of a chunk (ie. single/dup/raid1/…)

As mentioned before, the bits to check are inside a single buffer that represents the tree block and the check is really local to that. As an inode item can represent more than just files and directories, doing structural checks like where and how the inode item is linked to is not easy in this context. This is basically what would btrfs check do.

The checks need to be fast because they happen on each metadata block, so no additional IO is allowed. This still brings some overhead but is (considered) negligible compared to all other updates to the block. A measurement can be done by adding tracepoint, but that’s left as an exercise.

Selecting the next checksum for btrfs

kdave — Tue, 08 Oct 2019 00:00:00 +0200

The currently used and the only one checksum algorithm that’s implemented in btrfs is crc32c, with 32bit digest. It has served well for the years but has some weaknesses so we’re looking for a replacements and also for enhancing the use cases based on possibly better hash algorithms.

The advantage of crc32c is it’s simplicity of implementation, various optimized versions exist and hardware CPU instruction-level support. The error detection strength is not great though, the collisions are easy to generate. Note that crc32c has not been used in cryptographically sensitive context (eg. deduplication).

Side note: the collision generation weakness is used in the filesystem image dump tool to preserve hashes of directory entries while obscuring the real names.

Future use cases

The natural use case is still to provide checksumming for data and metadata blocks. With strong hashes, the same checksums can be used to aid deduplication or verification (HMAC instead of plain hash). Due to different requirements, one hash algorithm cannot possibly satisfy all requirements, namely speed vs. strength. Or can it?

The criteria

The frequency of checksumming is high, every data block needs that, every metadata block needs that.

During the discussions in the community, there were several candidate hash algorithms proposed and as it goes users want different things but we developers want to keep the number of features sane or at least maintainable. I think and hope the solution will address that.

The first hash type is to replace crc32c with focus on speed and not necessarily strength (ie. collision resistance).

The second type focus is strength, in the cryptographic sense.

In addition to the technical aspects of the hashes, there are some requirements that would allow free distribution and use of the implementations:

implementation available under a GPL2 compatible license
available in the linux kernel, either as a library function or as a module
license that allows use in external tools like bootloaders (namely grub2)

Constraints posed by btrfs implementation:

maximum digest width is 32 bytes
blocks of size from 4KiB up to 64KiB

For the enhanced use case of data verification (using HMAC), there’s a requirement that might not interest everybody but still covers a lot of deployments. And this is standard compliance and certification:

standardized by FIPS

And maybe last but not least, use something that is in wide use already, proven by time.

Speed

Implementation of all algorithms should be performant on common hardware, ie. 64bit architectures and hopefully not terrible on 32bit architectures or older and weaker hardware. By hardware I mean the CPU, not specialized hardware cards.

The crypto API provided by linux kernel can automatically select the best implementation of a given algorithm, eg. optimized implementation in assembly and on multiple architectures.

Strength

For the fast hash the collisions could be generated but hopefully not that easily as for crc32c. For strong hash it’s obvious that finding a collision would be jackpot.

In case of the fast hash the quality can be evaluated using the SMHasher suite.

The contenders

The following list of hashes has been mentioned and considered or evaluated:

xxhash
XXH3
SipHash
CRC64
SHA256
SHA3
BLAKE2

xxhash

criteria: license OK, implementation OK, digest size OK, not standardized but in wide use

The hash is quite fast as it tries to exploit the CPU features that allow instruction parallelism. The SMHasher score is 10, that’s great. The linux kernel implementation landed in 5.3.

Candidate for fast hash.

XXH3

Unfortunately the hash is not yet finalized and cannot be in the final round, but for evaluation of speed it was considered. The hash comes from the same author as xxhash.

SipHash

This hash is made for hash tables and hashing short strings but we want 4KiB or larger blocks. Not a candidate.

CRC64

Similar to crc32 and among the contenders only because it was easy to evaluate but otherwise is not in the final round. It has shown to be slow in the microbenchmark.

SHA256

criteria: license OK, implementation OK, digest size OK, standardized in FIPS

The SHA family of hashes is well known, has decent support in CPU and is standardized. Specifically, SHA256 is the strongest that still fits into the available 32 bytes.

Candidate for strong hash.

SHA3

criteria: license OK, implementation OK, digest size OK, standardized in FIPS

Winner of the 2012 hash contest, we can’t leave it out. From the practical perspective of checksum, the speed is bad even for the strong hash type. One of the criteria stated above is performance without special hardware, unlike what was preferred during the SHA3 contest.

Candidate for strong hash.

BLAKE2

criteria: license OK, implementation OK, digest size OK, not standardized

From the family of BLAKE that participated in the 2012 SHA contest, the ‘2’ provides a trade-off speed vs. strength. More and more projects adopt it.

Candidate for strong hash.

Final round

I don’t blame you if you skipped all the previous paragraphs. The (re)search for the next hash was quite informative and fun so it would be shame not to share it, also to document the selection process for some transparency. This is a committee driven process though.

fast hash: xxhash
strong hash: BLAKE2 and SHA256

Two hashes selected for the strong type is a compromise to get a fast-but-strong hash yet also something that’s standardized.

The specific version of BLAKE2 is going to be the ‘BLAKE2b-256’ variant, ie. optimized for 64bit (2b) but with 256bit digest.

Microbenchmark

A microbenchmark gives more details about performance of the hashes:

Block: 4KiB (4096 bytes), Iterations: 100000

Hash	Total cycles	Cycles/iteration
NULL-NOP	56888626	568
NULL-MEMCPY	60644484	606
CRC64	3240483902	32404
CRC32C	174338871	1743
CRC32C-SW	174388920	1743
XXHASH	251802871	2518
XXH3	193287384	1932
BLAKE2b-256-arch	1798517566	17985
BLAKE2b-256	2358400785	23584
BLAKE2s-arch	2593112451	25931
BLAKE2s	3451879891	34518
SHA256	10674261873	106742
SHA3-256	29152193318	291521

Machine: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (AVX2)

Hash implementations are the reference ones in C:

NULL-NOP – stub to measure overhead of the framework
NULL-MEMCPY – simple memcpy of the input buffer
CRC64 – linux kernel lib/crc64.c
CRC32C – hw assisted crc32c, (linux)
CRC32C-SW – software implementation, table-based (linux)
XXHASH – reference implementation
XXH3 – reference implementation
BLAKE2b-256-arch – 64bit optimized reference version (for SSE2/SSSE3/SSE41/AVX/AVX2)
BLAKE2b-256 – 64bit reference implementation
BLAKE2s-arch – 32bit optimized reference version
BLAKE2s – 32bit reference implementation
SHA256 – RFC 6234 reference implementation
SHA3-256 – C, based on canonical implementation

There aren’t optimized versions for all hashes so for fair comparison the unoptimized reference implementation should be used. As BLAKE2 is my personal favourite I added the other variants and optimized versions to observe the relative improvements.

Evaluation

CRC64 was added by mere curiosity how does it compare to the rest. Well, curiosity satisfied.

SHA3 is indeed slow on a CPU.

What isn’t here

There are non-cryptographic hashes like CityHash, FarmHash, Murmur3 and more, that were found unsuitable or not meeting some of the basic criteria. Others like FNV or Fletcher used in ZFS are of comparable strength of crc32c, so that won’t be a progress.

Merging

All the preparatory work in btrfs landed in version 5.3. Hardcoded assumptions of crc32c were abstracted, linux crypto API wired in, with additional cleanups or refactoring. With that in place, adding a new has is a matter of a few lines of code adding the specifier string for crypto API.

The work on btrfs-progs is following the same path.

Right now, the version 5.4 is in development but new features can’t be added, so the target for the new hashes is 5.5. The BLAKE2 algorithm family still needs to be submitted and merged, hopefully they’ll make it to 5.5 as well.

One of my merge process requirements was to do a call for public testing, so we’re going to do that once all the kernel and progs code is ready for testing. Stay tuned.

Btrfs hilights in 5.3

kdave — Wed, 11 Dec 2019 00:00:00 +0100

A bit more detailed overview of a btrfs update that I find interesting, see the pull request for the rest.

CRC32C uses accelerated versions on other architectures

… than just Intel-based ones. There was a hard coded check for the intel SSE feature providing the accelerated instruction, but this totally skipped other architectures. A brief check in linux.git/arch/*/crypto for files implementing the accelerated versions revealed that there’s a bunch of them: ARM, ARM64, MIPS, PowerPC, s390 and SPARC. I don’t have enough hardware to show the improvements, though.

Automatically remove incompat bit for RAID5/6

While this is not the fix everybody is waiting on, it’s demonstrating user-developer-user cycle how things can be improved. A filesystem created with RAID5 or -6 profiles sets an incompatibility bit. That’s supposed to be there as long as there’s any chunk using the profiles. User expectation is that the bit should be removed once the chunks are eg. balanced away. This is what got implemented and serves as a prior example for any future feature that might get removed on a given filesystem. (Note from the future: I did that for the RAID1C34 as well).

Side note: for the chunk profile it’s easy because the runtime check of presence is quick and requires only going through a list of chunk profiles types. Example of the worst case would be dropping the incompat bit for LZO after there are no compressed files using that algorithm. You can easily see that this would require either enumerating all extent metadata (one time check) or keeping track of count since mkfs time. This is IMHO not a typical request and can be eventually done using the btrfstune tool on an unmounted filesystem.

BLAKE3 vs BLAKE2 for BTRFS

kdave — Tue, 21 Jan 2020 00:00:00 +0100

Irony isn’t it. The paint of BLAKE2 as BTRFS checksum algorithm hasn’t dried yet, 1-2 weeks to go but there’s a successor to it. Faster, yet still supposed to be strong. For a second or two I considered ripping out all the work and … no not really but I do admit the excitement.

Speed and strength are competing goals for a hash algorithm. The speed can be evaluated by anyone, not so much for the strength. I am no cryptographer and for that area rely on expertise and opinion of others. That BLAKE was a SHA3 finalist is a good indication, where BLAKE2 is it’s successor, weakened but not weak. BLAKE3 is yet another step trading off strength and speed.

Regarding BTRFS, BLAKE2 is going to be the faster of strong hashes for now (the other one is SHA256). The argument I have for it now is proof of time. It’s been deployed in many projects (even crypto currencies!), there are optimized implementations, various language ports.

The look ahead regarding more checksums is to revisit them in about 5 years. Hopefully by that time there will be deployments, real workload performance evaluations and overall user experience that will back future decisions.

Maybe there are going to be new strong yet fast hashes developed. During my research I learned about Kangaroo 12 that’s a reduced version of SHA3 (Keccak). The hash is constructed in a different way, perhaps there might be a Kangaroo 2π one day on par with BLAKE3. Or something else. Why not EDON-R, it’s #1 in many of the cr.yp.to/hash benchmarks? Another thing I learned during the research is that hash algorithms are twelve in a dozen, IOW too many to choose from. That Kangaroo 12 is internally of a different construction might be a point for selecting it to have wider range of “building block types”.

Quick evaluation

For BTRFS I have a micro benchmark, repeatedly hashing a 4 KiB block and using cycles per block as a metric.

Block size: 4KiB
Iterations: 10000000
Digest size: 256 bits (32 bytes)

Hash	Total cycles	Cycles/iteration	Perf vs BLAKE3	Perf vs BLAKE2b
BLAKE3 (AVX2)	111260245256	11126	1.0	0.876 (-13%)
BLAKE2b (AVX2)	127009487092	12700	1.141 (+14%)	1.0
BLAKE2b (AVX)	166426785907	16642	1.496 (+50%)	1.310 (+31%)
BLAKE2b (ref)	225053579540	22505	2.022 (+102%)	1.772 (+77%)

Right now there’s only the reference Rust implementation and a derived C implementation of BLAKE3, claimed not to be optimized but from my other experience the compiler can do a good job optimizing programmers ideas away. There’s only one BLAKE3 entry with the AVX2 implementation, the best hardware support my testing box provides. As I had the other results of BLAKE2 at hand, they’re in the table for comparison, but the most interesting pair are the AVX2 versions anyway.

The improvement is 13-14%. Not much ain’t it, way less that the announced 4+x faster than BLAKE2b. Well, it’s always important to interpret results of a benchmark with respect to the environment of measurement and the tested parameters.

For BTRFS filesystem the block size is always going to be in kilobytes. I can’t find what was the size of the official benchmark results, the bench.rs script iterates over various sizes, so I assume it’s an average. Short input buffers can skew the results as the setup/output overhead can be significant, while for long buffers the compression phase is significant. I don’t have explanation for the difference and won’t draw conclusions about BLAKE3 in general.

One thing that I dare to claim is that I can sleep well because upon the above evaluation, BLAKE3 won’t bring a notable improvement if used as a checksum hash.

References

new hash for BTRFS selection, same testing box for the measurements
https://github.com/BLAKE3-team/BLAKE3 – top commit 02250a7b7c80ded, 2020-01-13, upstream version 0.1.1
DJB’s hash menu and per-machine results with numbers

Personal addendum

During the evaluations now and in the past, I’ve found it convenient if there’s an offer of implementations in various languages. That eg. Keccak project pages does not point me directly to a C implementation slightly annoyed me, but the reference implementation in C++ was worse than BLAKE2 I did not take the next step to compare the C version, wherever I would find it.

BLAKE3 is fresh and Rust seems to be the only thing that has been improved since the initial release. A plain C implementation without any warning-not-optimized labels would be good. I think that C versions will appear eventually, besides that Rust is now the new language hotness, there are projects not yet “let’s rewrite it in Rust”. Please Bear with us.

Btrfs hilights in 5.5: 3-copy and 4-copy block groups

kdave — Sun, 02 Feb 2020 00:00:00 +0100

A bit more detailed overview of a btrfs update that I find interesting, see the pull request for the rest.

New block group profiles RAID1C3 and RAID1C4

There are two new block group profiles enhancing capabilities of the RAID1 types with more copies than 2. Brief overview of the profiles is in the table below, for table with all profiles see manual page of mkfs.brtfs, also available on wiki.

Profile	Copies	Utilization	Min devices
RAID1	2	50%	2
RAID1C3	3	33%	3
RAID1C4	4	25%	4

The way all the RAID1 types work is that there are 2 / 3 / 4 exact copies over all available devices. The terminology is different from linux MD RAID, that can do any number of copies. We decided not to do that in btrfs to keep the implementation simple. Another point for simplicity is from the users’ perspective. That RAID1C3 provides 3 copies is clear from the type. Even after adding a new device and not doing balance, the guarantees about redundancy still hold. Newly written data will use the new device together with 2 devices from the original set.

Compare that with a hypothetical RAID1CN, on a filesystem with M devices (N <= M). When the filesystem starts with 2 devices, equivalent to RAID1, adding a new one will have mixed redundancy guarantees after writing more data. Old data with RAID1, new with RAID1C3 – but all accounted under RAID1CN profile. A full re-balance would be required to make it a reliable 3-copy RAID1. Add another device, going to RAID1C4, same problem with more data to shuffle around.

The allocation policy would depend on number of devices, making it hard for the user to know the redundancy level. This is already the case for RAID0/RAID5/RAID6. For the striped profile RAID0 it’s not much of a problem, there’s no redundancy. For the parity profiles it’s been a known problem and new balance filter stripe has been added to support fine grained selection of block groups.

Speaking about RAID6, there’s the elephant in the room, trying to cover write hole. Lack of a resiliency against 2 device damage has been bothering all of us because of the known write hole problem in the RAID6 implementation. How this is going to be addressed is for another post, but for now, the newly added RAID1C3 profile is a reasonable substitute for RAID6.

How to use it

On a freshly created filesystem it’s simple:

# mkfs.btrfs -d raid1c3 -m raid1c4 /dev/sd[abcd]

The command combines both new profiles for sake of demonstration, you should always consider the expected use and required guarantees and choose the appropriate profiles.

Changing the profile later on an existing filesystem works as usual, you can use:

# btrfs balance start -mconvert=raid1c3 /mnt/path

Provided there are enough devices and enough space to do the conversion, this will go through all metadadata block groups and after it finishes, all of them will be of the of the desired type.

Backward compatibility

The new block groups are not understood by old kernels and can’t be mounted, not even in the read-only mode. To prevent that a new incompatibility bit is introduced, called raid1c34. Its presence on a device can be checked by btrfs inspect-internal dump-super in the incompat_flags. On a running system the incompat features are exported in sysfs, /sys/fs/btrfs/UUID/features/raid1c34.

Outlook

There is no demand for RAID1C5 at the moment (I asked more than once). The space utilization is low already, the RAID1C4 survives 3 dead devices so IMHO this is enough for most users. Extending resilience to more devices should perhaps take a different route.

With more copies there’s potential for parallelization of reads from multiple devices. Up to now this is not optimal, there’s a decision logic that’s semi-random based on process ID of the btrfs worker threads or process submitting the IO. Better load balancing policy is a work in progress and could appear in 5.7 at the earliest (because 5.6 development is now in fixes-only mode).

Look back

The history of the patchset is a bit bumpy. There was enough motivation and requests for the functionality, so I started the analysis what needs to be done. Several cleanups were necessary to unify code and to make it easily extendable for more copies while using the same mirroring code. In the end change a few constants and be done.

Following with testing, I tried simple mkfs and conversions, that worked well. Then scrub, overwrite some blocks and let the auto-repair do the work. No hiccups. The remaining and important part was the device replace, as the expected use case was to substitute RAID6, replacing a missing or damaged disk. I wrote the test script, replace 1 missing, replace 2 missing. And it did not work. While the filesystem was mounted, everything seemed OK. Unmount, check again and the devices were still missing. Not cool, right.

Due to lack of time before the upcoming merge window (a code freeze before next development cycle), I had to declare it not ready and put it aside. This was in late 2018. For a highly requested feature this was not an easy decision. Should it be something less important, the development cycle between rc1 and final release provides enough time to fix things up. But due to the maintainer role with its demands I was not confident that I could find enough time to debug and fix the remaining problem. Also nobody offered help to continue the work, but that’s how it goes.

At the late 2019 I had some spare time and looked at the pending work again. Enhanced the test script with more debugging messages and more checks. The code worked well, the test script was subtly broken. Oh well, what a blunder. That cost a year, but on the other hand releasing a highly requested feature that lacks an important part was not an appealing option.

The patchset was added to 5.5 development queue at about the last time before freeze, final 5.5 release happened a week ago.

Btrfs hilights in 5.4: tree checker updates

kdave — Sat, 15 Feb 2020 00:00:00 +0100

A bit more detailed overview of a btrfs update that I find interesting, see the pull request for the rest.

There’s not much to show in this release. Some users find that good too, a boring release. But still there are some changes of interest. The 5.4 is a long-term support stable tree, stability and core improvements are perhaps more appropriate than features that need a release or two to stabilize.

? stable not known in advance so not pushing half-baked features to stable, possibly requiring more intrusive fixups

The development cycle happened over summer and this slowed down the pace of patch reviews and update turnarounds.

Tree-checker updates

The tree-checker is a sanity checker of metadata that are read from/written to devices. Over time it’s being enhanced by more checks, let’s have a look at two of them.

ROOT_ITEM checks

The item represents root of a b-tree, of the internal or the subvolume trees.

Let’s take an example from btrfs inspect dump-tree:

       item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
                generation 5 root_dirid 0 bytenr 30523392 level 0 refs 1
                lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
                uuid 00000000-0000-0000-0000-000000000000
                drop key (0 UNKNOWN.0 0) level 0

Some of the metadata inside the item allow only simple checks, following commit 259ee7754b6793:

key.objectid must match the tree that’s being read, though the code verifies only if the type is not 0
key.offset must be 0
block offset bytenr must be aligned to sector size (4KiB in this case)
itemsize depends on the item type, but for the root item it’s fixed value
level and drop_level is 0 to 7, but it’s not possible to cross check if the tree has really of that level
generation must be lower than the super block generation, same for lastsnap
flags can be simply compared to the bit mask of allowed flags, right now there are two, one represents a read-only subvolume and another a subvolume that has been marked as deleted but its blocks not yet cleaned

The refs is a reference counter and sanity check would require reading all the expected reference holders, bytes_used would need to look up the block that it accounts, etc. The subvolume trees have more data like ctime, otime and real uuids. If you wonder what’s byte_limit, this used to be a mechanism to emulate quotas by setting the limit value, but it has been deprecated and unused for a long time. One day we might to find another purpose for the bytes.

Many of the tree-checker enhancements are follow ups to fuzz testing and reports, as it was in this case. The bug report shows that some of the incorrect data were detected and even triggered auto-repair (as this was on filesystem with DUP metadata) but there was too much damage and it crashed at some point. The crash was not random but a BUG_ON of an unexpected condition, that’s sanity check of last resort. Catching inconsistent data early with a graceful error handling is of course desired and ongoing work.

Extent metadata item checks

There are two item types that represent extents and information about sharing. EXTENT_ITEM is older and bigger while METADATA_ITEM is the building block of skinny-metadata feature, smaller and more compact. Both items contain type of reference(s) and the owner (a tree id). Besides the generic checks that also the root item does (alignment, value ranges, generation), there’s a number of allowed combinations of the reference types and extent types. The commit f82d1c7ca8ae1bf implements that, however further explanation is out of scope of the overview as the sharing and references are the fundamental design of btrfs.

Example of METADATA_ITEM:

        item 170 key (88145920 METADATA_ITEM 0) itemoff 10640 itemsize 33
                refs 1 gen 27 flags TREE_BLOCK
                tree block skinny level 0
                tree block backref root FS_TREE

And EXTENT_ITEM:

        item 27 key (20967424 EXTENT_ITEM 4096) itemoff 14895 itemsize 53
                refs 1 gen 499706 flags DATA
                extent data backref root FS_TREE objectid 8626071 offset 0 count 1

This for a simple case with one reference, tree (for metadata) and ordinary data, so comparing the sizes shows 20 bytes saved. On my 20GiB root partition with about 70 snapshots there are XXX EXTENT and YYY METADATA items.

Otherwise there can be more references inside one item (eg. many snapshots of a file that is randomly updated over time) so the overhead of the item itself is smaller

Authenticated hashes for btrfs (part 1)

kdave — Mon, 24 May 2021 00:00:00 +0200

There was a request to provide authenticated hashes in btrfs, natively as one of the btrfs checksum algorithms. Sounds fun but there’s always more to it, even if this sounds easy to implement.

Johaness T. at that time in SUSE sent the patchset adding the support for SHA256 [1] with a Labs conference paper, summarizing existing solutions and giving details about the proposed implementation and use cases.

The first version of the patchset posted got some feedback, issues were found and some ideas suggested. Things have stalled a bit, but the feature is still very interesting and really not hard to implement. The support for additional checksums has provided enough support code to just plug in the new algorithm and enhance the existing interfaces to provide the key bytes. So until now I’ve assumed you know what an authenticated hash means, but for clarity and in simple terms: a checksum that depends on a key. The main point is that it’s impossible to generate the same checksum for given data without knowing the key, where impossible is used in the cryptographic-strength sense, there’s an almost zero probability doing that by chance and brute force attack is not practical.

Auth hash, fsverity

Notable existing solution for that is fsverity that works in read-only fashion, where the key is securely hidden and used only to verify that data that are read from media haven’t been tampered with. A typical use case is an OS image in your phone. But that’s not all. Images of OS appear in all sorts of boxed devices, IoT. Nowadays, with explosion of edge computing, assuring integrity of the end devices is a fundamental requirement.

Where btrfs can add some value is the read AND write support, with an authenticated hash. This brings questions around key handling, and not everybody is OK with a device that could potentially store malicious/invalid data with a proper authenticated checksum. So yeah, use something else, this is not your use case, or maybe there’s another way how to make sure the key won’t be compromised easily. This is beyond the scope of what filesystem can do, though.

As an example use case of writable filesystem with authenticated hash: detect outside tampering with on-disk data, eg. when the filesystem was unmounted. Filesystem metadata formats are public, interesting data can be located by patterns on the device, so changing a few bytes and updating the checksum(s) is not hard.

There’s one issue that was brought up and I think it’s not hard to observe anyway: there’s a total dependency on the key to verify a basic integrity of the data. Ie. without the key it’s not possible to say if the data are valid as if a basic checksum was used. This might be still useful for a read-only access to the filesystem, but absence of key makes this impossible.

Existing implementations

As was noted in the LWN discussion [2], what ZFS does, there are two checksums. One is the authenticated and one is not. I point you to the comment stating that, as I was not able to navigate far enough in the ZFS code to verify the claim, but the idea is clear. It’s said that the authenticated hash is eg. SHA512 and the plain hash is SHA256, split half/half in the bytes available for checksum. The way the hash is stored is a simple trim of the first 16 bytes of each checksum and store them consecutively. As both hashes are cryptographically strong, the first 16 bytes should provide enough strength despite the truncation. Where 16 bytes is 128 bits.

When I was thinking about that, I had a different idea how to do that. Not that copying the scheme would not work for btrfs, anything that the linux kernel crypto API provides is usable, the same is achievable. I’m not judging the decisions what hashes to use or how to do the split, it works and I don’t see a problem in the strength. Where I see potential for an improvement is performance, without sacrificing strength too much. Trade-offs.

The CPU or software implementation of SHA256 is comparably slower to checksums with hardware aids (like CRC32C instructions) or hashes designed to perform well on CPUs. That was the topic of the previous round of new hashes, so we now compete against BLAKE2b and XXHASH. There are CPUs with native instructions to calculate SHA256 and the performance improvement is noticeable, orders of magnitude better. But the support is not as widespread as eg. for CRC32C. Anyway, there’s always choice and hardware improves over time. The number of hashes may seem to explode but as long as it’s manageable inside the filesystem, we take it. And a coffee please.

Secondary hash

The checksum scheme proposed is to use a cryptographic hash and a non-cryptographic one. Given the current support for SHA256 and BLAKE2b, the cryptographic hash is given. There are two of them and that’s fine. I’m not drawing an exact parallel with ZFS, the common point for the cryptographic hash is that there are limited options and the calculation is expensive by design. This is where the non-cryptographic hash can be debated. Also I want to call it secondary hash, with obvious meaning that it’s not too important by default and comes second when the authenticated hash is available.

We have CRC32C and XXHASH to choose from. Note that there are already two hashes from the start so supporting both secondary hashes would double the number of final combinations. We’ve added XXHASH to enhance the checksum collision space from 32 bits to 64 bits. What I propose is to use just XXHASH as the secondary hash, resulting in two new hashes for the authenticated and secondary hash. I haven’t found a good reason to also include CRC32C.

Another design point was where to do the split and truncation. As the XXHASH has fixed length, this could be defined as 192 bits for the cryptographic hash and 64 bits for full XXHASH.

Here we are, we could have authenticated SHA256 accompanied by XXHASH, or the same with BLAKE2b. The checksum split also splits the decision tree what to do when the checksum partially matches. For a single checksum it’s a simple yes/no decision. The partial match is the interesting case:

primary (key available) hash matches, secondary does not – as the authenticated hash is hard to forge, it’s trusted (even if it’s not full length of the digest)
primary (key available) does not match, secondary does not – checksum mismatch for the same reason as above
primary (key not available) does not match, secondary does – this is the prime time for the secondary hash, the floor is yours

This leads to 4 outcomes of the checksum verification, compared to 2. A boolean type can simply represent the yes/no outcome but for two hashes it’s not that easy. It depends on the context, though I think it still should be straightforward to decide what to do that in the code. Nevertheless, this has to be updated in all calls to checksum verification and has to reflect the key availability eg. in case where the data are auto-repaired during scrub or when there’s a copy.

Performance considerations

The performance comparison should be now clear: we have the potentially slow SHA256 but fast XXHASH, for each metadata and data block, vs slow SHA512 and slow SHA256. As I reckon it’s possible to also select SHA256/SHA256 split in ZFS, but that can’t beat SHA256/XXHASH.

The key availability seems to be the key point in all that, puns notwithstanding. The initial implementation assumed for simplicity to provide the raw key bytes to kernel and to the userspace utilities. This is maybe OK for a prototype but under any circumstances can’t survive until a final release. There’s key management wired deep into linux kernel, there’s a library for the whole API and command line tools. We ought to use that. Pass the key by name, not the raw bytes.

Key management has it’s own culprits and surprises (key owned vs possessed), but let’s assume that there’s a standardized way how to obtain the key bytes from the key name. In kernel its “READ_USER_KEY_BYTES”, in userspace it’s either keyctl_read from libkeyutils or a raw syscall to keyctl. Problem solved, on the low-level. But, well, don’t try that over ssh.

Accessing a btrfs image for various reasons (check, image, restore) now needs the key to verify data or even the key itself to perform modifications (check + repair). The command line interface has to be extended for all commands that interact with the filesystem offline, ie. the image and not the mounted filesystem.

This results to a global option, like btrfs --auth-key 1234 ispect-internal dump-tree, compared to btrfs inspect-internal dump-tree --auth-key 1234. This is not finalized, but a global option is now the preferred choice.

Final words

I have a prototype, that does not work in all cases but at least passes mkfs and mount. The number of checksum verification cases got above what I was able to fix by the time of writing this. I think this has enough matter on itself so I’m pushing it out out as part 1. There are open questions regarding the command line interface and also a some kind of proof or discussion regarding attacks. Stay tuned.

References:

[1] https://lore.kernel.org/linux-fsdevel/20200428105859.4719-1-jth@kernel.org/
[2] https://lwn.net/Articles/819143/ LWN discussion under Authenticated Btrfs (2020)