[0:00] Grayson Penland: Hi, I'm Grayson. A while back, Eduard and I spent almost two hours interviewing Chris Murphy about BTRFS, a copy-on-write file system. It was just too interesting to stop learning more about it. Today, I'm going to play you part one of that interview, because we don't have two hours for just one episode.

    [0:22] [background music]

    [0:22] Grayson: Today, you will get to learn about BTRFS, how it compares to ext4, and how you can utilize it in Fedora. You're listening to the "Fedora Podcast," a proud member of the Destination Linux Network.

    [0:44] Before we get into the technical BTRFS-related things, I want to know who is Chris Murphy, and how are you involved in Fedora? 

    Chris Murphy: [0:55] It depends on how far back you want to go. [laughs] But as far as it relates to Fedora, I've been involved in Fedora mostly on the QA side for many years, maybe pushing a decade at this point. I jump in on various development-related activities as they come off as the interests me and as I have two cents worth of party knowledge or opinion.

    [1:24] About two years ago, I joined the Fedora Workstation Working Group where I helped steer the direction of mainly Fedora Workstation edition, but we also incorporate the interests of all the desktop spins since we're all tending to adopt pretty much the same under the hood changes among all the desktop editions and spins.

    [1:49] Grayson: Cool. Before we get into talking about BTRFS itself, how do you pronounce, B-T-R-F-S, or better FS, or B-tree FS?

    [1:58] Chris: I pronounce BTRFS as butter FS.

    [2:01] Eduard Lucena: That's cool, because a lot of people just have different ways of pronounce. I just call it better FS. I remember that is coming I think for one of the features of BTRFS that word is called like that, but people just call it "butter." [laughs]

    [2:20] Chris: It's a really good point, Eduard. There is a hint to what is BTRFS in the name BTRFS, which is actually B-tree FS because it is made with a bunch of B-trees.

    [2:39] Eduard: Now we have talked about how we call it but want to people to know, what is BTRFS?

    [2:47] Chris: BTRFS is a copy-on-write file system with full data integrity checking for data. That would be user data and metadata, which is the file system itself. this also includes cheap snapshots, support for multiple devices, Things like RAID , RAID 1, RAID, 10, and also self-healing.

    [3:22] Depending on what the configuration of a particular BTRFS file system happens to be, if there's a known good copy and a bad copy BTRFS file, of course not use the bad copy and then it can automatically use the good copy and self-heal. self-healing is also a function of BTRFS.

    [3:44] Grayson: Where is BTRFS being actively used in the Fedora community right now?

    [3:52] Chris: BTRFS has been available as an installation time option in Fedora for quite a long time going back to at least Fedora 18. You can do a custom installation and choose to use BTRFS going back that far. What happened in Fedora 33 for Fedora Workstation and all the desktop spins is we switched the default. If you do a default or automatic installation of Fedora, you get BTRFS out of the box now.

    [4:28] Eduard: The point of view of normal user, what are the benefits of their laptop, or a server can benefit from BTRFS?

    [4:39] Chris: That's a good question. I would say that there's maybe not any one thing that's the most important to any one particular person, but there's a laundry list of pros and features that end up making it a really useful file system to use by default in Fedora.

    [5:06] Since I have to pick some things instead of listing the entire laundry list, what I'll pick is a common problem that we had with the previous default layout, which is ext4 on LVM. Since the system root and home file systems were separate file systems and didn't do space sharing, it was possible for the user to run out of free space on either one or the other.

    [5:36] You could run out of space on root, or you could run out of space on home while the other file system still had plenty of space. What we decided to do and take advantage of and BTRFS is the ability of creating one big file system, so there are no hard barriers, like partitions or logical volumes, that separate system root and home.

    [5:58] We do have a BTRFS-specific feature called subvolume, where we are able to still have a soft barrier between system root and home. They share space, but we're able to get some limited benefit of keeping them separate, which we can also talk about.

    [6:19] Another benefit that I think is probably beneficial to Fedora, specifically, is a bit of a catch-22, and that storage devices can be flaky, resulting in data corruption. That was a problem that BTRFS particularly is sensitive to detecting and identifying corruption.

    [6:43] Everything is checksummed both data and metadata. It's verified on every read, so corruption is immediately detected and is not allowed to propagate. You can trust that. Now, applications are not going to receive corrupt data on reads, and your backups are not going to be getting corrupt files propagated to them.

    [7:08] The flip side of that is that if you do end up with a data corruption scenario, BTRFS is a little bit noisy. It can look a little bit scary. A lot of the growing pain history of BTRFS is that it is geared toward data integrity, and it will complain, whereas other file systems are more permissive about that sort of thing just by design.

    [7:47] What else would I pick? Probably not too common, but when you need it, you really need it. If a file system is shrunk, BTRFS supports shrinking the file system, which ext4 also supports. BTRFS does it online, so you can shrink and grow the file system while it's mounted and in use.

    [8:12] Maybe last and not least is that complex storage setups are complicated. What we have with BTRFS is simpler or at least a more comprehensive and consistent command line interface using the BTRFS command and a list of subcommands. This, I think, tends to consolidate a lot of the routine tasks that folks may end up in. It's just a little bit easier to have less complexity and have it dealt with by BTRFS.

    [8:47] I actually have a better last one, which is for the most part, BTRFS is transparent, that is you get a lot of the benefits of BTRFS without having to know esoteric BTRFS things.

    [8:59] Grayson: That's awesome. Simple is good. You mentioned extended4 or ext4. It's the most common file system around for Linux distros as far as I've seen. How does BTRFS differ from ext4?

    [9:19] Chris: Ex4 is a overwriting file system in contrast to BTRFS, which is a copy-on-write file system. The difference when we are looking at categorizing different file systems, ext4 tends to overwrite files and metadata. When there's a change, it just overwrites that particular inode or that particular file, whereas with BTRFS, it never does that.

    [9:47] Copy-on-write basically means don't overwrite. What it does is it writes files and metadata into free space before it deletes or deallocates the now stale metadata or file data.

    [10:08] What we get with that is if the writing process is interrupted, for free, we get some data integrity features because there's no such thing as things like torn rights, where part of a file has been overwritten, and the file system hasn't yet been updated.

    [10:30] There's probably a quite a long list that I'm not entirely prepared to dig through on ext4 since I'm not as much of an expert with ext4 as I am with BTRFS or at least familiar with it, but I would say that that's probably one of the top things is the data integrity features and lack of snapshotting.

    [10:59] We do have snapshotting via LVM. They're not quite as cheap as they are in BTRFS, even though we're not using them in Fedora automatically yet. That's a segue into a snapshot in conversation.

    [11:14] Grayson: Which we will have later.

    [11:16] Chris: [laughs] Sure. As far as the regular everyday user, I would say you're probably not going to notice too much of a difference between ext4, which is still a perfectly reasonable choice to continue using. I have no criticisms of ext4. All these things are tradeoffs, and that includes BTRFS, which is also not perfect.

    [11:42] Grayson: You've also mentioned just now LVM. What's that? How does it compare to BTRFS subvoluming system?

    [11:51] Chris: Good question. That's a little intense. LVM, the logical volume manager in Linux, is a way that we're able to abstract physical devices from logical devices. We're able to perhaps more efficiently partition drives. It does support multiple devices. There are some similarities between LVM and BTRFS.

    [12:23] BTRFS has its own logical volume manager built into it. It's not LVM. It's not device mapper-based. It's its own thing, but it is reasonable to call it a logical volume manager, perhaps with small letters, [laughs] to avoid confusion with LVM.

    [12:43] Then as for BTRFS subvolumes, the BTRFS subvolumes are a little bit different. They don't really directly relate. We make comparisons because they're easier to understand. Some folks compare BTRFS subvolumes to LVM logical volumes.

    [13:03] The distinction there is that an LVM LV is a blocked device, so you can format an LV. You create a new LV, and you can format it. Ext4, XFS, you can format it at BTRFS. It can be swapped. It can be many things, whereas BTRFS subvolumes are more like directories. They're not blocked devices. You can't format them.

    [13:28] The comparison starts to break down once we look at the details, but for the most part, the way they behave is like a fancy directory.

    [13:39] Grayson: That's cool. I believe that BTRFS subvolumes are also...Aren't they easier to adjust like you can make them smaller or bigger and make them have to move around other subvolumes more easily than an LVM one?

    [13:54] Chris:A BTRFS subvolume, much like a directory, has no size. You don't need to specify a size. You just create one, and it shares all of the space available on the BTRFS file system you create that subvolume on.

    [14:15] There is an advanced feature set called quotas or qgroups on BTRFS. Qgroups work at the subvolume level, so you can set up quotas and limit the amount of space consumption on a per-subvolume basis. It has all kinds of inheritance rules and things that experts like.

    [14:37] Grayson: You could say their limits are a bit softer, like melted butter?

    [14:41] Chris: [laughs] Yes, you could say that. That's good.

    [14:46] Eduard: That's cool. I have a question because as a sysadmin and having my own lab here at home, I relied too much in tools like du and df, and this kind of tool that allowed me to know how much space I still have or the empty space I can use. How these tools are going to work with this new BTRFS? Are they a little bit tied to the block level?

    [15:22] Chris: Good question. There is a potential for confusion with the results from du and d. For the most part, they report correctly in the configuration we use by default in Fedora which is single device as in a single drive for the Fedora installation. Du and df are generally reliable. If not, then I would say that it's a bug.

    [15:59] We do know that there are some bugs. Perhaps more correctly, there's a need for enhancements in du and df as it relates to BTRFS. I think that as BTRFS becomes more popular or more used, then there will be more resources available to tighten up those kinds of things.

    [16:22] For the most part, I use, rely on, and trust du and df on BTRFS the same as on ext4 or XFS. Off the top of my head, I can't think of where the default configuration would get a user into trouble.

    [16:45] It's where you start to peel back layers of the onion, and you start doing some things, like if you're making snapshots, then a snapshot of a subvolume is an exact copy of that subvolume. It's the duplicated, but it's an exact copy.

    [17:06] If you were to make the snapshot of a subvolume inside that same subvolume and then run du, du will see and count both the subvolume and the snapshot contents. You'll get double the results with the du in a case like that.

    [17:24] There are some of those sorts of, "OK, well, how could we avoid doing that?" It's not really incorrect. It's a matter of perspective. Would you ignore snapshots by default? There's various conversations we'd have to have about that.

    [17:42] Another thing is du does report uncompressed data sizes. We're now using compression by default in Fedora 34, so d u will report the uncompressed size of files and directories, which depend on what your expectations are. If you expect that it's going to show the compressed size, then your expectations aren't going to be met.

    [18:17] Whereas with df, df does take compression into account because df is looking at the physical blocks that are used or are not used on the block device itself. Df indirectly takes into account the effect of compression. It's not compression aware per se, but it ends up counting things just that way because it looks at the physical block.

    [18:50] There are some difficulties that come up when we start doing certain kinds of multiple device scenarios. Df, I know of one bug that it gets a little bit confused if you do BTRFS RAID 1 with three devices That itself kind of an unusual configuration that BTRFS can do, and it can do it correctly and efficiently, but df will give you some confusing results.

    [19:21] In that case, it's an expert layout. I think most people will be reasonably satisfied in the near term and medium term using BTRFS-specific commands, and the one for that is BTRFS file system du and BTRFS file system usage.

    [19:40] Eduard: I have another question. I was excited when I heard about BTRFS that I wasn't going to have an initial problem that I have because I consider myself like a software tester. I used to DNF everything they can, [laughs] DNF to install in my machine, so I'm not going to run out of space in my root directory, in my root file system.

    [20:08] Now, I have the idea. A lot of people think of that I used to have this separate home partition because it was easier to recover when I was going to do a new fresh install. This is possible to do with BTRFS?

    [20:26] Chris:  That's a really good question. What we have is for a long time in Anaconda, the Fedora installer, allows us to do a custom installation so we can re-install Fedora and use an existing home subvolume.

    [20:50] The interface is not exactly obvious how to do this. That's maybe a future podcast, video, or some docs showing users how to do this. Once you see it, it's really straightforward and really easy to do. You can reuse an existing home without having to reformat BTRFS.

    [21:13] The requirement is unique on BTRFS. The installer doesn't require it to be reformatted when doing a clean installation. What it requires is just that there's a new root subvolume created so it'll install the new Fedora into that subvolume, and it will reuse the existing home subvolume. At that point, you're ready to go.

    [21:40] It initially looks like you've done a clean install, like you haven't set up your user home. You get the initial setup and all that. It might freak people out initially. Once you go through that process, voila, all of a sudden, when you log in, all of your stuff is there.

    [21:57] Grayson: I can attest to all of this. I'm running on a system with an old home subvolume where I reinstalled the root partition. I can say it was a little [laughs] non-intuitive. It did take a couple tries to get it right, but once you wrap your head around it, it does make sense. It's just not easy to tell at a glance.

    [22:22] Chris: It's not easy to articulate how you would do that if you didn't have a graphical user interface. Then once you have a graphical user interface, it's a complex enough procedure that I don't know how I would do it better than the way that they did it. I'm sure it could be done, but I think the solution to the problem is probably a video. [laughs]

    [22:44] Grayson: Maybe, we should ask the video team about that. That's all the time we have today for BTRFS. Thank you, Chris, for coming to talk with us.

    [22:55] In the next episode, we're going to continue the discussion and talk about BTRFS compression, snapshots, and the future of BTRFS and Fedora. Make sure you are subscribed so you won't miss out on any of this amazing interview.

    [23:11] Some quick reminders this week as part of the "What's going on Fedora?" section, the call for papers at NEST is still open. If you don't know what NEST is, make sure you listen to Season Two, Episode One of the Fedora Podcast where we speak with Marie Nordin about NEST. The call for papers at AnsibleFest is also open. AnsibleFest is virtual, and you can find it in our show notes.

    [23:37] The Annual Fedora Contributor Survey is still open throughout the month of June. If you're a contributor and haven't taken this survey, please do. If you don't know what the Annual Fedora Contributor Survey is, please go back and listen to our last episode, Season Two Episode Two, where we speak with Aleksandra Fedorova about the Fedora Contributor Survey.

    [24:02] Last weekend, the Fedora Project sponsored a couple of conferences, KDE's Akademy and the openSUSE Conference. I even recorded an interview for Academy with the KDE SIG where we talked Fedora KDE, a fellow Fedoran and Mindshare member, Onuralp Sezer. It recorded an interview for openSUSE.

    [24:23] Make sure you go check those out. They're both on YouTube and in the show notes at podcast.fedoraproject.org.

    [24:31] That's all the news we have for today. Thank you, Ben Cotton, for your Friday Fedora facts where we got the source for this news. Thank you for listening. That was Season Two, Episode Three of the Fedora Podcast. Please visit us at podcast.fedoraproject.org to find our chat room, send us email, listen to other episodes, and more.

    [24:54] Remember, we want your ideas for future Fedora Podcast episodes. If you maintain a cool package, contribute in an interesting way, or have worked on a Fedora thing that you think would be a good topic, please reach out, again, podcast.fedoraproject.org. There's a contact form. There's our chat room. There's our forum. There's our issue tracker. There's all kinds of ways to contact us. You have no excuse. Thank you.

    [25:25] I'm your host Grayson. An original music, this episode was by Tricknology. Special thanks this episode to Ryan Gorley. He created some new artwork for the Fedora Podcast incorporating our new logo and new color. It really looks great. He's helped integrated into the website properly so that it looks good, and he's just done an all-around amazing job with the art. Thank you, Ryan.

    [25:53] [background music]

    [25:53] Grayson: Make sure you subscribe, and we'll see you in a couple of weeks when we continue to talk about BTRFS. This is the Fedora Podcast, a proud member of the Destination Linux Network.