[SOLVED]Upgrade xfsprogs to 3.2 and make new checksum option default


Recommended Posts

I was about to migrate my entire array to XFS but the current xfsprogs version in beta12 only contains the version 3.1.11 that doesn't have the new XFS on-disk format (v5) introduced in version 3.2 and that includes a metadata checksum scheme called Self-Describing Metadata.

Based upon CRC32 it provides additional protection against metadata corruption during unexpected power losses for example. It also supposed to speed up filesystem checks.

Checksum is not enabled by default when using mkfs.xfs tool for formatting a drive. It can be easily done using the -m crc=1 switch when calling mkfs.xfs. Lastly, the XFS v5 on-disk format has been considered stable for production workloads starting Linux Kernel 3.15.

 

Is there any plan to upgrade xfsprogrs to version 3.2 in upcoming beta releases? I truly believe that this extra level of protection would be very beneficial for unRAID builds. I'm probably going to wait before migrating over to XFS.

 

xfsprogs 3.2.2 has been added to unRAID 6.0 release with the additional flags (-m crc=1,finobt=1) as default when formatting a drive

Link to comment
  • 1 month later...

Now, I may be wrong, but I glanced over the xfs.org site, and while the last status update posted is from 2013, that status described the metadata checksum as an experimental feature. There do not seem to be anything saying it ever left the experimental stage, so that may be why it's not the default even in the new xfsprogs.

 

Latest status update: http://xfs.org/index.php/XFS_status_update_for_2013

 

Link to comment

The XFS website doesn't advertise much on development progress.

Now you can see bellow that the new v5 format with metadata checksum and free inode btree is stable since Linux 3.15.

 

Kernel 3.10

http://kernelnewbies.org/Linux_3.10#head-a067455cdad0bf4e5285255ecfec5a538d930eb8

In this release, XFS has a experimental implementation of metadata CRC32c checksums. These metadata checksums are part of a bigger project that aims to implement what the XFS developers have called "self-describing metadata". This project aims to solve the problem of verification scalability (fsck will need too much time to verify petabyte scale filesystems with billions of inodes). It requires a filesystem format change that will add to every XFS metadata object some information that allows to quickly determine if the metadata is intact and can be ignored for the purpose of forensic analysis. metadata type, filesystem identifier and block placement, metadata owner, log sequence identifier and, of course, CRC checksum.

 

This feature is experimental and requires using experimental xfsprogs. For more information, you can read the self-describing metadata Documentation

 

Kernel 3.15

New v5 format is now considered stable

http://kernelnewbies.org/Linux_3.15

See commit: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c99d609a16506602a7398eea7d12b13513f3d889

 

Kernel 3.16

http://kernelnewbies.org/Linux_3.16#head-42e03cfe84cece01b59e07958221dc509de9715a

XFS free inode btree, for faster inode allocation

In this release, XFS has added a btree that tracks free inodes. It is equivalent to the existing inode allocation btree with the exception that the free inode btree tracks inode chunks with at least one free inode. The purpose is to improve lookups for free inode clusters for inode allocation.

 

This feature adds does not change existing on-disk structures, but adds a new one that must remain consistent with the inode allocation btree; for this reason older kernels will only be able to mount read-only filesystems with the free inode btree feature.

****

The feature can be enabled with finobt=1 switch when formatting a XFS partition but requires the metadata checksum to be enabled as well via the crc=1 switch. Therefore the following options are required to take advantage of both the free inode btree and metadata checksum

 

mkfs.xfs -m crc=1,finobt=1 /dev/target_partition

 

According to the developers, the above -m crc=1,finobt=1 option is not only safe to use now but will be the default mkfs option with the upcoming xfsprogs 3.3 release.

Link to comment
  • 2 weeks later...
  • 1 month later...

Does someone know if the Slackware team has any plan to migrate xfsprogs to 3.2.x (I tried to get in touch in them but without much luck)? Would it be possible to update it on unRAID without waiting for the Slack team?

unRAID v6 uses a pretty recent kernel that has provision for the latest XFS improvements such as v5 format (metadata checksum) and free inode btree. All considered to be in a stable phase. The only missing piece is xfsprogs and to make the new settings default when formatting. Those are some really nice safety features that should be worth considering for a multi disk array. Thanks for your understanding.

Link to comment
  • 2 weeks later...

Yes,  it is theoretically possible to update the package yourself, however you'll be the one doing all the work. Sometimes you get lucky and you can reuse the previous versions .SlackBuild file, other times you have to work through all sorts of issue.

 

I suggest you send LimeTech a direct PM making sure they're awareof this request.

Link to comment
This feature adds does not change existing on-disk structures, but adds a new one that must remain consistent with the inode allocation btree; for this reason older kernels will only be able to mount read-only filesystems with the free inode btree feature.

 

I don't know at all as much as you guys, but the statement above concerns me.  If I'm reading it right, you want a more recent version with the free inode tree feature and the checksum feature, but a drive formatted previously would not be read/write compatible with one formatted with these features turned on?  Does this mean a very similar process to having to convert a drive from ReiserFS to XFS?  Or does it just mean you can go forward (and it will be a straightforward easy process), but you cannot go back, once you've turned those features on?

Link to comment

This feature adds does not change existing on-disk structures, but adds a new one that must remain consistent with the inode allocation btree; for this reason older kernels will only be able to mount read-only filesystems with the free inode btree feature.

 

I don't know at all as much as you guys, but the statement above concerns me.  If I'm reading it right, you want a more recent version with the free inode tree feature and the checksum feature, but a drive formatted previously would not be read/write compatible with one formatted with these features turned on?  Does this mean a very similar process to having to convert a drive from ReiserFS to XFS?  Or does it just mean you can go forward (and it will be a straightforward easy process), but you cannot go back, once you've turned those features on?

 

Unraid 6 betas have kept up with linux kernel releases like there's no tomorrow (and like a stable release is never meant to arrive):

- Beta 6 Linux 3.15.0

- Beta 7/8 Linux 3.16.0

- Beta 9 Linux 3.16.2

- Beta 10 Linux 3.16.3

- Beta 12 Linux 3.16.5

- Beta 13/14 Linux 3.18.5

 

See in the 3.rd post above that the format was considered stable since kernel 3.15. Thus, the new format is already supported (and considered stable according to the XFS authors), in unraid versions starting with v6 beta6.

 

Needless to say, the XFS filesystem that comes with these kernels are backward compatible otherwise XFS wouldn't work for us! As long as you don't plan on reverting to a beta prior to beta6, you shouldn't have a problem.

 

The only thing being requested here is the tool that is used for formatting drives (which is not part of kernel), is updated to a recent version that includes the new switch. IMHO it's pretty lame to be updating kernels like we change our underwear, but stick to an ancient version of the format tool.

Link to comment

Unraid 6 betas have kept up with linux kernel releases like there's no tomorrow (and like a stable release is never meant to arrive):

- Beta 6 Linux 3.15.0

- Beta 7/8 Linux 3.16.0

- Beta 9 Linux 3.16.2

- Beta 10 Linux 3.16.3

- Beta 12 Linux 3.16.5

- Beta 13/14 Linux 3.18.5

 

See in the 3.rd post above that the format was considered stable since kernel 3.15. Thus, the new format is already supported (and considered stable according to the XFS authors), in unraid versions starting with v6 beta6.

 

Needless to say, the XFS filesystem that comes with these kernels are backward compatible otherwise XFS wouldn't work for us! As long as you don't plan on reverting to a beta prior to beta6, you shouldn't have a problem.

 

The only thing being requested here is the tool that is used for formatting drives (which is not part of kernel), is updated to a recent version that includes the new switch. IMHO it's pretty lame to be updating kernels like we change our underwear, but stick to an ancient version of the format tool.

 

And beta15 will have Linux Kernel 4.0.x, but no word yet on getting an update on xfsprogs.

Link to comment

Can we please get an answer from LT about the following:

 

-estimate when new version of xfs tools will make it into unRAID,

-is there any issue with manually formatting disks with additional switches (outside of unRAID) and using them with current unRAID versions?

-does any of the new xfs feature seem useful to LT in the future?

 

Rationale: if possible update xfstools asap and add those switches to 'format' command by default. Even if the new features are not going to be used any time soon by unRAID, that will save another round of moving data and re-formatting disks.

 

Thank you.

Link to comment

Can we please get an answer from LT about the following:

 

-estimate when new version of xfs tools will make it into unRAID,

-is there any issue with manually formatting disks with additional switches (outside of unRAID) and using them with current unRAID versions?

-does any of the new xfs feature seem useful to LT in the future?

 

Rationale: if possible update xfstools asap and add those switches to 'format' command by default. Even if the new features are not going to be used any time soon by unRAID, that will save another round of moving data and re-formatting disks.

 

Thank you.

 

Is it sufficient to just update the package without the new switch statements yet?  Adding the switch statements is relatively trivial, but without having had a chance to thoroughly test this yet, would just getting an update to xfs progs 3.2.2 satisfy this for now?

 

Link to comment

jonp->

 

It's a good first step.

According to XFS wiki those switches (-m crc=1,finobt=1) will be invoked by default in upcoming xfsprogs 3.3 - mkfs.xfs

They create additional data structures during formatting and the only downside, which is not relevant to unRAID, is that kernels <3.15 will mount such disk as RO.

Link to comment

jonp->

 

It's a good first step.

According to XFS wiki those switches (-m crc=1,finobt=1) will be invoked by default in upcoming xfsprogs 3.3 - mkfs.xfs

They create additional data structures during formatting and the only downside, which is not relevant to unRAID, is that kernels <3.15 will mount such disk as RO.

Ok, we saw the same info as well.  Will provide an update on this later today.

Link to comment

Ok, so we took a quick shot at this today to see if it would be simple enough for inclusion in beta 15 but the make process failed during configuration, which means it'll take a bit more work on our part to update this (not a ton, but not going to happen for beta 15).

 

It's also worth noting that the CRC benefit here isn't to data, but rather, metadata, which means this feature request for checksumming isn't really going to do much for improving data integrity, but could improve performance for file system check operations.  This is a minor feature.  If it was actually checksumming the real data like BTRFS does, then it'd be a major feature.  This is why updating XFS progs hasn't been a big priority for us.  There isn't much to be gained here.

Link to comment

It's also worth noting that the CRC benefit here isn't to data, but rather, metadata

 

Ah, thanks for explaining that.  I was wondering why this wasn't getting more traction.

 

On the flip side, we were over here wondering why so much fuss about xfs progs ;-).  It's not like we wouldn't upgrade them eventually anyway, but there just really is no good reason to prioritize it right now as the new progs won't change really any major functionality for the most part.

Link to comment

It's also worth noting that the CRC benefit here isn't to data, but rather, metadata

 

Ah, thanks for explaining that.  I was wondering why this wasn't getting more traction.

 

On the flip side, we were over here wondering why so much fuss about xfs progs ;-).  It's not like we wouldn't upgrade them eventually anyway, but there just really is no good reason to prioritize it right now as the new progs won't change really any major functionality for the most part.

 

I'd say its less fuss over that and more fuss over this request sitting out there since December 2014 without being addressed.

Link to comment

I'm not hellbent on getting this feature. But there are still reasons you should be interested in it. It hardens the filesystem against power interruptions, and keep in mind new users who set out to evaluate UnRAID may now yet own an UPS. Put two and two together, your product will cause those new potential customers less grief if and when they experience their first power loss. Recovery from a power loss is the one weakness XFS seems to have over other traditional filesystems.

 

What irks me is the pace with which you have changed kernels without updating simple stuff like this. I completely agree, with the new xfsprogs the default should be the old format, that way advanced users could still start to evaluate the new format, just to be safe.

 

Jon, mocking androidcat is kind of unprofessional. Of course users will only need to reformat if they want to take advantage of the V5 format, and I think we all understand that.

Link to comment

Jon, mocking androidcat is kind of unprofessional. Of course users will only need to reformat if they want to take advantage of the V5 format, and I think we all understand that.

 

?  I'm not mocking him.  I was just really shocked at what he posted (upgrades to XFS = reformat for existing users of a previous version).  Hence the  :o

Link to comment

What I understood reading xfs docs:

Kernels above 3.14 support v5 of xfs without any additional work,

To use v5 you have to do mkfs.xfs with additional switches and that will create additional  structures on disk allowing new functionalities,

UnRAID will not have problems handling either old or v5 format as it has kernel new enough to support both,

Newest version of xfsprogs is needed to format HDD with v5,

Upcoming xfsprogs 3.3 will format disks with v5 by default (no cmdline switches needed).

 

Seems pretty straightforward to me and no harm is done to anybody.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.