bonienl

Dynamix File Integrity plugin

614 posts in this topic

WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS.

 

UPDATE

Version 2016.01.05 marks the official release of this plugin

This plugin is still work in progress and provides a GUI interface to the (new) bunker utility.

 

It has two sections: settings and control

 

The settings page specifies how automatic file hashing and verification is done. These are processes which - once started - will keep running in the background and notify the user when a silent file corruption is detected.

 

For the automation of the hashing of new and modified files, the program inotifywait is used, this is part of the inotify-tools package which is installed together with the plugin as needed (it is not present on unRAID by default unless the NerdPack package is installed). It is possible to exclude both disks and folders (shares) from the hashing system.

 

The verification process is based on cron entries. The user can define groups of disks which are verified at the given schedule. For example if a weekly schedule is chosen then every other group is verified the following week. With setting up groups it is possible to do all in once or spread out the load over several windows. This is done because the disk verification processes are started in parallel, the bigger a group the more horse power is required from your processor.

 

The control page is used for general operations. Here it is possible to select the disks and choose from the commands: build, remove, export, import, check.

 

An initial build needs to be done to have a checksum value added to the existing files. Building the checksums can be done either in a single go (when all disks are selected), or with intermediate steps (each time another disk(s) is selected.

 

Once the initial build is complete it is possible to export the checksum values to a text file (normal operation is that a checksum value is linked to a file directly using its extended attributes).

 

With the export files present (they are stored on flash), it is possible to import then back into the extended attributes or do a manual check to find file corruption.

 

Once an operation is started its progress is shown in real-time, and includes an ETA for those operations which tend to take a long time.

 

There is no need to stay at the same page or keep the browser open all the time. Whenever the control page is revisited from the same browser, it will update itself to the latest state.

 

Attached the settings and control screens to give an indication of what it looks like

 

If you want to use this functionality, you can install the plugin using the plugin manager. Copy and paste this URL

 

https://raw.githubusercontent.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg

 

People who are using an earlier version of bunker and associated cron jobs should stop and uninstall them, as they will interfere with this package.

 

Extensive HELP is available on the GUI pages. Please read this carefully before setting up and using this new functionality.

 

Hope you like this new addition to the Dynamix family.

file-integrity-settings.png.f1999ed3eab9a3181d9e695c6cd49e43.png

file-integrity-control.png.b5a18822340320286bb2d7db9876d48e.png

2

Share this post


Link to post
Share on other sites

for those of us trying to understand file integrity apps..

 

is this the same ? http://lime-technology.com/forum/index.php?topic=43396.msg414410#msg414410

no. 

 

This is an alternative approach and only provides file integrity checking via checksumming (it has no recovery capability).    It also stores the checksum information within the file system metadata so you do not see separate checksum files on the system.

0

Share this post


Link to post
Share on other sites

The checksum utility of squid is also dealing with file integrity (or bit rot) detection, but is a different approach, as explained by itimpi.

 

The approach of bunker is to store the relevant information in the extended attributes which go with the file. Besides the checksum value also the file modification date, file size and hash check date are stored in the extended attributes. This allows bunker to check for silent file corruption, i.e. the file content has changed but not the file modification date. A notification is sent and optional logging is done when such an event occurs.

 

This utility only reports file corruption but does not have any means to repair it. This action is with the user.

 

To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens.

 

The GUI plugin has the ability to visualize the background operations carried out by bunker. On the control page this is made visible, e.g. when starting a build operation you can follow here its proceedings, including an estimated time to finish for each disk which was included. And no worries leaving the page, all keeps on running and a revisit will show the current status.

 

I've made a few fixes and released version 2015.12.29a. Those who do an upgrade, you must disable/enable the automatic hash function after installation to ensure that inotifywait is running (again).

 

0

Share this post


Link to post
Share on other sites

To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens.

 

Just installed and it looks great! Awesome work as always. I removed my old checksums and decided to use blake2 so it is recreating them as I type this. So inotifywait will add any new or modified file automatically?

0

Share this post


Link to post
Share on other sites

To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens.

 

Just installed and it looks great! Awesome work as always. I removed my old checksums and decided to use blake2 so it is recreating them as I type this. So inotifywait will add any new or modified file automatically?

 

inotifywait watches for file changes and whenever a new file or modified file is saved to disk, it triggers bunker with the necessary options to do the checksum calculation and add this to the extended attributes.

 

This all goes transparantly for the user, and there should be minimum performance impact in the normal file operations.

 

0

Share this post


Link to post
Share on other sites

I have made a new beta version 2015.12.30b available. Please update to this version.

 

After installation you need to stop and start the automatic protection function (see settings page) to allow the updated version to work.

 

Thanks for testing.

 

0

Share this post


Link to post
Share on other sites

inotifywait watches for file changes and whenever a new file or modified file is saved to disk, it triggers bunker with the necessary options to do the checksum calculation and add this to the extended attributes.

 

This all goes transparantly for the user, and there should be minimum performance impact in the normal file operations.

 

Wow that's pretty slick. From your pictures above it looks like you are using blake2; however, the help section says that "this is a proprietary solution though". What (if any) are the potential risks of using blake2 being proprietary?

0

Share this post


Link to post
Share on other sites

I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works.

 

Great work as usual!

 

PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable?

 

0

Share this post


Link to post
Share on other sites

There is no risk in using BLAKE2, it is just not acknowledged as an official standard, though these guys promote it a lot.

 

I am using BLAKE2 indeed, it is blistering fast.

 

0

Share this post


Link to post
Share on other sites

I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works.

 

Great work as usual!

 

PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable?

 

The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not.

 

Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched.

 

0

Share this post


Link to post
Share on other sites

I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works.

 

Great work as usual!

 

PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable?

 

The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not.

 

Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched.

You can check to see if blake2 works at startup by running it against something and then disabling it depending upon the error code returned.  I'll look at my code for this later tonight

 

0

Share this post


Link to post
Share on other sites

Excellent idea, will add that and let me know what you have in mind.

 

Meanwhile another update version 2015.12.30c is available with several corrections and enhancements.

 

After installing this update you need to stop/start the service (see settings page) to make the changes active.

 

I have changed the naming of the export files. If you have exported files earlier, you need to redo and discard the old ones (see also the release notes for more details).

 

 

 

0

Share this post


Link to post
Share on other sites

Little bug, with SHA2 selected in settings, Hashing method in File integrity control appears as none, works as expect with MD5 or BLAKE2.

0

Share this post


Link to post
Share on other sites

Little bug, with SHA2 selected in settings, Hashing method in File integrity control appears as none, works as expect with MD5 or BLAKE2.

 

ok, thanks it's corrected for a next release.

 

0

Share this post


Link to post
Share on other sites

I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works.

 

Great work as usual!

 

PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable?

 

 

The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not.

 

Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched.

 

Well, it looks like BLAKE2 works after all on my CPU, I remember failing a test on Squid’s Checksum plugin, because SSE4.1 was required.

0

Share this post


Link to post
Share on other sites

Hope you guys can keep up with the pace ;)

 

Version 2015.12.25d adds a new option, does a check on BLAKE2 support and some bug fixing.

 

See the release notes for more details.

 

0

Share this post


Link to post
Share on other sites

 

 

Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched.

 

So if I were to copy some hashed files to a backup server running the same plugin I'd have to rehash them?

 

0

Share this post


Link to post
Share on other sites

 

 

Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched.

 

So if I were to copy some hashed files to a backup server running the same plugin I'd have to rehash them?

 

If you have the plugin running on your backup server, as soon as you copy files to it, they will get their checksum automatically added.

 

One quick way to set all extended attributes on your backup server is to do an export command on your main server, copy the hash file over to your backup server and perform an import command.

 

0

Share this post


Link to post
Share on other sites
Whenever the control page is revisited it will update itself to the latest state.

While running a build the statement above works for the same browser only. If you load the control page on a different browser or computer it will not show the latest state.

0

Share this post


Link to post
Share on other sites

I followed the bunker thread but never tried bunker because it didn't really fit with what I needed. I create NTFS disks containing specific unRAID shares for offsite backups. I assume the extended attributes hash would not transfer to the NTFS files, and I didn't want to have to manually export these every time I want to make a backup. I am currently using Squid's checksum since it creates separate hash files which get copied to NTFS along with everything else, and it works with shares or disks. I do like some aspects of the UI you have created for this though.

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.