archedraft Posted July 10, 2015 Share Posted July 10, 2015 I can make that an option in bunker to send notifications when corruption is detected. That sounds like a much better option than me riggin something up. Although i will say I did manage to figure out the command I probably need is "notifiy smpt-init" so I was getting somewhere! But it makes more sense for it to be built-in. Do you think it would be possible for the email to print the log in the body of the email? That way you could just read the email and find the file without having to look through logs. Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 OK so here are the two scripts I created. One question one the first script. $bunker -a -q -l -f $log/new.disk1.$day /mnt/disk1 && $bunker -U -q -l -f $log/updates.disk1.$day /mnt/disk1 & Will this first add checksums to new files AND THEN update any user modified files? Just wanted to make sure bunker will not add new and update at the same time. Although I guess the & at the end ins't going to allow disk2 to start that the same time as disk1 because of the &&? bunker-newupdate #!/bin/bash bunker=/boot/bunker var=/var/local/emhttp/var.ini log=/boot/logs/bunker day=$(date +%Y%m%d) array=$(awk -F'"' '/^fsState=/{print $2}' $var) rsync=$(awk -F'"' '/^mdResync=/{print $2}' $var) # Every day add new and update user modified checksums if [[ $array == Started && $rsync -eq 0 ]]; then $bunker -a -q -l -f $log/new.disk1.$day /mnt/disk1 && $bunker -U -q -l -f $log/updates.disk1.$day /mnt/disk1 & $bunker -a -q -l -f $log/new.disk2.$day /mnt/disk2 && $bunker -U -q -l -f $log/updates.disk2.$day /mnt/disk2 & $bunker -a -q -l -f $log/new.disk3.$day /mnt/disk3 && $bunker -U -q -l -f $log/updates.disk2.$day /mnt/disk3 & fi bunker-verify #!/bin/bash bunker=/boot/bunker var=/var/local/emhttp/var.ini log=/boot/logs/bunker day=$(date +%Y%m%d) array=$(awk -F'"' '/^fsState=/{print $2}' $var) rsync=$(awk -F'"' '/^mdResync=/{print $2}' $var) # Every month verify checksums if [[ $array == Started && $rsync -eq 0 ]]; then $bunker -V -q -l -f $log/verify.disk1.$day /mnt/disk1 & $bunker -V -q -l -f $log/verify.disk2.$day /mnt/disk2 & $bunker -V -q -l -f $log/verify.disk3.$day /mnt/disk3 & fi Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 I can make that an option in bunker to send notifications when corruption is detected. That sounds like a much better option than me riggin something up. Although i will say I did manage to figure out the command I probably need is "notifiy smpt-init" so I was getting somewhere! But it makes more sense for it to be built-in. Do you think it would be possible for the email to print the log in the body of the email? That way you could just read the email and find the file without having to look through logs. Ok, new version 1.8 of bunker is available (see OT). This version has the new option -n (notify) which let bunker send notifications when file corruption is detected. File corruption is sent as an "alert" message, so make sure you have configured the correct recipients (browser, email or agents) to receive the alert. Only emails will have a message body detailing the corrupted file names, therefore I would advice to select at least email as a recipient. Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 I can make that an option in bunker to send notifications when corruption is detected. That sounds like a much better option than me riggin something up. Although i will say I did manage to figure out the command I probably need is "notifiy smpt-init" so I was getting somewhere! But it makes more sense for it to be built-in. Do you think it would be possible for the email to print the log in the body of the email? That way you could just read the email and find the file without having to look through logs. Ok, new version 1.8 of bunker is available (see OT). This version has the new option -n (notify) which let bunker send notifications when file corruption is detected. File corruption is sent as an "alert" message, so make sure you have configured the correct recipients (browser, email or agents) to receive the alert. Only emails will have a message body detailing the corrupted file names, therefore I would advice to select at least email as a recipient. Awesome! Thank you very much! Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 Any thoughts on why I am getting this error: I run this command: bunker -U -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 Specified file doesn't exist If I remove -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) then it runs. ALSO: I typed this and got this message: bunker -a -l /mnt/disk3 && bunker -U -l /mnt/disk3 ./bunker: line 414: plural$: command not found Quote Link to comment
tr0910 Posted July 10, 2015 Share Posted July 10, 2015 Bunker -a lockup. Processing file 152 of 195168, and rather than processing as normal, it just stopped there for several hours. Ctrl C would not break out of it. Htop reports the following: Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 Any thoughts on why I am getting this error: I run this command: bunker -U -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 Specified file doesn't exist If I remove -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) then it runs. ALSO: I typed this and got this message: bunker -a -l /mnt/disk3 && bunker -U -l /mnt/disk3 ./bunker: line 414: plural$: command not found You've found two bugs, I have corrected them and will issue version 1.9 with bug fixes. Thanks. Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 Bunker -a lockup. Processing file 152 of 195168, and rather than processing as normal, it just stopped there for several hours. Ctrl C would not break out of it. Htop reports the following: I had similar issues when using ReiserFS as the file system, once in a while the writing of the extended attributes causes a hang up. Since moving to XFS I never had this problem anymore. Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 Version 1.9 now available. Quote Link to comment
tr0910 Posted July 11, 2015 Share Posted July 11, 2015 Bunker -a lockup. Processing file 152 of 195168, and rather than processing as normal, it just stopped there for several hours. Ctrl C would not break out of it. Htop reports the following: I had similar issues when using ReiserFS as the file system, once in a while the writing of the extended attributes causes a hang up. Since moving to XFS I never had this problem anymore. I have run bunker on 30tb of rfs and this was the first I've seen this. Would the new versions (post 1.5) be more likely to see this? Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 I have run bunker on 30tb of rfs and this was the first I've seen this. Would the new versions (post 1.5) be more likely to see this? All versions prior to 1.7 did write the same extended attributes, and no change in coding for that part. From version 1.7 onwards the new extended attribute filedate is added, though coding-wise the same approach is still done. Personally I lost a bit faith in RFS under v6 and moved all to XFS, so far I am quite pleased with the result. Quote Link to comment
tr0910 Posted July 11, 2015 Share Posted July 11, 2015 Yes I plan to move to xfs too, but need a life changing reason to do so. So far v6 has new disks xfs old disks rfs. Quote Link to comment
archedraft Posted July 11, 2015 Share Posted July 11, 2015 Thanks for 1.9 I can confirm that both messages go away. I think something is still messed up with the -f flag. It does not create the log in the folder I have specified. I have tried all the variations of but it never creates the log file in the folder (even when I made new files and made changes to the files) bunker -a -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 bunker -U -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 bunker -V -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 Also, I think I found another bug as well bunker -a -l -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 Scanning for files to add... - awk: cmd. line:1: fatal: division by zero attempted Quote Link to comment
archedraft Posted July 11, 2015 Share Posted July 11, 2015 Yes I plan to move to xfs too, but need a life changing reason to do so. So far v6 has new disks xfs old disks rfs. FWIW, I have also moved all my disk (I only have 3) to xfs and have been very pleased with it as well. Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 Also, I think I found another bug as well I released version 1.10 which has a small correction that may prevent this bug (can't really reproduce myself) and also in v1.10 is a more comprehensive reporting, telling the execution duration and average hashing speed after the task is finished. Quote Link to comment
bonienl Posted July 11, 2015 Author Share Posted July 11, 2015 Addendum: you can use the command -e (export) to create a file explicitely. bunker -e -f /boot/logs/bunker/new.disk3.$(date +%Y%m%d) /mnt/disk3 Quote Link to comment
bonienl Posted July 12, 2015 Author Share Posted July 12, 2015 I have created two scripts for daily and monthly processing. These might be useful for you, but you need to adapt them to your needs, see the red sections. Make sure you are running the latest version of bunker. Update: adjusted bunker-daily to use command -U -D 1 (which speeds up execution by limiting to files changed in the last day) bunker-daily is copied to folder /etc/cron.daily and checks for new additions, reports about errors for changed files and keeps the export file up to date. #!/bin/bash [color=red]bunker=/boot/custom/bin/bunker[/color] [color=red]log=/boot/custom/hash[/color] var=/proc/mdcmd day=$(date +%Y%m%d) array=$(grep -Po '^mdState=\K\S+' $var) rsync=$(grep -Po '^mdResync=\K\S+' $var) mkdir -p $log # Daily check on new files, report errors and create export file if [[ $array == STARTED && $rsync -eq 0 ]]; then for i in [color=red]1 2 3 4 5 6 7 8 9 10[/color] ; do $bunker -A -D 1 -q -l -f $log/disk$i.$day.new.txt /mnt/disk$i $bunker -U -D 1 -q -l -n -f $log/disk$i.$day.bad.txt /mnt/disk$i if [[ -s $log/disk$i.export.txt ]]; then if [[ -s $log/disk$i.$day.new.txt || -s $log/disk$i.$day.bad.txt ]]; then mv $log/disk$i.export.txt $log/disk$i.$day.txt $bunker -e -q -l -f $log/disk$i.export.txt /mnt/disk$i fi else $bunker -e -q -l -f $log/disk$i.export.txt /mnt/disk$i fi done fi bunker-monthly is copied to folder /etc/cron.monthly and verifies a group of disks on file corruption. #!/bin/bash [color=red]bunker=/boot/custom/bin/bunker[/color] [color=red]log=/boot/custom/hash[/color] var=/proc/mdcmd day=$(date +%Y%m%d) array=$(grep -Po '^mdState=\K\S+' $var) rsync=$(grep -Po '^mdResync=\K\S+' $var) mkdir -p $log # Monthly verification of different group of disks (quarterly rotation) if [[ $array == STARTED && $rsync -eq 0 ]]; then case $(($(date +%m)%4)) in 0) for i in [color=red]1 2[/color] ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 1) for i in [color=red]3 4 5[/color] ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 2) for i in [color=red]6 7 8[/color] ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 3) for i in [color=red]9 10[/color] ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; esac fi bunker-daily.txt bunker-monthly.txt Quote Link to comment
bonienl Posted July 12, 2015 Author Share Posted July 12, 2015 For the brave users ... You can adjust the script bunker-daily to check for files changed in the last hour and copy it to /etc/cron.hourly (might change the name to bunker-hourly). $bunker -A [color=red]-D 1h[/color] -q -l -f $log/disk$i.$day.new.txt /mnt/disk$i $bunker -U [color=red]-D 1h[/color] -q -l -n -f $log/disk$i.$day.bad.txt /mnt/disk$i Quote Link to comment
tr0910 Posted July 12, 2015 Share Posted July 12, 2015 Thanks for the example scripts. Looks like you set it up for a 10 disk array? I could use your monthly script on a weekly basis for a larger server couldn't I ? Quote Link to comment
bonienl Posted July 12, 2015 Author Share Posted July 12, 2015 Thanks for the example scripts. Looks like you set it up for a 10 disk array? I could use your monthly script on a weekly basis for a larger server couldn't I ? Sure you can. Below a weekly example for 16 drives (the reason it is split over several weeks is because a complete scan/verification per disk is done). Running 4 disks concurrently is about the max my processor can do, yours may be different. #!/bin/bash bunker=/boot/custom/bin/bunker log=/boot/custom/hash var=/proc/mdcmd day=$(date +%Y%m%d) array=$(grep -Po '^mdState=\K\S+' $var) rsync=$(grep -Po '^mdResync=\K\S+' $var) mkdir -p $log # Weekly verification of different group of disks (monthly rotation) if [[ $array == STARTED && $rsync -eq 0 ]]; then case $(($([color=red]date +%U[/color])%4)) in 0) for i in 1 2 3 4 ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 1) for i in 5 6 7 8 ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 2) for i in 9 10 11 12 ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; 3) for i in 13 14 15 16 ; do $bunker -v -n -q -l -f $log/disk$i.$day.bad.txt /mnt/disk$i & done ;; esac fi Quote Link to comment
archedraft Posted July 12, 2015 Share Posted July 12, 2015 Thanks for the scripts. If we manually copy the scripts to their respective cron folders do they survive a reboot or do we need to set up a cp command in our go files to copy them in on every reboot? Quote Link to comment
bonienl Posted July 12, 2015 Author Share Posted July 12, 2015 Thanks for the scripts. If we manually copy the scripts to their respective cron folders do they survive a reboot or do we need to set up a cp command in our go files to copy them in on every reboot? After a reboot files will be gone, you need to copy them back. I have the following in my go file # Copy scheduler programs cp /boot/custom/bin/bunker-daily /etc/cron.daily cp /boot/custom/bin/bunker-monthly /etc/cron.monthly Quote Link to comment
archedraft Posted July 13, 2015 Share Posted July 13, 2015 The bunker-daily script threw up a bunch of errors before I canceled it. EDIT: I think I had a / at the end of the log path. I removed it and it appears to be running OK now. Quote Link to comment
archedraft Posted July 13, 2015 Share Posted July 13, 2015 I have a question with the bunker-monthly script. Does it ever get triggered at the same time as parity check? If so how do you handle that? Quote Link to comment
archedraft Posted July 13, 2015 Share Posted July 13, 2015 sooo I am doing some testing with the bunker-daily script (sorry) and whenever I add a test file to disk1 and run bunker-daily I get the following errors. I think it is because it is trying and failing to create the new txt log of the added file. I will also attach my modified bunker-daily file, just in case I messed it up. bunker-daily.txt Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.