bblue Posted September 26, 2009 Share Posted September 26, 2009 I run mover every hour on my machine, and more often than not it spins up all the drives each time. It's caused by the 'sync' command. There doesn't seem to be any reason to spin the drives up if there's nothing for mover to do, so I made a change in mover to test the cache drive before anything else. If cache is empty, skip over the finds and sync operations. If cache is non-empty, everything proceeds as usual. Here's just the code part of my mover which also leaves the top level share hierarchies instead of removing them (-mindepth 2 in the second find if you don't want that). Some might find this useful. # Only run script if cache disk enabled if [ ! -d /mnt/user0 ]; then exit 0 fi # If a previous invokation of this script is already running, exit if [ -f /var/run/mover.pid ]; then if ps h `cat /var/run/mover.pid` | grep mover ; then echo "mover already running" exit 0 fi fi echo $$ >/var/run/mover.pid # print without a newline echo -n mover started sz=$(du -s /mnt/cache | cut -f1) if [ $sz == 0 ] then echo ":no work" else echo (cd /mnt/cache ; \ find . \ \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; -exec echo moving {} \; -exec mv -v {} /mnt/user0/{} \; \) ; \ find . -mindepth 2 -type d -regex '[.]/[^.].*' ! -exec fuser -s {} \; -empty -delete \ ) sync fi echo "mover finished" rm /var/run/mover.pid --Bill Quote Link to comment
justinkas Posted November 13, 2009 Share Posted November 13, 2009 Hi Bill, Newbie here...Would you mind explaining exactly how to add this to my unraid box please? I'm not sure how & where to place this code? Thanks! Quote Link to comment
RobJ Posted November 13, 2009 Share Posted November 13, 2009 I believe this should be helpful: Mover Quote Link to comment
PeterB Posted September 28, 2010 Share Posted September 28, 2010 Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread. Apart from the fact that the current version uses rsync instead of mv, this modification looks okay. However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories. I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command. So, it occurred to me that what is required is to suppress the sync when no files have been moved. Would something like the following have the desired effect? .... echo "mover started" # Modification to prevent drives being spun up if there's nothing in the write cache # (but allowing files to exist in hidden directories). moved=0 (cd /mnt/cache ; find -depth -print \ \( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \ -exec moved=1 \; \ -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \ \( -type d -regex '[.]/[^.].*' \ -exec moved=1 \; \ -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \ ) if [ $moved == 0 ]; then echo "nothing to move" else sync fi rm /var/run/mover.pid .... Quote Link to comment
Joe L. Posted September 28, 2010 Share Posted September 28, 2010 No it would not work. there is no command "moved=1" to execute AND even if there was a shell involved it is a child shell and any variables set in it are not visible in the parent shell's environment AND the way you coded it would have been exec'ed if any file or directory exists in the cache drive, not if any are moved. Far easier to just remove the "sync" command. (and get a UPS so an un-expected shutdown will not occur) or add these commands after it (one line for each of your drives). sync /root/mdcmd spindown 1 /root/mdcmd spindown 2 /root/mdcmd spindown 3 /root/mdcmd spindown 4 The spindown commands will not affect a drive currently being used but will spin down any not in use. Quote Link to comment
WeeboTech Posted September 28, 2010 Share Posted September 28, 2010 Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread. Apart from the fact that the current version uses rsync instead of mv, this modification looks okay. However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories. I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command. So, it occurred to me that what is required is to suppress the sync when no files have been moved. Would something like the following have the desired effect? .... echo "mover started" # Modification to prevent drives being spun up if there's nothing in the write cache # (but allowing files to exist in hidden directories). moved=0 (cd /mnt/cache ; find -depth -print \ \( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \ -exec moved=1 \; \ -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \ \( -type d -regex '[.]/[^.].*' \ -exec moved=1 \; \ -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \ ) if [ $moved == 0 ]; then echo "nothing to move" else sync fi rm /var/run/mover.pid .... It is easier to just remove the sync. Another idea which may provide some food for thought. Add this to the top. rm -f /tmp/mover.sync change this -exec moved=1 to -exec touch /tmp/mover.sync then somewhere before the sync do an if [ -f /tmp/mover.sync ] then rm -f /tmp/mover.sync sync fi I do not believe the moved=1 will exist past the find command. Although it will assign the variable. each -exec is run in a new shell as in /bin/sh -c "moved=1". which means it would never get set in the current shell. So use of a flag file on the filesystem may work better. you can also add your spindown commands after the sync. I would just remove the sync and use a UPS. Quote Link to comment
PeterB Posted September 29, 2010 Share Posted September 29, 2010 Thanks for the advices. Although I have a lot of experience writing command scripts in many environments (predominantly DEC - VAX/VMS, RSX-11M, RT-11/TSX, RSTS/E), I've not had a lot of hands-on with UNIX systems, and am working from O'Reilly - Linux in a Nutshell ... I clearly need to do some more reading!. Simply spinning the drives down after the sync is not quite what I was after - I was hoping to eliminate wear and tear on the drives! Yes, I do have a UPS and the powerdown does include a sync command - perhaps removal of the sync from mover would be the simplest and most effective. However, I may experiment with a filesystem flag as suggested. I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved. Quote Link to comment
Joe L. Posted September 29, 2010 Share Posted September 29, 2010 I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved. A file will not be moved if it is in the process of being written. Quote Link to comment
PeterB Posted September 29, 2010 Share Posted September 29, 2010 A file will not be moved if it is in the process of being written. Yes, indeed. That is the most likely condition for a file move not to take place. Quote Link to comment
PeterB Posted October 4, 2010 Share Posted October 4, 2010 Just to report that I've implemented the file system flag file, as suggested, and that appears to have achieved the desired goal. I no longer see spindown entries in the logfile following every mover. Thanks for the help! Quote Link to comment
jonlai9 Posted April 10, 2011 Share Posted April 10, 2011 Sorry for bumping an old thread, as I just installed my cache drive today. So is there a new mover script that I can just copy onto my drive? Thanks! Quote Link to comment
prostuff1 Posted April 11, 2011 Share Posted April 11, 2011 There are a couple of modifications that are part of unMenu. Quote Link to comment
isochronous Posted July 8, 2011 Share Posted July 8, 2011 That's not especially helpful. What are those modifications? What's different from the stock mover script? Quote Link to comment
prostuff1 Posted July 8, 2011 Share Posted July 8, 2011 That's not especially helpful. What are those modifications? What's different from the stock mover script? Installing unMenu to find out is the best way. Quote Link to comment
Joe L. Posted July 8, 2011 Share Posted July 8, 2011 That's not especially helpful. What are those modifications? What's different from the stock mover script? depending on what version of unRAId you are running, it might not make any difference anyways. What version of unRAID are you running? None of the alternate mover scripts have any use any longer in the 5.0beta8 release. Unless you are doing something unusual, they do not apply to most anybody setting up a new server. One removes a "sync" command that has been removed from unRAID for several releases. It is no longer needed. A second will on older releases Exclude directories beginning with "_" from being moved from the cache drive. This is now handled entirely differently in the 5.0beta8 series onward, it too is not needed as it has no effect. The logic is completely changed. A third will modify it to log fewer lines to the syslog. This too has been fixed for a few releases in unRAID. You can find them here if you don't want to load unMENU itself: http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_conditional_sync-unmenu-package.conf http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_exclude_underscore-unmenu-package.conf http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_fix-logging-unmenu-package.conf I hope you are good at regular expressions Quote Link to comment
Eisi2005 Posted September 13, 2013 Share Posted September 13, 2013 Hi, since yesterday i have installed a cache drive. All works fine for me, tonight all files got moved to the array. But then i saw that my dir structure on the cache disk get deleted also. I search in the forum and found this thread, but i could not read the code, sorry i´m not a scripter Here is my code of the mover script: if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then exit 0 fi # If a previous invokation of this script is already running, exit if [ -f /var/run/mover.pid ]; then if ps h `cat /var/run/mover.pid` | grep mover ; then echo "mover already running" exit 0 fi fi echo $$ >/var/run/mover.pid echo "mover started" cd /mnt/cache for Share in */ ; do if [ -d "/mnt/user0/$Share" ]; then echo "moving $Share" find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \ -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete else echo "skipping $Share" fi done rm /var/run/mover.pid echo "mover finished" if [ -f /tmp/mover.sync ] then for i in `grep disk /proc/mdcmd | grep diskNumber | sed 's/.*=//'` do /root/mdcmd spindown $i done rm /tmp/mover.sync fi My dir structure on my parrity is: server/Filme/A-C server/Filme 3D/A-C server/Bilder/ So i make dirs on my cache disc called: server/Filme/A-C server/Filme 3D/A-C . . Now i want that the script dont delete these "Top-Level" dirs after moving. Could anyone help me with the code ? Thanks Eisi Quote Link to comment
itimpi Posted September 13, 2013 Share Posted September 13, 2013 This is how mover is meant to work! Why do you want the empty folders left behind? The folders that correspond to paths under user shares are automatically created as needed when you copy files to the user shares and unRAID temporarily puts them on the cache disk until mover runs. This means that in normal running the creation of folders on the cache disk and the removal of them from the cache disk once files have been moved to the parity protected array is an automated process. Quote Link to comment
Eisi2005 Posted September 13, 2013 Share Posted September 13, 2013 Hi, thanks for your answer. Maybe i dont understand the function of the mover. But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction. So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories Eisi Quote Link to comment
itimpi Posted September 13, 2013 Share Posted September 13, 2013 Hi, thanks for your answer. Maybe i dont understand the function of the mover. But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction. So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories Eisi The answer is not to put it directly to the cache drive! If you put it to the user share, then unRAID will automatically write it to a cache drive if active (and create the relevant folders as needed). Quote Link to comment
Eisi2005 Posted September 13, 2013 Share Posted September 13, 2013 Hi, thanks, life could be so easy. Eisi Quote Link to comment
Eisi2005 Posted September 13, 2013 Share Posted September 13, 2013 Hmmm, when i copy direct to the cache disk i have write speed about 60 mb/s If i write to the cache over the array i have write speed about 40mb/s. Is there anything i can do to become the same speed ? Eisi Quote Link to comment
Harpz Posted September 13, 2013 Share Posted September 13, 2013 When your copying to the array your copying to the disk and the parity info is getting written also to the parity drive so your getting 2 writes. 40mb/s to the array is normal, I get 32-48mb/s to the array and 70-80 to the cache. Quote Link to comment
BobPhoenix Posted September 13, 2013 Share Posted September 13, 2013 Sounds like your user share does not have your cache disk enabled for it. You should turn it on for each share that you want writes to go to the cache drive rather than straight to the array. Quote Link to comment
Eisi2005 Posted September 13, 2013 Share Posted September 13, 2013 Thanks for your answers Cache is enabled for the share. Without cache i have 25 mb/s write speed. Eisi Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.