bblue

Mover modification

Recommended Posts

I run mover every hour on my machine, and more often than not it spins up all the drives each time.  It's caused by the 'sync' command.

 

There doesn't seem to be any reason to spin the drives up if there's nothing for mover to do, so I made a change in mover to test the cache drive before anything else.  If cache is empty, skip over the finds and sync operations.  If cache is non-empty, everything proceeds as usual.

 

Here's just the code part of my mover which also leaves the top level share hierarchies instead of removing them  (-mindepth 2 in the second find if you don't want that).  Some might find this useful.

 

# Only run script if cache disk enabled
if [ ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid

# print without a newline
echo -n mover started

sz=$(du -s /mnt/cache | cut -f1)
if [ $sz == 0 ]
then
echo ":no work"
else
echo
(cd /mnt/cache ; \
   find . \
       \( -type d  -regex '[.]/[^.].*'                             -exec mkdir -p /mnt/user0/{} \; \) , \
       \( -type f  -regex '[.]/[^.].*/.*'  ! -exec fuser -s {} \;  -exec echo moving {} \;  -exec mv -v {} /mnt/user0/{} \; \) ; \
   find . -mindepth 2 -type d  -regex '[.]/[^.].*'     ! -exec fuser -s {} \;  -empty -delete \ )
sync

fi

echo "mover finished"

rm /var/run/mover.pid

 

--Bill

Share this post


Link to post
Share on other sites

Hi Bill,

 

Newbie here...Would you mind explaining exactly how to add this to my unraid box please? I'm not sure how & where to place this code?

 

Thanks!

Share this post


Link to post
Share on other sites

Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread.  Apart from the fact that the current version uses rsync instead of mv, this modification looks okay.  However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories.

 

I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command.  So, it occurred to me that what is required is to suppress the sync when no files have been moved.

 

Would something like the following have the desired effect?

 

....
echo "mover started"

# Modification to prevent drives being spun up if there's nothing in the write cache
# (but allowing files to exist in hidden directories).

moved=0

(cd /mnt/cache ; find -depth -print \
\( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \
\( -type d -regex '[.]/[^.].*' \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \
)

if [ $moved == 0 ]; then
echo "nothing to move"
else
sync
fi


rm /var/run/mover.pid
....

Share this post


Link to post
Share on other sites

No it would not work.    there is no command "moved=1" to execute AND even if there was a shell involved it is a child shell and any variables set in it are not visible in the parent shell's environment AND the way you coded it would have been exec'ed if any file or directory exists in the cache drive, not if any are moved.

 

Far easier to just remove the "sync" command. (and get a UPS so an un-expected shutdown will not occur)

or

add these commands after it  (one line for each of your drives).

 

sync

/root/mdcmd spindown 1

/root/mdcmd spindown 2

/root/mdcmd spindown 3

/root/mdcmd spindown 4

 

The spindown commands will not affect a drive currently being used but will spin down any not in use.

 

 

 

 

Share this post


Link to post
Share on other sites

Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread.  Apart from the fact that the current version uses rsync instead of mv, this modification looks okay.  However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories.

 

I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command.  So, it occurred to me that what is required is to suppress the sync when no files have been moved.

 

Would something like the following have the desired effect?

 

....
echo "mover started"

# Modification to prevent drives being spun up if there's nothing in the write cache
# (but allowing files to exist in hidden directories).

moved=0

(cd /mnt/cache ; find -depth -print \
\( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \
\( -type d -regex '[.]/[^.].*' \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \
)

if [ $moved == 0 ]; then
echo "nothing to move"
else
sync
fi


rm /var/run/mover.pid
....

 

It is easier to just remove the sync.

 

Another idea which may provide some food for thought.

 

Add this to the top.

rm -f /tmp/mover.sync

 

change this

-exec moved=1

to

-exec touch /tmp/mover.sync

 

then somewhere before the sync

do an

if [ -f /tmp/mover.sync ]

  then rm -f /tmp/mover.sync

        sync

fi

 

I do not believe the moved=1 will exist past the find command.

Although it will assign the variable. each -exec is run in a new shell as in /bin/sh -c "moved=1".

which means it would never get set in the current shell.

So use of a flag file on the filesystem may work better.

 

you can also add your spindown commands after the sync.

I would just remove the sync and use a UPS.

Share this post


Link to post
Share on other sites

Thanks for the advices.  Although I have a lot of experience writing command scripts in many environments (predominantly DEC - VAX/VMS, RSX-11M, RT-11/TSX, RSTS/E), I've not had a lot of hands-on with UNIX systems, and am working from O'Reilly - Linux in a Nutshell ... I clearly need to do some more reading!.

 

Simply spinning the drives down after the sync is not quite what I was after - I was hoping to eliminate wear and tear on the drives!  Yes, I do have a UPS and the powerdown does include a sync command - perhaps removal of the sync from mover would be the simplest and most effective.  However, I may experiment with a filesystem flag as suggested.

 

I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved.

Share this post


Link to post
Share on other sites

I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved.

A file will not be moved if it is in the process of being written.

Share this post


Link to post
Share on other sites
A file will not be moved if it is in the process of being written.

 

Yes, indeed.  That is the most likely condition for a file move not to take place.

Share this post


Link to post
Share on other sites

Just to report that I've implemented the file system flag file, as suggested, and that appears to have achieved the desired goal.  I no longer see spindown entries in the logfile following every mover.  Thanks for the help!

Share this post


Link to post
Share on other sites

Sorry for bumping an old thread, as I just installed my cache drive today.

 

So is there a new mover script that I can just copy onto my drive? Thanks!

Share this post


Link to post
Share on other sites

That's not especially helpful.  What are those modifications?  What's different from the stock mover script?

 

Installing unMenu to find out is the best way.

Share this post


Link to post
Share on other sites

That's not especially helpful.  What are those modifications?  What's different from the stock mover script?

depending on what version of unRAId you are running, it might not make any difference anyways.

 

What version of unRAID are you running?  None of the alternate mover scripts have any use any longer in the 5.0beta8 release.  Unless you are doing something unusual, they do not apply to most anybody setting up a new server.   

 

One removes a "sync" command that has been removed from unRAID for several releases.  It is no longer needed.

A second will on older releases Exclude directories beginning with "_" from being moved from the cache drive.  This is now handled entirely differently in the 5.0beta8 series onward, it too is not needed as it has no effect.  The logic is completely changed.

A third will modify it to log fewer lines to the syslog.  This too has been fixed for a few releases in unRAID.

You can find them here if you don't want to load unMENU itself:

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_conditional_sync-unmenu-package.conf

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_exclude_underscore-unmenu-package.conf

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_fix-logging-unmenu-package.conf

I hope you are good at regular expressions  ;D

 

 

Share this post


Link to post
Share on other sites

Hi,

 

since yesterday i have installed a cache drive. All works fine for me, tonight all files got moved to the array. But then i saw that my dir structure on the cache disk get deleted also. I search in the forum and found this thread, but i could not read the code, sorry i´m not a scripter :(

 

Here is my code of the mover script:

 

if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

rm /var/run/mover.pid
echo "mover finished"
if [ -f /tmp/mover.sync ]
then
  for i in `grep disk /proc/mdcmd | grep diskNumber | sed 's/.*=//'`
  do
    /root/mdcmd spindown $i
  done
  rm /tmp/mover.sync
fi

 

 

My dir structure on my parrity is:

 

server/Filme/A-C

server/Filme 3D/A-C

server/Bilder/

 

So i make dirs on my cache disc called:

 

server/Filme/A-C

server/Filme 3D/A-C

.

.

 

Now i want that the script dont delete these "Top-Level" dirs after moving.

 

Could anyone help me with the code ?

 

Thanks

Eisi

Share this post


Link to post
Share on other sites

This is how mover is meant to work!    Why do you want the empty folders left behind?

 

The folders that correspond to paths under user shares are automatically created as needed when you copy files to the user shares and unRAID temporarily puts them on the cache disk until mover runs.  This means that in normal running the creation of folders on the cache disk and the removal of them from the cache disk once files have been moved to the parity protected array is an automated process.

Share this post


Link to post
Share on other sites

Hi,

 

thanks for your answer.

 

Maybe i dont understand the function of the mover.

 

But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction.

 

So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories

 

Eisi

 

Share this post


Link to post
Share on other sites

Hi,

 

thanks for your answer.

 

Maybe i dont understand the function of the mover.

 

But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction.

 

So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories

 

Eisi

The answer is not to put it directly to the cache drive!    If you put it to the user share, then unRAID will automatically write it to a cache drive if active (and create the relevant folders as needed).

Share this post


Link to post
Share on other sites

Hmmm,

 

when i copy direct to the cache disk i have write speed about 60 mb/s

 

If i write to the cache over the array i have write speed about 40mb/s. Is there anything i can do to become the same speed ?

 

Eisi

 

Share this post


Link to post
Share on other sites

When your copying to the array your copying to the disk and the parity info is getting written also to the parity drive so your getting 2 writes.

 

40mb/s to the array is normal, I get 32-48mb/s to the array and 70-80 to the cache.

Share this post


Link to post
Share on other sites

Sounds like your user share does not have your cache disk enabled for it.  You should turn it on for each share that you want writes to go to the cache drive rather than straight to the array.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.