Mover modification


bblue

Recommended Posts

I run mover every hour on my machine, and more often than not it spins up all the drives each time.  It's caused by the 'sync' command.

 

There doesn't seem to be any reason to spin the drives up if there's nothing for mover to do, so I made a change in mover to test the cache drive before anything else.  If cache is empty, skip over the finds and sync operations.  If cache is non-empty, everything proceeds as usual.

 

Here's just the code part of my mover which also leaves the top level share hierarchies instead of removing them  (-mindepth 2 in the second find if you don't want that).  Some might find this useful.

 

# Only run script if cache disk enabled
if [ ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid

# print without a newline
echo -n mover started

sz=$(du -s /mnt/cache | cut -f1)
if [ $sz == 0 ]
then
echo ":no work"
else
echo
(cd /mnt/cache ; \
   find . \
       \( -type d  -regex '[.]/[^.].*'                             -exec mkdir -p /mnt/user0/{} \; \) , \
       \( -type f  -regex '[.]/[^.].*/.*'  ! -exec fuser -s {} \;  -exec echo moving {} \;  -exec mv -v {} /mnt/user0/{} \; \) ; \
   find . -mindepth 2 -type d  -regex '[.]/[^.].*'     ! -exec fuser -s {} \;  -empty -delete \ )
sync

fi

echo "mover finished"

rm /var/run/mover.pid

 

--Bill

Link to comment
  • 1 month later...
  • 10 months later...

Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread.  Apart from the fact that the current version uses rsync instead of mv, this modification looks okay.  However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories.

 

I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command.  So, it occurred to me that what is required is to suppress the sync when no files have been moved.

 

Would something like the following have the desired effect?

 

....
echo "mover started"

# Modification to prevent drives being spun up if there's nothing in the write cache
# (but allowing files to exist in hidden directories).

moved=0

(cd /mnt/cache ; find -depth -print \
\( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \
\( -type d -regex '[.]/[^.].*' \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \
)

if [ $moved == 0 ]; then
echo "nothing to move"
else
sync
fi


rm /var/run/mover.pid
....

Link to comment

No it would not work.    there is no command "moved=1" to execute AND even if there was a shell involved it is a child shell and any variables set in it are not visible in the parent shell's environment AND the way you coded it would have been exec'ed if any file or directory exists in the cache drive, not if any are moved.

 

Far easier to just remove the "sync" command. (and get a UPS so an un-expected shutdown will not occur)

or

add these commands after it  (one line for each of your drives).

 

sync

/root/mdcmd spindown 1

/root/mdcmd spindown 2

/root/mdcmd spindown 3

/root/mdcmd spindown 4

 

The spindown commands will not affect a drive currently being used but will spin down any not in use.

 

 

 

 

Link to comment

Having discovered that my drives get spun up every time that mover gets invoked I searched the forum and came across this thread.  Apart from the fact that the current version uses rsync instead of mv, this modification looks okay.  However, I then realised that if you make use of the ability to keep files in hidden directories, this modification will not be effective since du will include the sizes of the files in the hidden directories.

 

I believe that the only reason that the drives get spun up when no files are moved is the use of the sync command.  So, it occurred to me that what is required is to suppress the sync when no files have been moved.

 

Would something like the following have the desired effect?

 

....
echo "mover started"

# Modification to prevent drives being spun up if there's nothing in the write cache
# (but allowing files to exist in hidden directories).

moved=0

(cd /mnt/cache ; find -depth -print \
\( -type f -regex '[.]/[^.].*/.*' ! -exec fuser -s {} \; \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} /mnt/user0/ \; \) -o \
\( -type d -regex '[.]/[^.].*' \
    -exec moved=1 \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -empty -delete \) \
)

if [ $moved == 0 ]; then
echo "nothing to move"
else
sync
fi


rm /var/run/mover.pid
....

 

It is easier to just remove the sync.

 

Another idea which may provide some food for thought.

 

Add this to the top.

rm -f /tmp/mover.sync

 

change this

-exec moved=1

to

-exec touch /tmp/mover.sync

 

then somewhere before the sync

do an

if [ -f /tmp/mover.sync ]

  then rm -f /tmp/mover.sync

        sync

fi

 

I do not believe the moved=1 will exist past the find command.

Although it will assign the variable. each -exec is run in a new shell as in /bin/sh -c "moved=1".

which means it would never get set in the current shell.

So use of a flag file on the filesystem may work better.

 

you can also add your spindown commands after the sync.

I would just remove the sync and use a UPS.

Link to comment

Thanks for the advices.  Although I have a lot of experience writing command scripts in many environments (predominantly DEC - VAX/VMS, RSX-11M, RT-11/TSX, RSTS/E), I've not had a lot of hands-on with UNIX systems, and am working from O'Reilly - Linux in a Nutshell ... I clearly need to do some more reading!.

 

Simply spinning the drives down after the sync is not quite what I was after - I was hoping to eliminate wear and tear on the drives!  Yes, I do have a UPS and the powerdown does include a sync command - perhaps removal of the sync from mover would be the simplest and most effective.  However, I may experiment with a filesystem flag as suggested.

 

I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved.

Link to comment

I realise that rsync makes it's own decision about whether to move files but, in the context of the mover script, I'm guessing the fact that the find condition is satisfied makes it virtually certain that files will be moved.

A file will not be moved if it is in the process of being written.
Link to comment
  • 6 months later...
  • 2 months later...

That's not especially helpful.  What are those modifications?  What's different from the stock mover script?

depending on what version of unRAId you are running, it might not make any difference anyways.

 

What version of unRAID are you running?  None of the alternate mover scripts have any use any longer in the 5.0beta8 release.  Unless you are doing something unusual, they do not apply to most anybody setting up a new server.   

 

One removes a "sync" command that has been removed from unRAID for several releases.  It is no longer needed.

A second will on older releases Exclude directories beginning with "_" from being moved from the cache drive.  This is now handled entirely differently in the 5.0beta8 series onward, it too is not needed as it has no effect.  The logic is completely changed.

A third will modify it to log fewer lines to the syslog.  This too has been fixed for a few releases in unRAID.

You can find them here if you don't want to load unMENU itself:

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_conditional_sync-unmenu-package.conf

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_exclude_underscore-unmenu-package.conf

http://code.google.com/p/unraid-unmenu/source/browse/trunk/mover_fix-logging-unmenu-package.conf

I hope you are good at regular expressions  ;D

 

 

Link to comment
  • 2 years later...

Hi,

 

since yesterday i have installed a cache drive. All works fine for me, tonight all files got moved to the array. But then i saw that my dir structure on the cache disk get deleted also. I search in the forum and found this thread, but i could not read the code, sorry i´m not a scripter :(

 

Here is my code of the mover script:

 

if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

rm /var/run/mover.pid
echo "mover finished"
if [ -f /tmp/mover.sync ]
then
  for i in `grep disk /proc/mdcmd | grep diskNumber | sed 's/.*=//'`
  do
    /root/mdcmd spindown $i
  done
  rm /tmp/mover.sync
fi

 

 

My dir structure on my parrity is:

 

server/Filme/A-C

server/Filme 3D/A-C

server/Bilder/

 

So i make dirs on my cache disc called:

 

server/Filme/A-C

server/Filme 3D/A-C

.

.

 

Now i want that the script dont delete these "Top-Level" dirs after moving.

 

Could anyone help me with the code ?

 

Thanks

Eisi

Link to comment

This is how mover is meant to work!    Why do you want the empty folders left behind?

 

The folders that correspond to paths under user shares are automatically created as needed when you copy files to the user shares and unRAID temporarily puts them on the cache disk until mover runs.  This means that in normal running the creation of folders on the cache disk and the removal of them from the cache disk once files have been moved to the parity protected array is an automated process.

Link to comment

Hi,

 

thanks for your answer.

 

Maybe i dont understand the function of the mover.

 

But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction.

 

So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories

 

Eisi

 

Link to comment

Hi,

 

thanks for your answer.

 

Maybe i dont understand the function of the mover.

 

But if i put a new movie on the cache drive i must create the folder structe i have on my share to see the movie just in time in my "secure" raid. If i dont create the file structure i didnt see the movie in my protected server and mover could not move the movie to the right direction.

 

So i must every day create new the dirs /server/Filme/A-C as example. For my personaly use i found it better when mover only move the files in these directories

 

Eisi

The answer is not to put it directly to the cache drive!    If you put it to the user share, then unRAID will automatically write it to a cache drive if active (and create the relevant folders as needed).

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.