diskmv -- A set of utilities to move files between disks


Recommended Posts

No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example:

[color=blue]consld8 -f /mnt/user/Movies disk8[/color]

will move the Movies user share from all other disks to disk8 (if there is enough space).

So my question is what if there isn't enough space on disk8?

 

I have a couple of tasks I would like to perform.

[*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.)

[*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method.

Is there a way to achieve this using your tools? Or is there a better way?

 

Link to comment

No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example:

[color=blue]consld8 -f /mnt/user/Movies disk8[/color]

will move the Movies user share from all other disks to disk8 (if there is enough space).

So my question is what if there isn't enough space on disk8?

 

I have a couple of tasks I would like to perform.

[*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.)

[*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method.

Is there a way to achieve this using your tools? Or is there a better way?

 

Sorry, no, my tools completely bypass the allocation method and you have to specify a single destination disk.

Link to comment

@huladaddy,

 

The excel sheet I put together may help you.  Basically, it read your drives, builds a grid showing drives where the various directories are located.  You can elect to "move"  a single directory from say disk2 to disk1 or you can move  the directories from say disk2, disk3 and disk7 to disk1.  In this case you use the stacked rsync command.

 

Based on the selections you make, it builds a rsync command you copy and paste into a telnet session you you open to your Unraid server.  Every time you run the data import scripts, it resets the rsync command to "dryrun".  You have to change it in the drop-down.

 

You really want to read through the entire thread, but at least these two links.

http://lime-technology.com/forum/index.php?topic=33689.msg310759#msg310759

http://lime-technology.com/forum/index.php?topic=33689.msg415201#msg415201

 

Also, unraid allocations are not involved, you make the decision on the file locations and how to get them there.

 

Link to comment
  • 5 months later...

I ran consld8 to consolidate my music directory, and now SyncToy thinks heaps of files have changed that shouldn't have. Can you confirm that the time modified stamp doesn't change when using consld8?

I've posted my problem - in excruciating detail - here: https://lime-technology.com/forum/index.php?topic=50458.0

Your help is appreciated.

 

The file modified time stamps do not change, but the directory time stamps do change.

 

The consld8 script uses diskmv which in turn uses rsync to copy files with the -t option flag. The -t option flag preserves the modification times on all files, however, when files are copied to the destination disk and removed from the source disk, the parent directory's modification time is updated by the filesystem.

Link to comment
  • 2 weeks later...

new user of unRAID coming from Windows...

 

Would it be possible to explain to me how to get the script and "install it"?

 

Usually, I just try till it works, but this could be a bit destructive considering the functionality.

 

Thanks in advance.

Might be simpler for you to just use the unBalance plugin which is based on this.
Link to comment

Might be simpler for you to just use the unBalance plugin which is based on this.

 

Great suggestion.

 

Got it installed in a flash.

 

Unfortunately, it does not seem to be able to move a folder from Unassigned Devices :-( And that is what I am trying to do.

 

Currently trying to move my old Windows (NTFS) data onto the array. I figured it would be quicker that doing a move from a Windows VM.

Link to comment

Might be simpler for you to just use the unBalance plugin which is based on this.

 

Great suggestion.

 

Got it installed in a flash.

 

Unfortunately, it does not seem to be able to move a folder from Unassigned Devices :-( And that is what I am trying to do.

 

Currently trying to move my old Windows (NTFS) data onto the array. I figured it would be quicker that doing a move from a Windows VM.

Google Midnight Commander (mc). It is built-in to unRAID and it gives you a sort of text-based, two-panel, file explorer that is easy to figure out and use. Just navigate to /mnt and look around, you will be able to see your disks, including the unassigned device. Then you can copy/move from one panel to the other with some function keys.
Link to comment

new user of unRAID coming from Windows...

 

Would it be possible to explain to me how to get the script and "install it"?

 

Usually, I just try till it works, but this could be a bit destructive considering the functionality.

 

Thanks in advance.

Use Putty to remote into your server

 

mkdir /boot/scripts
wget https://raw.githubusercontent.com/trinapicot/unraid-diskmv/master/diskmv -O /boot/scripts/diskmv
wget https://raw.githubusercontent.com/trinapicot/unraid-diskmv/master/consld8 -O /boot/scripts/consld8
echo "ln -s /boot/scripts/diskmv /usr/sbin" >> /boot/config/go
echo "ln -s /boot/scripts/consld8 /usr/sbin" >> /boot/config/go

That'll:

  • Create a directory to hold the scripts that survives reboots.
  • Download the scripts to that directory.
  • Add lines to your startup script to create symlinks (shortcuts, sort of) to those permanent files in a non-permanent space that makes them more usable.

After doing that, if you don't want to reboot, you can just run the below to create the symlinks right now. They'll get recreated every boot by the go script, so you only have to do this part the first time.

ln -s /boot/scripts/diskmv /usr/sbin
ln -s /boot/scripts/consld8 /usr/sbin

 

 

Link to comment
  • 4 months later...

I've created a couple of bash scripts to facilitate moving files between disks.  As these utilities are touching user data, I'd like to have some experienced users review and/or test them.  If there is interest and no major problems, I will look into making them easier to use (first a plugin for command line usage, then maybe gui)

 

  • 'diskmv'

    will move a user share directory from one disk to another. It uses a find/rsync command similar to what is found in the standard mover script. It is suitable for merging a user share directory onto a disk that already contains that directory.  Files that have duplicate file names  on the destination disk will not moved. By default, 'diskmv' will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced.
     

  • 'consld8'

    can consolidate a user share directory from multiple disks onto one disk. If a destination disk is not specified, it will pick the best disk based on max usage and available space.  By default, consld8 will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced.

    'diskmv'

    is required to actually move files.
     

Example usage:

 

diskmv "/mnt/user/video/tv/Pushing Daisies" disk4 disk1
    
consld8 /mnt/user/video/tv/Wonderfalls disk3

 

To get get a help message:

 

diskmv -h

 

I am tagging releases for code that I trust and have used on my real data. Releases are here: https://github.com/trinapicot/unraid-diskmv/releases

 

The master branch has code that I have tested on sample data, but I have not necessarily used it on my real data.

 

These utilities move files around on an unRAID server. I have done my best to prevent any data loss, but there is always a chance something can go wrong. Use at your own risk and please ensure you have everything backed up.

 

Note for those wishing to use diskmv in the process of changing filesystems:

 

The diskmv script will not verify that copied files are the same as the source files.

 

With the default options, diskmv will rsync each file and if the rsync is successful, that file is then deleted before moving to the next file. There is no opportunity to compare the source and destination files. The script is relying on the OS, rsync and whatever else is involved to correctly read data from one disk and write it to another.

 

If you still want to use diskmv to move all the files and directories from one disk to another, you can use this syntax:

diskmv -f "" disk1 disk2

Or to keep source files (copy instead of move):

diskmv -f -k "" disk1 disk2

 

Thank you so much for this work !!!

 

I think this script should be pinned on forum for other guys to use - I took me a while when I found this great tool.

 

I also think it would be to have it as WebUI plugin, if someone wants to do it :) I would but I never done any plugin :)

Link to comment

BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM.

 

That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight.

 

How much RAM your machine have ?

 

This happens only when some big files are moving..

 

I think it has to do with low RAM...4GB ? what ya think ?

Link to comment

BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM.

 

That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight.

 

How much RAM your machine have ?

 

This happens only when some big files are moving..

 

I think it has to do with low RAM...4GB ? what ya think ?

 

Now I have 8 but at the time I used it I had 4GB.

Link to comment

BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM.

 

That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight.

 

How much RAM your machine have ?

 

This happens only when some big files are moving..

 

I think it has to do with low RAM...4GB ? what ya think ?

 

Now I have 8 but at the time I used it I had 4GB.

 

I had one reply in different post where guy told me his 6.x system was unstable with 4gb of RAM, and after he upgraded to 8GB everything is fine now. I have been using this HW for 4 years stable with UNRAID.

Link to comment
  • 6 months later...

I just wanted to say this script is just... amazing. I have been working on cleaning up nearly 20TB of movies and tv shows for several years now split incorrectly over time due to this or that, settings errors on splits, etc.

 

I have been having success over that time with many different methods and a hit and miss effort, but yours wrapped it all up in one fell swoop.

 

Thank you.

Link to comment
  • 5 months later...

Jumping on the THANK YOU bandwagon!

 

I was using unBalance, but its one big drawback is resolved with @Benni-chan's excellent commands here. I used that to straighten up my Movie share in about 2 hours. I'd spent a couple of hours at a time using unBalance one movie at a time because uB won't automatically pick a disk to move everything to.

 

I did note that it left a few files behind, but when I went back through the log ( | tee movielog.txt to the rescue!) to investigate, I discovered that it was skipping duplicate files. Somehow, I'd gotten different versions of the <movie>.nfo file. I deleted about a dozen of them and all is good!

 

Off to clean up my TV shows next.

 

 

On 11/22/2015 at 1:10 PM, Archivist said:

My moving file/directory is very large and it's still testing after 15 minutes with no way that I'm aware to stop the process.

 

Also, Ctrl-C is a great way to stop a shell script without killing the telnet session and losing all your command history. /late to the party...

Link to comment

Once again, @Benni-chan has come to the rescue and gotten my TV share about 98% cleaned up. However, what I'm finding is this:

image.png.e766e66e527c2d984e01efd691a70598.png

 

A season has been nicely consolidated to a single disk, but the various show-level metadata is still scattered about. So I ran a combo script that, when I ran it in test mode seemed to have taken care of things nicely:

 


find "/mnt/user/Media/TV" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 consld8 -f | tee TVlog-season.txt

However, when I started running it in real mover mode, it started to move season folders.

 

I read the find man page and changed -type d to -type f in the hopes of finding only the files at the <show> level. That worked, but... it ended up passing file names to consld8, instead of directory names.

 

It looks like the issue I'm running into is that consld8 will, by default, move all files and sub-directories to the specified destination disk, and I want it to only move files. Is my understanding of this issue correct? If so, is there a possibility of adding a switch to consld8 that would have it move files only and leave the sub-directories where they are?

 

Still really a fan of these two tools! Even with the OCD cleanup of the <show>/<files> I'm certain that this will end up saving me many, many, many hours of manual work vs what I was doing before. I've accomplished more clean up by letting two command lines run on their own than I had in a week of sessions with mc or even unBalance!

Link to comment
  • 3 weeks later...

Is there a way to expand on this command to find a group of folders starting with the letter M...something like this for example:

find "/mnt/user/Media/Movies" -name "M*" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 consld8 -f disk1

Well obviously that didn't work since consld8 requires a share specified!

 

Right now having to do one folder at a time sucks since i'm trying to move stuff to Disk1 or Disk2 for organizational purposes.

So I don't want to do a full disk4 to disk1.  Not enough room even if I did! :)

 

Thanks

Urlryn

Edited by Urlryn
Link to comment

Figured it out....

Did a couple test runs for some of my anime stuff and everything transferred nicely!

--Finds any anime folder starting with Kidou and merges them to disk1
find "/mnt/user/media/anime" -name "Kidou*" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1

--Finds any folder starting with A or B and merges them to disk1. This will omit any folder whose name is only a single letter A or b
find "/mnt/user/media/anime" \( -name '[ab]*' \) -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1

--If you also want to include those above, you will need to specify another pattern: the name must match either [ab]* or [ab]
find "/mnt/user/media/anime" \( -name '[ab]*' -o -name '[ab]' \) -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1

*Note...the commands are set with the -t switch.  So will need to change this to -f when you are ready to do the actual move/merge.

Please test before changing it! :)

 

Urlryn

Edited by Urlryn
Link to comment
  • 4 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.