[Plugin] unbalanced


Recommended Posts

v3.2.0 is out !

 

----------------------

2017-07-07 - 3.2.0

VALIDATE operation is now implemented !!

 

Available after a COPY operation, it performs a checksum copy, thus validating the integrity
of the data.

 

It works by replacing the rsync flags from

 

-avPRX (default flags)

 

to

 

-rcPRX

 

If you set custom rsync flags, "-a" MUST still be the first one.

 

Also note that this operation takes a long time to complete.
--------------

 

I'd like to stress how long this operation will take. I will take a LOT of time :)

 

I'll now tackle the GATHER operation.

 

I would like to thank all the people that has contributed to the beer/coffee/soft drinks fund :)

 

I have enjoyed reading the notes on how unBALANCE helped you.

 

Thanks a lot !

 

  • Upvote 2
Link to comment

Great tool!

 

I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...)

However when I looked today, it never deleted the source files.  They all got copied, but never deleted...

 

I basically used all the stock settings..

 

Jim

Link to comment

Here is the relevant stuff from the log file..

I: 2017/07/10 20:54:50 core.go:596: _calc:End:srcDisk(/mnt/disk4)
I: 2017/07/10 20:55:06 core.go:782: Running transfer operation ...
I: 2017/07/10 20:55:06 core.go:814: Command Started: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/"
I: 2017/07/10 23:22:17 core.go:143: Sending config
W: 2017/07/11 04:58:25 core.go:886: Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer
due to error)
I: 2017/07/11 04:58:25 core.go:991:
unBALANCE - TRANSFER operation INTERRUPTED


Started: Jul 10, 2017 20:55:06
Ended: Jul 11, 2017 04:58:25

Elapsed: 8h3m18.906s

Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer due to error)

Transferred 962.2G at ~ 33.98 MB/s

And I don't see anything in the syslog around 4:58...   What error?

 

hmm...

Link to comment

Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy).

 

Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all :o

 

But it's generally associated with permission errors or date-time issues with some of your files/folders.

 

Did you get any permission issues during the calculate phase ?

 

 

Link to comment
7 minutes ago, jbrodriguez said:

Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy).

 

Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all :o

 

But it's generally associated with permission errors or date-time issues with some of your files/folders.

 

Did you get any permission issues during the calculate phase ?

 

 

I don't recall if I had any permission issues.  I think I might have.

 

And I just installed unbalance several days ago!  I didn't think to check for an update so soon! :-)

Link to comment
23 hours ago, jbuszkie said:

Great tool!

 

I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...)

However when I looked today, it never deleted the source files.  They all got copied, but never deleted...

 

I basically used all the stock settings..

 

Jim

You need to fix the permission issues (run "newperms path_to_folder") - on the source AND on the target directory!

Then run exactly the same move command again that you used yesterday.

The sources should get removed then.

Edited by Fireball3
Link to comment

Just after some direction here, and what im looking for might be the gather function but im not sure what exactly that is planned to provide.

 

Basically i have just moved all my data using unbalance in order to move from reinfs to xfs, now i have one completely empty disk.

 

I would like to distribute the data on the other disks to this one, so that i end up with all drives having relatively the same amount of free space.

 

Whats the besr way to accomplish this?

Link to comment

I can tell you what GATHER will do with an example.

 

Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4.

 

GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space.

 

What it won't do is  fill an empty disk to "rebalance" the space among all available disks.

 

Hope it makes it a bit clearer.

 

 

Link to comment
16 minutes ago, jbrodriguez said:

I can tell you what GATHER will do with an example.

 

Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4.

 

GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space.

 

What it won't do is  fill an empty disk to "rebalance" the space among all available disks.

 

Hope it makes it a bit clearer.

 

 

Definitely clears that up, obviously a useful feature.

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

 

Any suggestions to speed up manually doing this?

Link to comment
54 minutes ago, kingy444 said:

Definitely clears that up, obviously a useful feature.

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

 

Any suggestions to speed up manually doing this?

 

Not the best way... but could you move it all to the cache disk and then let the mover "rebalance" it with the share set to the correct setting (I forget which one it is..)?

 

Link to comment
56 minutes ago, kingy444 said:

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

Depending on your use, rebalancing may be counter productive. As you fill a disk, the speeds get lower and lower. Many people would rather fill a disk to capacity and leave it alone, so new writes and fresh data are written to the newest and fastest disks. Older less used data is left on the slower full disks. The more you erase and move data around on a disk, the more fragmented it gets, and modern filesystems don't really have a good method of defragmenting. They are designed in such a way that fragmentation doesn't have as much of an effect as in the past, but it's still noticeable on a badly thrashed disk.

 

The default fill method is high water for a reason, it's the best compromise between spreading data all over the disks evenly and filling each disk fully before moving to the next one.

 

tl;dr Rebalancing may hurt performance.

Link to comment
  • 2 weeks later...
  • 2 weeks later...

I've been using unbalance to move data off two disks to convert them to btrfs from rfs. My process was to move the data after calculating (no issues in calculation), and then stop the array and format the old drive. I noticed that it had copied the data not moved it after I did the second drive. 

 

The problem is, some of my shares are not showing the data that was moved... They are showing missing folders and files. I checked the share with my important folders and it's not showing all data, in crashplan it shows files are on the disk when you navigate through /mnt/user/share (pretty sure that's what it's saying, not showing as deleted from a backup) but the share isn't showing them.

 

I'm running a parity sync now. Is there something that needs to be done for unraid to recognize the files moved disks? It seems like that should have been happening realtime with unbalance but it looks like the share is looking for data on disk 1 when it was moved to disk 2. All my share information has "all disks" and no excluded disks.

Edited by ptirmal
Link to comment
41 minutes ago, trurl said:

How about your Global Share Settings?

Same, set to default, nothing is excluded.

 

Just want to reiterate that I'm pretty sure the data has been copied correctly, because the drive usage percentage increased respective to the data I moved (which was copied) prior to formatting (which then went back to where it was before any file moving).

 

Edit: So, looking at this thread: 

 

 

I think this is a samba issue. I'll have to check when I get home to be certain. I've never had to fix permissions before, is there a way I can go about doing that?

Edited by ptirmal
Link to comment
2 hours ago, ptirmal said:

it shows files are on the disk when you navigate through /mnt/user/share


What about taking a look at the disk itself ? 

$ ls -al /mnt/disk2/share

I understand disk2 is where data is supposed to be now.

 

You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want).

Link to comment
1 hour ago, graywolf said:

BTW. Looking forward to the Gather function. Been meaning to consolidate some shares to specific drives but have not done it yet due to time constraints. How close are you for it?

 

It should be ready sometime next week, if there are no issues in the mean time :)

Link to comment
33 minutes ago, jbrodriguez said:


What about taking a look at the disk itself ? 


$ ls -al /mnt/disk2/share

I understand disk2 is where data is supposed to be now.

 

You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want).

Thanks, I'll check when I get back.

1 hour ago, jonathanm said:

Why?

At the time I thought maybe the parity disk wasn't updating during the transfer. 

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.