[Plugin] unbalanced

[trurl]

2 hours ago, ptirmal said:

At the time I thought maybe the parity disk wasn't updating during the transfer. 

You can forget about that ever happening unless parity is disabled.

[ptirmal]

So I'm not quite sure what's going on. I ran docker safe permissions, nothing changed. Crashplan shows missing files same as when I access via samba or console so I'm guessing they didn't move before the format?  . I couldn't see anything that looks like it didn't copy, and unraid is showing the capacity was used.

 

However, it appears Crashplan is not showing the data is there, but it's telling me there isn't enough space to restore (attached) which makes it seem permission related. 

 

Backup is the share with most of the stuff that's missing.

 

Thoughts?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Edited August 8, 2017 by ptirmal

[jbrodriguez]

Ok, I checked the logs

 

There were two separate errors transferring files

 

The first one

I: 2017/08/06 07:27:43 core.go:740: Running Move operation ...
I: 2017/08/06 07:27:43 core.go:857: Command Started: rsync -avPRX "Media/movies" "/mnt/disk5/"
W: 2017/08/06 12:39:00 server.go:259: closing socket (&{id: ws:0xc4200e4240 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF
I: 2017/08/06 16:15:34 core.go:939: Command Finished
I: 2017/08/06 16:15:34 core.go:945: Current progress: 64.79% done ~ 4h46m52s left (39.78 MB/s)
I: 2017/08/06 16:15:34 core.go:957: Removing: rm -rf "/mnt/disk2/Media/movies"
W: 2017/08/06 16:15:34 shell.go:53: transferProgress:: rm: cannot remove '/mnt/disk2/Media/movies/Cake (2014)/Cake (2014) 720p.mkv': Permission denied
W: 2017/08/06 16:15:35 shell.go:101: transferProgress:: waitError: signal: broken pipe
W: 2017/08/06 16:15:35 core.go:968: Unable to remove source folder (/mnt/disk2/Media/movies): signal: broken pipe

It fully copied "Media/movies" to disk5, but failed to delete the source folder (/mnt/disk2/Media/movies), due to a permission issue.

In this case, the "Media/movies" data is in disk 5. Run 

$ ls -al /mnt/disk5/Media/movies

to list the files.

 

After you've double checked that, you can delete the files from disk2 (rm -rf "/mnt/disk2/Media/movies"), so you can further debug the permission issue.

 

The second error

I: 2017/08/06 16:15:35 core.go:857: Command Started: rsync -avPRX "Media/shows" "/mnt/disk5/"
I: 2017/08/06 17:00:16 core.go:143: Sending config
W: 2017/08/06 19:57:57 server.go:259: closing socket (&{id: ws:0xc4201841b0 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF
I: 2017/08/06 19:57:58 core.go:143: Sending config
W: 2017/08/06 20:16:13 server.go:259: closing socket (&{id: ws:0xc4201842d0 onMessage:0x6cf050 onClose:0x6cf0a0}): EOF
I: 2017/08/06 20:16:14 core.go:143: Sending config
W: 2017/08/06 20:22:59 core.go:929: Command Interrupted: rsync -avPRX "Media/shows" "/mnt/disk5/" (exit status 23 : Partial transfer due to error)
I: 2017/08/06 20:22:59 core.go:1039: 
unBALANCE - MOVE operation INTERRUPTED

Started: Aug  6, 2017 07:27:43
Ended: Aug  6, 2017 20:22:59
Elapsed: 12h55m16.176s
Command Interrupted: rsync -avPRX "Media/shows" "/mnt/disk5/" (exit status 23 : Partial transfer due to error)

happened while transferring "Media/shows", so in this case the data still exists whole in disk2. Run

$ ls -al /mnt/disk2/Media/shows

After you've checked that, you can delete the incomplete transfer on disk5

 

As for the other share (Backup), the app didn't get to transfer it (because of the previous error), so the files are still present in disk2.

 

Check that by running

$ ls -al /mnt/disk2/Backup

Not sure what you mention about crashplan, but the data in the raw disk should be there as per above.

 

Also is it possible that you were receiving crashplan data at the same time as the app was running ?

 

The app calculated these sizes for the folders you've shown in the screenshot, while the screenshot shows biggers folder sizes ?

[64.9G] /mnt/disk2/Backup/./Pictures
[40.4G] /mnt/disk2/Backup/./Documents
[263.2M] /mnt/disk2/Backup/./Videos

 

[ptirmal]

Those disks were already formatted... I assumed (incorrectly) that the disks had moved the files because it said it was complete. 

 

I'm having a separate issue restoring from my crashplan backup.

[jbrodriguez]

11 hours ago, ptirmal said:

Those disks were already formatted

 

Mmm, that's definitely not good 

 

Not sure if your notifications were enabled, but you'd have been able to read that the transfer was interrupted 

[ptirmal]

5 minutes ago, jbrodriguez said:

 

Mmm, that's definitely not good 

 

Not sure if your notifications were enabled, but you'd have been able to read that the transfer was interrupted 

Yea it sucks but that's what backups are for right?

 

I had unbalance on the default settings so I don't know if notifications are enabled by default. I did notice when it was complete it wasn't let me click validate or anything else. I should have looked into it further but I figured it transferred it all since unraid was showing what looked like the correct percentage increase. Oh well, live and learn, could have been worse.

 

And it looks like I'm restoring from my backup so whatever I deemed important survives.

Edited August 8, 2017 by ptirmal

[jbrodriguez]

1 minute ago, ptirmal said:

I had unbalance on the default settings so I don't know if notifications are enabled by default

 

No, they're not enabled by default.

 

In any case, this gives me the idea to implement a history (similar to the parity history) in an upcoming release.

[trurl]

1 hour ago, ptirmal said:

I don't know if notifications are enabled by default

Do you have unRAID Notifications setup? It is very important that unRAID be able to notify you by email or other agent if it detects problems. Many people have allowed themselves to get multiple disks with problems, for example, because they didn't have unRAID configured to notify them.

 

I haven't used this plugin lately, but I assume it uses unRAID's builtin notifications to notify you, which can't happen if you haven't configured unRAID Notifications.

[ptirmal]

47 minutes ago, trurl said:

Do you have unRAID Notifications setup? It is very important that unRAID be able to notify you by email or other agent if it detects problems. Many people have allowed themselves to get multiple disks with problems, for example, because they didn't have unRAID configured to notify them.

 

I haven't used this plugin lately, but I assume it uses unRAID's builtin notifications to notify you, which can't happen if you haven't configured unRAID Notifications.

Yes I do. I currently have them notify me via Pushbullet. I also check in on it pretty regularly.

Edited August 8, 2017 by ptirmal

[dabl]

Is it possible to use this plugin to move files from a disabled disk to a good disk?

[trurl]

2 minutes ago, dabl said:

Is it possible to use this plugin to move files from a disabled disk to a good disk?

The emulated disk can be read and written just like a real disk. It won't be very efficient since all other drives are used to emulate the disabled disk.

 

Why do you want to do this? Rebuilding the disabled disk is the normal way to keep that data.

[dabl]

15 hours ago, trurl said:

The emulated disk can be read and written just like a real disk. It won't be very efficient since all other drives are used to emulate the disabled disk.

 

Why do you want to do this? Rebuilding the disabled disk is the normal way to keep that data.

 

Thanks, that's what I figured but for whatever reason when click the CALCULATE button nothing is happening except the CALCULATE button then as expected becomes disabled.  I get no console as shown in the doc. Nothing shows in the Log page and the REFRESH button on that page is disabled. Attached is a screenshot.  

 

The reason I want to do this is I was headed down the path of converting my reiserfs drives to xfs using a new empty 8 TB drive.  The disk 3 that just became disabled is 1.5 TB and I have space available on the 8 TB disk 8. The 8 TB disk 9 is my new empty drive.  I'd prefer not to have to rebuild the 1.5 TB reiserfs disk 3 using disk 9 but perhaps that is the best thing to do at this point and then move the data to disk 8 and start the reiserfs to xfs conversion again.  To complicate matters disk 6 was showing signs of going bad so I moved the data off that disk using the krusader docker.  Disk 6 is now empty and I would like to take it out of service so things don't become even worse if/when disk 6 gets disabled, right in the middle of a move operation for example.  I'm open to any/all suggested approaches.

 

EDIT, UPDATE: I turned notifications on and restarted the plugin and am now seeing the console on CALCULATE and output on the Log page so the plugin does appear to at least be working for me now.  Am still interested in general input on suggested approaches to my situation.

 

Edited August 12, 2017 by dabl

[jbrodriguez]

2 minutes ago, dabl said:

Nothing shows in the Log

 

Can you run a quick 

$ tail -f /boot/logs/unbalance.log

Just to check what it's doing.

 

If you already stopped the process, then just check the end of the log.

[dabl]

1 hour ago, jbrodriguez said:

 

Can you run a quick 

$ tail -f /boot/logs/unbalance.log

Just to check what it's doing.

 

If you already stopped the process, then just check the end of the log.

 

Sorry, at the time of the problem CALCULATE run I did not know to look for the log and have since successfully run CALCULATE.

 

For what it's worth, attached is my latest unbalance.log from a successful run after fixing some permissions on my disabled disk 3 per the OPERATION FEEDBACK screen.

 

Also attached are screenshots from the successful run of CALCULATE with the OPERATION FEEDBACK and the run after fixing the permissions.

 

I wanted to avoid any kind of system wide newperms run so ran it manually from an ssh session with newperms /mnt/disk3

 

After doing that I now seem to be getting a 'clean' CALCULATE with no orange OPERATION FEEDBACK screen.

 

One thing I don't understand is in the unbalance.log after running newperms /mnt/disk3 it still seems to be showing a (group?) [dabl] for some files but when I look at those via ssh ls -l it shows the group as 'users'.  Maybe I'm just confused on that.

 

EDIT: also attaching the unbalance.log generated during the run with the OPERATION FEEDBACK screen since I had saved that one.

 

EDIT: Hmm comparing the .log files I see the second run just got appended so that was probably uneccessary (duh)  

Also what I was seeing with the [dabl] group was actually from the first run and was NOT in the second run......

 

unbalance.log

unbalance log from CALCULATE results OPERATION FEEDBACK .log

Edited August 12, 2017 by dabl

[jbrodriguez]

Ok, well looks like the plugin is doing what it's supposed to

 

Not sure how to proceed though, knowing your disk is in 'bad' shape as you mention.

 

rsync will probably generate some error and interrupt any operation you run.

 

Since you have enough space in at least one disk, why don't you run rsync from a (tmux/screen) command line, so you can monitor progress more closely.

[jbrodriguez]

v3.3.0 is now available !

 

2017-08-14 - 3.3.0

GATHER operation is now implemented !!

 

Consolidate folders from your user shares into a single drive, always checking that enough
space is available for the operation.

 

Also, as soon as any transfer operation gets underway, a notification is sent showing which
rsync commands will be executed (this happens after you click Move, Copy or Proceed).

 

Additionally, the following changes were made:
- Verbosity setting controls permission issue logging
- Fix issue with console panel not allowing mouse scroll
- Drive sizes in logs are shown in 'base 1000'
- Update internal components
 

The underlying transfer is the exact same as the one used for SCATTER.

 

Let me know how it works for you.

 

The next planned changes are:

- Implement an operation history 

- Run docker safe new permissions (at the source folder level) (needs discussion with Squid

[stefer]

I made the mistake (from what I gather after reading some posts) of setting my default fs to btrfs...

 

Now i came across this post and I'm wondering how to get this done (converting to xfs).   

 

I would use unbalance to get stuff off one disk, then format it to xfs? how?  stop array, ssh to array, format it with fdisk? or is there a tool in the webgui that I don't see, or a plugin to help?

 

Then what?  When I start the array, unraid will use the newly formatted drive in the storage pool and then I unbalance the others one by one until I have formattted all of them to xfs?

 

OR does unraid create a 2nd pool that is xfs formatted?

 

Or should I not fret and keep it as btrfs?

[JonathanM]

29 minutes ago, stefer said:

I would use unbalance to get stuff off one disk, then format it to xfs? how?  stop array, ssh to array, format it with fdisk? or is there a tool in the webgui that I don't see, or a plugin to help?

 

Then what?  When I start the array, unraid will use the newly formatted drive in the storage pool and then I unbalance the others one by one until I have formattted all of them to xfs?

Very close, but it's easier. With the array stopped, when you click on the disk in question you can change the format to XFS. Then when you start the array, that disk (and hopefully ONLY that disk) will show as unmountable, with an option to format ALL unmountable disks.

 

You can then use the newly empty XFS disk as a target for copying the contents of one of your other disks, lather, rinse, repeat.

 

If you have a stable server with no power issues, no forced shutdowns or crashes, then BTRFS is fine, great even. It has several benefits, but I would describe it as fragile, or brittle. It doesn't survive crashes well at all, and data recovery tools on BTRFS are sketchy as well.

 

If you keep current backups of everything you care about, which you should be doing anyway, then don't fret and just leave it BTRFS. The chances of you messing something up in the conversion are non-zero, so backup, backup, backup.

[stefer]

I have very stable power (now), but power failures do occur from time to time...  Twice this year so far.   And my UPS is not strong enough to support the PC for more than 5 seconds...

 

My important stuff is on dropbox.  Yeah the conversion sounds painful... even if it's easier like you explained.  Since i get this error when I run the calculate option :

 

There are some permission issues with the folders/files you want to move
5080 file(s)/folder(s) with an owner other than 'nobody'
0 file(s)/folder(s) with a group other than 'users'
268 folder(s) with a permission other than 'drwxrwxrwx'
1356 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'
You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)
At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation
You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

I'm thinking it's either my Dropbox docker, or NZBget/Sickbeard/Couchpotato dockers...   I don't really know how to find out though, I checked the log and i'm not really sure what to look for.

 

[stefer]

From what i can see when I ssh to my server, some shares have folders that are owned by nobody, and some are owned by a user that i created just for myself for personal stuff, budget etc...  BUT the items that are owned by my user are nowhere near the personal folder.  I wonder if it's because I used a Windows machine to copy stuff over those shares and I might have used that username when I mapped the drive in explorer.... which would explain the ownership.

I COULD fix it with the Docker Safe New Permissions command... 

 

Question, when you say :

Quote

You can then use the newly empty XFS disk as a target for copying the contents of one of your other disks, lather, rinse, repeat.

 

Once it's formatted... the array is started, but it won't part of it?  Or will it?

And how do you copy content over?  Using unBalance again?

At what point do the XFS disks become part of the array and are protected by the parity?  Once they've all been formatted?  

Edited August 15, 2017 by stefer

[JonathanM]

15 minutes ago, stefer said:

Once it's formatted... the array is started, but it won't part of it?  Or will it?

And how do you copy content over?  Using unBalance again?

At what point do the XFS disks become part of the array and are protected by the parity?  Once they've all been formatted?  

You can copy from disk to disk using any copy tool, rsync, unbalance (which is a gui frontend for rsync commands), krusader or dolphin docker, cp command, whatever.

Parity doesn't protect files or file systems, it protects entire disks. Content and format are irrelevant to parity.

All disks assigned to data slots are part of the array, whether or not they participate in user shares is up to the settings.

I recommend disabling user shares and turning on disk shares while working with converting disks, until you understand what user shares are and how they work. Since user shares are a consolidated view of the disks, it's possible to wipe out data if you mix copying between user and disk shares.

[stefer]

Thanks.  I understand what user shares are vs disk shares.  It's the whole adding it back to the pool ordeal that confused me...  I'm used to my NASes and raid in general...

 

Using unbalance should fix the issue as long as I stop my dockers so they don't write anything new and to make sure the drive is completely empty before stopping the array and formatting. 

It's funny you replied as I was reading this (one of your replies) in another tab : 

 

 

Will have to run the script to fix my access rights first though...

 

After reading a bunch of posts, I think I will stick with btrfs...  maybe down the line invest in a good APC ups...

I have to upgrade most of my drives sooner than later anyway so I could format the new ones as XFS as I replace them.

 

Edited August 15, 2017 by stefer

[wreave]

Having an issue when trying to use the gather operation. It is scanning for the show and detects that it is on 6 drives. When I select the drive to move the data to and run the move operation (dry run is disabled) it quickly spits some logs and goes back to the main gather page.

 

The logs (attached) seem to indicate it should work but is showing silly fast transfer speeds and nothing is actually being moved.

Transferred 4.68 GB at ~ 131030.23 MB/s

Any help would be appreciated, been waiting for the gather operation for a while

 

Edit: I am on unRAID 6.4.0-rc7a

unblanace_log.txt

Edited August 17, 2017 by wreave

[jbrodriguez]

Hi wreave, you didn't send the whole log, but what I can see it's only transferring 4.68gb, it does seem like it's running a dry-run.

 

Can you send the first part of the log, with the calculate section ?

[abs0lut.zer0]

Hi All

 

I know that i am going to get flammed for this but is there a PLUGIN that does the exact opposite of unbalance.

 

 

Thanks