[Plugin] unbalanced


Recommended Posts

48 minutes ago, insaneavi said:

I installed the plugin via the github link

 

I will check if this causes trouble.

 

49 minutes ago, insaneavi said:

i am unable to get the status of it to change from Stopped

 

Do you mean that all the buttons are disabled ?

 

If that's the case, follow the quick start instructions.

 

If that's not the case, please do the following

$ killall unbalance
$ cd /usr/local/emhttp/plugins/unbalance
$ ./unbalance

and post the output here to check it out.

Link to comment
34 minutes ago, jbrodriguez said:

 

I will check if this causes trouble.

 

 

Do you mean that all the buttons are disabled ?

 

If that's the case, follow the quick start instructions.

 

If that's not the case, please do the following


$ killall unbalance
$ cd /usr/local/emhttp/plugins/unbalance
$ ./unbalance

and post the output here to check it out.

 

I have also attached what the web inference displays when i try to start the application. 


as requested:

 

root@TOPSITE:/usr/local/emhttp/plugins/unbalance# killall unbalance
unbalance: no process found
root@TOPSITE:/usr/local/emhttp/plugins/unbalance# cd /usr/local/emhttp/plugins/unbalance
root@TOPSITE:/usr/local/emhttp/plugins/unbalance# ./unbalance                  
root@TOPSITE:/usr/local/emhttp/plugins/unbalance# ls
CHANGES    VERSION  event/   img/        scripts/    unbalance.page
README.md  app/     images/  index.html  unbalance*  unbalance.png

 

2017-06-28 19_37_16-TOPSITE_unbalance.png

Edited by insaneavi
Link to comment
11 minutes ago, insaneavi said:

root@TOPSITE:/usr/local/emhttp/plugins/unbalance# ./unbalance    

 

This is the problem here. The executable isn't starting.

 

This used to be an issue in 1.8.x, but it was "solved" :D

 

I believe that installing via the github link doesn't create the conf file it expects, although it should work without one.

 

Try the following :

 

$ nano /boot/config/plugins/unbalance/unbalance.conf
# <paste the following>

SERVICE="disable"
PORT="6237"
RUNAS="nobody"

# click control + x, then y, then <enter> to save the file

Then go the settings page and enable the server.

Link to comment
9 minutes ago, jbrodriguez said:

 

This is the problem here. The executable isn't starting.

 

This used to be an issue in 1.8.x, but it was "solved" :D

 

I believe that installing via the github link doesn't create the conf file it expects, although it should work without one.

 

Try the following :

 


$ nano /boot/config/plugins/unbalance/unbalance.conf
# <paste the following>

SERVICE="disable"
PORT="6237"
RUNAS="nobody"

# click control + x, then y, then <enter> to save the file

Then go the settings page and enable the server.


tried it, same issue:

 

root@TOPSITE:/boot/config/plugins/unbalance# ls
unbalance-3.0.0.tgz*  unbalance.cfg*  unbalance.conf*
root@TOPSITE:/boot/config/plugins/unbalance#
 

I was able to create the .conf file and save with the above code, but still unable to start it.

 

Do i need to reboot the server after creating the conf file?

 

here is the system log

 

Jun 28 20:05:33 TOPSITE php: /usr/local/emhttp/plugins/unbalance/scripts/start
Jun 28 20:05:33 TOPSITE sudo: root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
 

Edited by insaneavi
Link to comment
7 minutes ago, jbrodriguez said:

 

Not really.

 

So, if you try 


$ cd /usr/local/emhttp/plugins/unbalance
$ ./unbalance

it comes up empty again ?

root@TOPSITE:~# cd /usr/local/emhttp/plugins/unbalance
root@TOPSITE:/usr/local/emhttp/plugins/unbalance# ./unbalance
configuration variable provided but not defined: SERVICE
Usage of ./unbalance:
  -apiFolders string
        folders to look for api endpoints (default "/var/local/emhttp")
  -config string
        config location (default "/boot/config/plugins/unbalance/unbalance.conf")
  -dryRun
        perform a dry-run rather than actual work (default true)
  -folders string
        deprecated - do not use
  -log string
        pathname where log file will be written to (default "/boot/logs/unbalance.log")
  -notifyCalc int
        notify via email after calculation operation has completed (unraid notifications must be set up first): 0 - No notifications; 1 - Simple notifications; 2 - Detailed notifications
  -notifyMove int
        notify via email after move operation has completed (unraid notifications must be set up first): 0 - No notifications; 1 - Simple notifications; 2 - Detailed notifications
  -port string
        port to run the server (default "6237")
  -rsyncFlags string
        custom rsync flags
root@TOPSITE:/usr/local/emhttp/plugins/unbalance#

it now gives me the above results

log entry when trying to enable:

 

Jun 28 20:15:41 TOPSITE php: /usr/local/emhttp/plugins/unbalance/scripts/start
Jun 28 20:15:41 TOPSITE sudo: root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237

 

 

 

 

Edited by insaneavi
Link to comment
5 minutes ago, jbrodriguez said:

Ok, remove the contents of the file (/boot/config/plugins/unbalance/unbalance.conf), but leave it there (an empty file).

 

Then start it from the command line again.

 

Thank you!, it now runs properly, I was also able to stop the service, then turn it back on via the web interface!

giphy-downsized.gif

Link to comment

I did some massive file movement - in fact it was consolidation of shares to dedicated drives, as the spread led to significant spin-up times.
I'd like to feed back some of my experience and suggest some points that can possibly be improved.

 

1. Most of my issues rooted in missing permissions.
Unfortunately rsync's failure messages are very cryptic thus useless most of the time.
It will never tell you that you have permission issues! The plugin does, but not "strong enough".
Therefore it is a matter of usability to add this utility to the plugin.
Maybe provide a "change permissions" button if the dry run finds a related issue.
It's annoying if you do the "dry run" and are told to "install docker xy first".
Much more simple and without the need of a docker is the "newperms" script that comes with unRAID.

Long story short, it is recommended to run the "newperms" on the source AND on the target!
The plugin should have this option included.

 

2. Due to 1) I had many aborted runs which left me with lots of duplicates.
Obviously the command aborted at some point during the copy so it didn't even come to the "remove sources".
After figuring out that the permissions were the issue, I ran the job again and that removed the sources but
I had to remember the exact task! My task was consolidating so it was easy to find the dupes as they were the sources,
but I imagine this can turn into a problem. e.g. I have some shares spanning multiple drives and I
found a couple of identical files on different drives of the share. Must have been moving the source to
different drives in different sessions. Need to hunt the dupes now.
I have no solution how to handle this, but if an rsync-command is failing at some point, dupes will be
created. Maybe there is an rsync option to "remove sources" right after the "copy" of each file?
Also a more comprehensive log of commands issues could be useful.
Maybe somewhat like:

date - rsync xyz source target etc. success - 0 errors - run again knob

At the moment the log is too much. Maybe let's have some log levels (normal/debug/full)?

That leads straight to

3. Logs are fine but it turned out that the unRAID OS update aborted due to space shortage.
You know what I'm gonna say...there must be some sort of log handling.
Rotation, size limit, whatever.
I had hundreds of MB unbalance log data...

 

Anyway, many thanks for the hard work!

 

...sry, it got quite long

Edited by Fireball3
Link to comment

Hello,

 

I am trying to update the plugin and I am receiving the error "plugin: wget: "https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz" download failure (Invalid URL / Server error response)"

 

I used the App Tab (Community Application) and also copy and pasting the url "https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg" in the plugins tab. It's been happening for a couple of days now, is this only on my end?

 

Thanks

Link to comment
1 minute ago, mikegiann said:

Hello,

 

I am trying to update the plugin and I am receiving the error "plugin: wget: "https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz" download failure (Invalid URL / Server error response)"

 

I used the App Tab (Community Application) and also copy and pasting the url "https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg" in the plugins tab. It's been happening for a couple of days now, is this only on my end?

 

Thanks

I just tried (in chrome) pasting the url for the .tgz you quoted in your first paragraph, and it downloaded the file just fine so it looks like that file is there. Can you ping github from your server?

Link to comment
8 minutes ago, trurl said:

I just tried (in chrome) pasting the url for the .tgz you quoted in your first paragraph, and it downloaded the file just fine so it looks like that file is there. Can you ping github from your server?

Yes, I can ping and get a response, here is the complete log when I try to install:

 

plugin: installing: https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg
plugin: downloading https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg
plugin: downloading: https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg ... done
plugin: downloading: "https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz" ... failed (Invalid URL / Server error response)
plugin: wget: "https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz" download failure (Invalid URL / Server error response)

 

If I click on the link https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg I do see the contents of that page, just not through Unraid.

 

I tried with Firefox and Chrome all latest versions.

Edited by mikegiann
Browser Used
Link to comment
1 hour ago, mikegiann said:

If I click on the link https://raw.githubusercontent.com/jbrodriguez/unraid/master/plugins/unbalance.plg I do see the contents of that page

If you instead click on the link https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz does it download the file? That's what I did earlier when I said

1 hour ago, trurl said:

I just tried (in chrome) pasting the url for the .tgz you quoted in your first paragraph, and it downloaded the file just fine so it looks like that file is there.

 

And just now I tried using CA to install this plugin and it worked for me.

 

Have you tried any other plugins? Do they work?

Link to comment

Thanks for your comments Fireball3 !

 

6 hours ago, Fireball3 said:

1. Most of my issues rooted in missing permissions.
....

Long story short, it is recommended to run the "newperms" on the source AND on the target!
The plugin should have this option included

 

You're right about rsync errors being cryptic O.o

About the "dockers safe newperms" (by Squid), it runs on ALL the disks, so that's why it's suggested to run that first :)

In any case, I understand that sometimes you just want to run it on the source (disk or folders), and probably on the target as you mention.

I'll take this into consideration.

 

6 hours ago, Fireball3 said:

2. Due to 1) I had many aborted runs which left me with lots of duplicates.
......
I had to remember the exact task! My task was consolidating so it was easy to find the dupes as they were the sources,
....

Maybe there is an rsync option to "remove sources" right after the "copy" of each file?
Also a more comprehensive log of commands issues could be useful.
Maybe somewhat like:


date - rsync xyz source target etc. success - 0 errors - run again knob

At the moment the log is too much. Maybe let's have some log levels (normal/debug/full)?

 

You're right, I feel the log is too much too. It logs everything rsync outputs, and if you have many files, it's a lot of output :)

I will look into improving that.

Oh, I wanted to mention, it reports each command it ran (even after a failure) in the log file, so you don't need to remember the exact task. It will also send an email if you have notifications set up.

 

6 hours ago, Fireball3 said:

3. Logs are fine but it turned out that the unRAID OS update aborted due to space shortage.

 

It currently rotates logs, a max of 10 log files of 10Mb each. It shouldn't grow to the hundreds of Mb.

In your case, do you have a single 100Mb+ file ? Or is it multiple log files in excess of that ?

 

6 hours ago, Fireball3 said:

Anyway, many thanks for the hard work!

 

Thanks again for the comments ! They help improve the plugin.

 

Link to comment
58 minutes ago, trurl said:

And just now I tried using CA to install this plugin and it worked for me.

 

Have you tried any other plugins? Do they work?

 

mikegiann, the same for me, I installed with CA with no issues.

 

Just as a test, if you 

$ wget https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz

from the a command line on the server, does it work ?

Link to comment
1 hour ago, jbrodriguez said:

 

mikegiann, the same for me, I installed with CA with no issues.

 

Just as a test, if you 


$ wget https://github.com/jbrodriguez/unbalance/releases/download/3.0.0/unbalance-3.0.0.tgz

from the a command line on the server, does it work ?

I was able to download the files from the browser but not through Unraid, then when trying it from the command line, as you suggested, I saw where the error occurred, it was refusing the certificate from my self signed Untangle UTM appliance. Once I added the github to the exception of not to be scanned by SSL Inspector, the plugin installed quickly and correctly.

 

All the other plugins that I have update automatically without problem, that is why I did not think about the Untangle appliance.

 

Thanks for pointing me in the right direction and the help.

Edited by mikegiann
Link to comment

v3.1.0 is available !

 

2017-06-30 - 3.1.0

 

Log verbosity can now be modified through a setting:

- Normal
Don't print rsync output

 

- Full
Print full rsync output

 

When a transfer has finished, it will now log how much data was
transferred and the approximate speed (in MB/s).

 

Additionally, the following changes were made:
- Fix an issue when refreshing the UI with notifications enabled
- Improve handling of config file (when not present)
- Other bug fixes and improvements

Link to comment

1st time using Unbalance (v3.1.0) and I ran into an issue.

 

I followed the steps as listed:
selected the "From" drive,
selected 1 "To" drive
selected folders to move
clicked the CALCULATE button - reported 330GB of files to move
clicked "move" with "Dry Run" checked - log output look OK


Repeated the above steps, only with "Dry Run" unchecked

 

Process finished in 20 seconds

 

The "To" drive has no files, the the "From" drive is now empty.

 

The entries from the Log shows:
....
I: 2017/06/30 22:14:07 core.go:802: Command Started: rsync -avPRX --dry-run "Thumbnails/2" "/mnt/disk6/"

I: 2017/06/30 22:14:07 core.go:884: Command Finished

I: 2017/06/30 22:14:07 core.go:890: Current progress: 99.71% done ~ 0s left (47655.92 MB/s)

I: 2017/06/30 22:14:07 core.go:900: Removing: rm -rf "/mnt/disk4/Thumbnails/2"

I: 2017/06/30 22:14:08 core.go:802: Command Started: rsync -avPRX --dry-run "Thumbnails/a" "/mnt/disk6/"

I: 2017/06/30 22:14:08 core.go:884: Command Finished

I: 2017/06/30 22:14:08 core.go:890: Current progress: 99.74% done ~ 0s left (45807.58 MB/s)

I: 2017/06/30 22:14:08 core.go:900: Removing: rm -rf "/mnt/disk4/Thumbnails/a"
...

As you can see the rsync command still has "--dry-run" even though the "Dry Run" box had been unchecked.

 

Is there a way of seeing the complete log to see what was all removed?

 

Thanks

Edited by KeithG
Fix typos
Link to comment
On 7/1/2017 at 10:31 AM, KeithG said:

Is there a way of seeing the complete log to see what was all removed?

 

Hi, logs are available at /boot/logs/unbalance.log.

 

I made a change to how  dry-run is handled even after a calculation.

 

Send me the logs (on pm if you want) to further check.

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.