[Plugin] rclone


Waseh

Recommended Posts

Been beating my head against the wall for awhile on a config issue. I've been using this plugin and rclone for months now with great success, first on ACD and now on GSuite. My commandline is as follows:

 

rm /mnt/rclone-bak.log # remove the old backup
mv /mnt/rclone.log /mnt/rclone-bak.log # create the new backup

rclone sync --log-file /mnt/rclone.log --max-size=40G --copy-links --transfers=4 --exclude '/FOO/*.*' --bwlimit=10M --verbose /mnt/user/ google_crypt:/ --checkers=10 -v

 

This removes a logfile from the previous run 2 prior, copies over the existing current logfile, then kicks off rclone in verbose mode with output piped to a file. I can tail these files to monitor status or just open them (hint: do not place them in a directory being backed up). The bolded portion however is a real problem...

 

I have recently moved data to a new share we'll call FOO. Foo is about 60gig of 50Kb files, it's not something I require to be backed up and the tiny file sizes push rclone into spasms so my backup never completes! I'm in a 403 tarpit... How the bloody hell do I skip the FOO directory? I have tried each of the following to no result:

 

--exclude '/FOO/*.*'

--exclude "/FOO/*.*"

--exclude='/FOO/*.*'

--exclude '/FOO/*.*'

--exclude="/FOO/*.*"

--exclude "FOO/*.*"

--exclude="FOO/*.*"

--exclude "/mnt/user/FOO/*.*"

 

I have probably tried a few other combinations too! I have been told that the path should be relative to the one given to rclone for backup so I think I've got this right not giving it the full path. I have tried with and without a leading slash, I have tried dual and single quotes - I'm pretty sure single is correct. For bonus points there's another share I'd ALSO like to skip but it's much less critical as it's simply a working directory.

 

Link to comment
  • 2 weeks later...
  • 2 weeks later...

I decided to try and use RClone to backup my appdata, which contains the Plex database.  

A large portion of it is copying but also it looks like it doesn't like some file types and has a large number of the following errors:

 

Any advice on why or what to do with this?  I see there is an appdata backup that is part of community applications and may just try to go that route.

 

RClone.thumb.PNG.811dc3ac1b2dd05faf730c0f51b1d8da.PNG

Link to comment
16 hours ago, TODDLT said:

I decided to try and use RClone to backup my appdata, which contains the Plex database.  

A large portion of it is copying but also it looks like it doesn't like some file types and has a large number of the following errors:

 

Any advice on why or what to do with this?  I see there is an appdata backup that is part of community applications and may just try to go that route.

 

RClone.thumb.PNG.811dc3ac1b2dd05faf730c0f51b1d8da.PNG


The Plex data must be using symlinks. Did you use -L or --copy-links like the error suggests?

Link to comment
6 hours ago, bobbintb said:


The Plex data must be using symlinks. Did you use -L or --copy-links like the error suggests?

 

Not knowing what "symlinks" are or being familiar with the -L or --copy-links , I didn't want to go add swtiches into the script without checking first.  

 

I'll see if I can find explanations of the switches on the Rclone page this evening but any help with symlinks or what this really means is appreciated.

Link to comment
5 minutes ago, TODDLT said:

Not knowing what "symlinks" are

Symlink / hardlinks in English (vastly simplified)

 

Imagine a file named "x" that requires 100Meg to store.

 

Such and such program wants to store a copy of that file under the filename "y"

 

Without symlinks / hardlinks, the copy would take up another 100Meg.  With links, the copy doesn't take up any extra space, as the "y" filename simply points to the exact same data that "x" does.

 

Plex makes very extensive use of them, and any backup of Plex's appdata without handling symlinks / hardlinks is pointless.  Either add the switches, or use CA's appdata backup.

Edited by Squid
Link to comment
4 minutes ago, Squid said:

Symlink / hardlinks in English (vastly simplified)

 

Imagine a file named "x" that requires 100Meg to store.

 

Such and such program wants to store a copy of that file under the filename "y"

 

Without symlinks / hardlinks, the copy would take up another 100Meg.  With links, the copy doesn't take up any extra space, as the "y" filename simply points to the exact same data that "x" does.

 

Plex makes very extensive use of them, and any backup of Plex's appdata without handling symlinks / hardlinks is pointless.  Either add the switches, or use CA's appdata backup.

 

So Plex, rather than having an extremely large database hog of a file, is saving all the image files used for the cover art etc in individual files and then referencing those locations in the database.   correct??

 

Do I need to add both switches as they are written? or is this an either / or situation?

 

I looked at the CA backup and if I can handle this in a single line for Rclone I'll do it.  If not I'll go investigate the settings I need to learn in CA Backup.

 

Edited by TODDLT
Link to comment
Just now, TODDLT said:

So Plex, rather than having an extremely large database hog of a file, is saving all the image files used for the cover art etc in individual files and then referencing those locations in the database.   correct??

Yeah.  Multiple files all referencing the exact same picture.

1 minute ago, TODDLT said:

Do I need to add both switches as they are written? or is this an either / or situation?

 

Honestly, I would assume just add the switches. (don't use rclone - I did the appdata backup plg, which may or may not suit your purposes depending upon the destination)

Link to comment
  • 3 weeks later...

I keep getting this error when I try to update the plugin

 

plugin: updating: rclone.plg
plugin: downloading: http://beta.rclone.org/rclone-beta-latest-linux-amd64.zip ... done

+==============================================================================
| Skipping package rclone-2016.11.19-bundle (already installed)
+==============================================================================


Archive: /boot/config/plugins/rclone-beta/install/rclone-beta-latest.zip
creating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/rclone 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/README.txt 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/README.html 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/rclone.1 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/git-log.txt 
Copy failed - rclone running
plugin: run failed: /bin/bash retval: 1

 

Link to comment

I should probably change that error text. What is happening is the copy command to copy the rclone file is failing. This is usually because the rclone file is being used in some process.
What used to happen before i added this was that the update went through without any errors but rclone wouldnt actually be updated.
 

  • Like 1
Link to comment
On 9/28/2017 at 7:49 PM, Waseh said:

I should probably change that error text. What is happening is the copy command to copy the rclone file is failing. This is usually because the rclone file is being used in some process.
What used to happen before i added this was that the update went through without any errors but rclone wouldnt actually be updated.
 

 

I'm an idiot. How can I stop it and update? :) 

Link to comment
  • 2 weeks later...
On 9/24/2017 at 8:43 AM, GilbN said:

I keep getting this error when I try to update the plugin

 


plugin: updating: rclone.plg
plugin: downloading: http://beta.rclone.org/rclone-beta-latest-linux-amd64.zip ... done

+==============================================================================
| Skipping package rclone-2016.11.19-bundle (already installed)
+==============================================================================


Archive: /boot/config/plugins/rclone-beta/install/rclone-beta-latest.zip
creating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/rclone 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/README.txt 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/README.html 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/rclone.1 
inflating: /boot/config/plugins/rclone-beta/install/rclone-v1.37-157-g5a3a56ab#U03b2-linux-amd64/git-log.txt 
Copy failed - rclone running
plugin: run failed: /bin/bash retval: 1

 

Same error here.

Have tried "killall -9 rclone" but dosent work.  Prob because of mount points

Link to comment
  • 3 weeks later...
On 10/14/2017 at 6:02 PM, tooviral said:

Same error here.

Have tried "killall -9 rclone" but dosent work.  Prob because of mount points

I'd been seeing this error for a while, just took a look to actually fix it.  After searching processes there was also an rcloneorig process that needed to be killed before I could update.

 

root@unraid:~# ps aux | grep rclone
root      5954  0.0  0.0   9640  1852 pts/1    S+   05:09   0:00 grep rclone
root     10526  0.0  0.0  39764 16280 ?        Sl   Oct10   1:23 rcloneorig --config /boot/config/plugins/rclone-beta/.rclone.conf mount --max-read-ahead 1024k --allow-other securedb: /mnt/disks/vault
root@unraid:~# kill -9 10526

 

Link to comment
45 minutes ago, ajgriglak said:

Hey guys.

 

Looks like this docker hasn't been updated in quite a while and there hasn't been any activity in the thread.  If I'm coming in fresh now, is this still a viable Unraid <> Google Drive sync tool?  Is there anything better / newer?


To be clear, it's the plugin version, not a docker. Just in case you specifically wanted that one. I don't recall but does this need frequent updating. Does the plugin just pull the latest stable release?

Link to comment
  • 4 weeks later...

Hello. I am attempting to use rclone to backup my unraid nas to B2. I have been following the instructions here. Unfortunately, when I use rclone to create a mount point in /mnt/disks that should be linked to my B2 account, the command never creates the mount point. I have tried many ways. I execute the script with 'user scripts' plugin, running in background. The logs in 'user scripts' show no errors. However, never is there a folder created in /mnt/disks. Has anyone had this issue?

 

Thanks.

Link to comment
  • 3 weeks later...
  • 2 weeks later...

I used RClone some time ago, before the issue with Amazon Cloud Drive and encrypted secrets in the code.  I've just re-installed the plugin on my new build but I'm really struggling to setup RClone again.  I don't know how to get my Amazon Cloud Drive client_id and client_secret or use a a third party ouath proxy as suggested in the documentation here https://rclone.org/amazonclouddrive/ as the wizard setup just worked in the past.

 

Is there an updated setup walk-through to help novices like me set this up, or can anyone help guide me through the initial setup please?

Link to comment
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.