wsume99

Help with rsync and deltacopy

Recommended Posts

I've followed all the instruction in the infamous Tibbar & WeeboTech rsync thread. I am running deltacopy client on my windows 7 desktop and I'm trying to backup photos from that machine onto my unraid server.

 

Here is my /boot/custom/etc/rsyncd.conf file

uid             = root
gid             = root
use chroot      = no
max connections = 4
pid file        = /var/run/rsyncd.pid
timeout         = 600
log file        = /var/log/rsyncd.log

[photobackups]
    path = /mnt/user/photobackups
    comment = photobackups
    read only = FALSE
list = yes

 

And the contents of /boot/custom/etc/rc.d/S20-init.rsyncd

#!/bin/bash

if ! grep ^rsync /etc/inetd.conf > /dev/null ; then 
cat <<-EOF >> /etc/inetd.conf
rsync   stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/bin/rsync --daemon 
EOF
read PID < /var/run/inetd.pid
kill -1 ${PID}
fi

 

I setup a new profile named temp to test out the system and here is the response I get

Executing: rsync.exe  -v -rlt -z --chmod=a=rw,Da+x --delete "/cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf"
Profile 'temp' executed in 2 milliseconds. It ran successfully.

 

The only problem is that there are no files in my photobackups share. I'm guessing that I don't have the path to my share setup correctly but it looks right to me.

 

I've also attached a screenshot of the deltacopy profile I created.

 

Any ideas? Thanks in advance.

deltacopy.jpg.6c0835941089ffdcc80a8447cd763fc4.jpg

Share this post


Link to post
Share on other sites

Well I've searched all over the unraid forums and the web and I can't seem to find any answer. Hopefully someone will help me out?

Share this post


Link to post
Share on other sites

This line:

/cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf"

 

You only want the one file?

 

If there's a /mnt/user/photobackups directory on unRAID, does it have a documents directory inside containing the PDF?

 

If the file hasn't changed, any rsyncs after the first would finish almost instantly, so the 2 milliseconds message wouldn't be surprising.

 

What do you see in /var/log/rsyncd.log?

 

 

Share this post


Link to post
Share on other sites

I'm only copying a single file right now to make sure everything works properly. I did try copying over the directory I actually plan on backing up and I got the same result.

 

/mnt/user/photobackups is completely empty. I manually created a Documents folder in /mnt/user/photobackups just to see if that made a difference and it did not.

 

Here are the contents of /var/log/rsync.log (192.168.1.151 is the IP address of the client PC I'm copying from):

2011/09/01 22:34:21 [1578] name lookup failed for 192.168.1.151: Name or service not known
2011/09/01 22:34:21 [1578] connect from UNKNOWN (192.168.1.151)
2011/09/01 22:34:21 [1578] module-list request from UNKNOWN (192.168.1.151)
2011/09/01 22:35:11 [1580] name lookup failed for 192.168.1.151: Name or service not known
2011/09/01 22:35:11 [1580] connect from UNKNOWN (192.168.1.151)
2011/09/01 22:35:11 [1580] module-list request from UNKNOWN (192.168.1.151)
2011/09/01 22:35:13 [1581] name lookup failed for 192.168.1.151: Name or service not known
2011/09/01 22:35:13 [1581] connect from UNKNOWN (192.168.1.151)
2011/09/01 22:35:13 [1581] module-list request from UNKNOWN (192.168.1.151)
2011/09/01 22:47:56 [1883] name lookup failed for 192.168.1.151: Name or service not known
2011/09/01 22:47:56 [1883] connect from UNKNOWN (192.168.1.151)
2011/09/01 22:47:56 [1883] module-list request from UNKNOWN (192.168.1.151)
2011/09/02 16:58:47 [20828] name lookup failed for 192.168.1.151: Name or service not known
2011/09/02 16:58:47 [20828] connect from UNKNOWN (192.168.1.151)
2011/09/02 16:58:47 [20828] module-list request from UNKNOWN (192.168.1.151)
2011/09/02 16:59:08 [20893] name lookup failed for 192.168.1.151: Name or service not known
2011/09/02 16:59:08 [20893] connect from UNKNOWN (192.168.1.151)
2011/09/02 16:59:08 [20893] module-list request from UNKNOWN (192.168.1.151)
2011/09/02 17:39:55 [25142] name lookup failed for 192.168.1.151: Name or service not known
2011/09/02 17:39:55 [25142] connect from UNKNOWN (192.168.1.151)
2011/09/02 17:39:55 [25142] module-list request from UNKNOWN (192.168.1.151)
2011/09/02 17:39:57 [25143] name lookup failed for 192.168.1.151: Name or service not known
2011/09/02 17:39:57 [25143] connect from UNKNOWN (192.168.1.151)
2011/09/02 17:39:57 [25143] module-list request from UNKNOWN (192.168.1.151)

 

Based on the timestamps I believe that all these messages are generated when I'm creating the profile in Deltacopy. When I actually run the profile nothing is documented in the log file. Instead I get the message in the Deltacopy window that I attached in my previous post that says it completed successfully.

Share this post


Link to post
Share on other sites

The results you showed earlier:

 

Executing: rsync.exe  -v -rlt -z --chmod=a=rw,Da+x --delete "/cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf"
Profile 'temp' executed in 2 milliseconds. It ran successfully.

 

Was that everything that was in the deltacopy status window? That's tells me it sees the file at the destination already.

 

Try switching things up. Pick a different source file or even use a smallish folder. Change the destination to something different in your rsyncd module. Change options in the deltacopy profile so that only verbose logging and recursive/preserve time are set. Now what do you get in the status window? Does anything show up at /mnt/user/whatever?

Share this post


Link to post
Share on other sites

I just installed deltacopy again, after many a moon. Now I remember why I use command-line rsync. :)

 

Anyway, how are you running the profile? It works fine for me when I use "run now. I tried "display run command" instead and got:

 

Executing: rsync.exe  -v -rlt -v "/cygdrive/C/Users/Me/Desktop/filetest/" "tower::backupst/filetest/"
Profile 'testb' executed in 1 milliseconds. It ran successfully.

 

Looks familiar.

Share this post


Link to post
Share on other sites

I just installed deltacopy again, after many a moon. Now I remember why I use command-line rsync. :)

If you know of a better way to backup files from a windows machine to the server then I'm all ears. I just want to automate the process of backing up my photos to the server.

 

Anyway, how are you running the profile?
I was using only the "Display Run Command" option. I tried the "Run Now" method and it worked with my test file. I knew I was doing something really dumb. Now let's see if it'll work as a scheduled task.

 

Thanks a lot for the help!

 

Share this post


Link to post
Share on other sites

If you know of a better way to backup files from a windows machine to the server then I'm all ears.

 

Command-line rsync. :)http://www.itefix.no/i2/node/10650

 

Really, once you see how a few of those options work it isn't any more difficult, and behavior is definitely simpler to control via scripting, for backup rotation, versioning, determining if you're on the right network, etc. My production rsync setups are mostly done through universal scripts located on the server that assume minimum backup sources, then add include/exclude pattern files customized for and located on each client.

 

I tried the "Run Now" method and it worked with my test file. I knew I was doing something really dumb. Now let's see if it'll work as a scheduled task.

 

Thanks a lot for the help!

 

Always nice when it's something simple.

Share this post


Link to post
Share on other sites

Command-line rsync. :)http://www.itefix.no/i2/node/10650

 

Really, once you see how a few of those options work it isn't any more difficult, and behavior is definitely simpler to control via scripting, for backup rotation, versioning, determining if you're on the right network, etc. My production rsync setups are mostly done through universal scripts located on the server that assume minimum backup sources, then add include/exclude pattern files customized for and located on each client.

Ok. I'll check that out. I assume there is a way to make the script wake up the client PC if it is in sleep mode (S3). Sounds like it's time to do some more reading.

Share this post


Link to post
Share on other sites

There's no auto-wake with cwrsync. It's a raw tool with a few batch samples. However, you will recognize everything you've done with deltacopy with many more options available.

 

To wake a system you can use wake-on-lan, triggered from your server, or set systems to auto-boot from the BIOS shortly before the scheduled task. There are countless tools to send WOL magic packets. Here's a start:

 

  http://www.wlug.org.nz/WakeOnLanNotes

 

I use etherwake on nix but it's usually wolcmd on Windows since that's what I'm working with at client sites.

 

  http://www.depicus.com/wake-on-lan/wake-on-lan-cmd.aspx

 

Because of the hardware requirements and variety it's rare to get every system working with WOL. That means using the BIOS auto-boot facility, which means changing the schedule is a pain. Do everything possible with WOL and then BIOS auto-boot where you have to, spreading systems out across the available backup time.

 

Share this post


Link to post
Share on other sites

Well now when I setup a scheduled task in Deltacopy it does not execute so I've decided to abandon that method and use cwrsync. My plan is to create a script that will wake my windows pc and then run a rsync job on the server.  I'll load the script as a cron job so it'll run regularly. Does this make sense?

 

I found this post where WeeboTech posted a compiled version of ether-wake. I copied the file onto my flash drive and I can now wake my windows machine over the LAN.

 

I also installed cwrsync server onto my windows machine.  I then modified the rsync.conf file on the windows machine and added a "test" module. Here are the contents of the rsync.conf file:

use chroot = false
strict modes = false
hosts allow = *
log file = rsyncd.log

# Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
[test]
path = /cygdrive/c/Users/John/Downloads
read only = true
transfer logging = yes

 

As a test I tried to run rsync from the server and copy the contents of c:/Users/John/Downloads to the photobackups share on the server.  Here is the command I used:

rsync -v -rlt -z --chmod=a=rw,Da+x --delete 192.168.1.152::test /mnt/user/photobackups/

 

I received the following response:

root@Tower:~# rsync -v -rlt -z --chmod=a=rw,Da+x --delete 192.168.1.152::test /m
nt/user/photobackups/
@ERROR: invalid uid nobody
rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec
eiver=3.0.2]

 

I also tried this command thinking I needed to have a user account identified on the windows pc:

rsync -v -rlt -z --chmod=a=rw,Da+x --delete John@192.168.1.152::test /mnt/user/photobackups/

But I got the same error as before.

 

What am I doing wrong here?

 

Share this post


Link to post
Share on other sites

I prefer using cwrsync as a client on backup clients but it'll work either way.

 

I also installed cwrsync server onto my windows machine.  I then modified the rsync.conf file on the windows machine and added a "test" module. Here are the contents of the rsync.conf file:

 

rsync.conf or rsyncd.conf? Sticking with convention of rsyncd.conf for the server, it needs uid & gid = 0 to play well.

 

uid = 0
gid = 0
use chroot = false
strict modes = false
hosts allow = *
log file = rsyncd.log
...

 

I'd start over with the rsync options. Try -a (useful combo), -P (progress and partial), and as many v's as are needed/interesting. -z is compression and doesn't usually help over a LAN though it does work your cpus.

 

rsync -avP maybe with --delete

 

Look into -b, --backup-dir, --include-from and --exclude-from to help manage things.

Share this post


Link to post
Share on other sites

I prefer using cwrsync as a client on backup clients but it'll work either way.

Is that just a personal preference or is there reason using cwrsync as a client on the windows pc is superior?

 

rsync.conf or rsyncd.conf? Sticking with convention of rsyncd.conf for the server, it needs uid & gid = 0 to play well.

It was rsyncd.conf. I added the uid & gid = 0 to the file and it worked.

 

I used rsync -avP --delete and it worked fine, except it did not copy any of the files that were in the source folder that had a .exe extension. I was thinking that it would be an exact duplicate of the source but it wasn't.  I guess I'll need to start experimenting with the options.

 

Thank you for all your assistance.

Share this post


Link to post
Share on other sites

Might be blood sugar and sunshine but that's odd. -C would do it. What's the entire command line?

 

About using client vs. server, it's mostly how you think about things. I've never liked publishing entire drives for external access or leaving extra code running. But those aren't exactly show stoppers. Either way will work fine, each with its own flavor.

Share this post


Link to post
Share on other sites

The command I used was

rsync -avP --delete 192.168.1.152::test /mnt/user/photobackup

 

I've been running more tests and I've determined that the files are there but for whatever reason I cannot see them in windows explorer. I can see all the files if I use the file browser in unmenu.

Share this post


Link to post
Share on other sites

Ah, the joys of Windows vs. nix permissions and Samba... Try adding the option "--chmod=Fo-X" to your rsync command. That keeps the other execute bit (001) clear for files. Samba maps it to the hidden attribute.

 

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test /mnt/user/photobackup

Share this post


Link to post
Share on other sites

I added the --chmod=Fo-X option to the command and now the .exe files are visible in windows explorer. Thanks so much. I'm actually beginning to like the CLI.

 

I have another question. I assume that only one rsync instance can be running at a time between the client and the server. Is this correct? For example, if I wanted to backup c:/users/Ziggy/Downloads/ and c:/users/Ziggy/My Pictures/ I would have to write a single script like this:

#!/bin/bash

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test /mnt/user/photobackups/Downloads/
rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/My Pictures/

 

Where rsyncd.conf defines test as /cygdrive/c/Users/Ziggy/Downloads/ and test1 as /cygdrive/c/Users/Ziggy/My Pictures/. I could then place the script into /etc/cron.daily/ and it should execute a backup once a day of the two folders, first test and then test1. Is that correct?

 

 

Share this post


Link to post
Share on other sites

Multiple transfers can run simultaneously, up to the limits of max connections (in rsyncd.conf) and your system. Really though, it makes little sense to fill memory and thrash your disks between just two boxes. Better to do them serially as you are now.

 

I spoke earlier about not publishing an entire drive, but I'll go against that for now. You can make your setup simpler by backing out the module one or more levels. e.g. Change the module path for [test] to /cygdrive/c/Users/John. Then, in your rsync command, use:

 

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test/Downloads/ /mnt/user/photobackups/Downloads/

 

If you plan on more than a few paths, better yet would be to put it in a loop using include-from to pull from a file containing your list of backup sources. Then add exclude-from to filter out the unwanted.

 

Echoing myself, you might also look into the -b and --backup-dir options which make versioning simple. Create a timestamp, use it in a directory name for --backup-dir, and anything that's changed or been removed will be moved there instead of being lost with the --delete option.

Share this post


Link to post
Share on other sites

Now I'm having trouble with the spaces in the path /mnt/user/photobackups/My Pictures/. I had this issue before but I just eliminated the spaces from my test directories. That is not really a solution because I don't want to have to rename all my existing directories. I've tried all of the variations below and it still won't work?

 

root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes
t1 /mnt/user/photobackups/My Pictures/
Unexpected local arg: /mnt/user/photobackups/My
If arg is a remote file/dir, prefix it with a colon (.
rsync error: syntax or usage error (code 1) at main.c(1207) [receiver=3.0.2]
root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes
t1 "/mnt/user/photobackups/My Pictures/"
@ERROR: chdir failed
rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec
eiver=3.0.2]
root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes
t1 '/mnt/user/photobackups/My Pictures/'
@ERROR: chdir failed
rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec
eiver=3.0.2]
root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes
t1 /mnt/user/photobackups/My\ Pictures/
@ERROR: chdir failed
rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec
eiver=3.0.2]
root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes
t1 /mnt/user/photobackups/My"\" Pictures/

 

I think I will look into the -b and --backup-dir options. I like the thought of having older versions of files for maybe a month or so. But first thing for me is to get the automation working the way I need it to, then I can play with the options. I never imagined that I would need to do so much experimenting with test files - silly me.

 

EDIT: I'm such a noob. All the options I tried did work - well at least the space in the path wasn't the problem.  The first error was a result of the space in the path but the rest of them were a problem with the test1 module I've created. I'm not sure what's wrong with it yet but I'll have to look into it.

 

Also, I found that /mnt/user/photobackups/"My Pictures"/ syntax works as well.

Share this post


Link to post
Share on other sites

I think libraries are getting in your way. There isn't a /users/name/My Pictures from cygwin's perspective. Try navigating from the windows command line to see what cwrsync sees.

 

Just "Pictures" should do.

 

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/Pictures/

Share this post


Link to post
Share on other sites

The problem appears to be the spaces in the path for the module I defined in the rsyncd.conf file on the windows machine.

 

I have the following:

uid = 0
gid = 0
use chroot = false
strict modes = false
hosts allow = 192.168.1.150
log file = rsyncd.log

# Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
[test]
path = /cygdrive/c/Users/John/"My Pictures"/
read only = true
transfer logging = yes

 

Again the problem is the fact that there are spaces in the path. The folder c:\Users\John\My Pictures\ does exist on the machine. I've read quite a few posts with similar issues but haven't seen a solid solution yet.

 

Share this post


Link to post
Share on other sites

Ah Ha - I guess I should have listened better when you warned me about libraries but I didn't think that was the problem because I could see the My Pictures folder in windows explorer. I guess the lesson I should learn here is not to trust what windows explorer says.

 

C:\Users\John>dir

Directory of C:\Users\John

07/14/2011  03:18 AM    <DIR>          .
07/14/2011  03:18 AM    <DIR>          ..
10/10/2010  01:27 PM    <DIR>          .idlerc
04/12/2011  10:04 PM    <DIR>          Contacts
09/04/2011  10:55 PM    <DIR>          Desktop
07/16/2011  02:32 PM    <DIR>          Documents
08/19/2011  09:05 PM    <DIR>          Downloads
04/12/2011  10:04 PM    <DIR>          Favorites
04/12/2011  10:04 PM    <DIR>          Links
02/23/2011  07:43 PM    <DIR>          Logitech
04/12/2011  10:04 PM    <DIR>          Music
09/01/2011  11:00 PM    <DIR>          Pictures
04/12/2011  10:04 PM    <DIR>          Saved Games
04/12/2011  10:04 PM    <DIR>          Searches
03/14/2010  05:58 PM                 0 Sti_Trace.log
04/12/2011  10:04 PM    <DIR>          Videos
09/12/2010  04:51 PM                99 wakeup.bat
               2 File(s)             99 bytes
              15 Dir(s)  13,327,364,096 bytes free

 

So according to the dir listing there is no My Pictures folder. The proper path should be c:/Users/John/Pictures/. So the rsyncd.conf file should have a module named test1 with this path:

path = /cygdrive/c/Users/John/Pictures/

 

and the rsync command would be:

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/"whatever I want to call it but I need quotes if there are spaces"/

Share this post


Link to post
Share on other sites

I well remember my own shock at seeing that the first time.

 

I'm sure it's all about compatibility. With software, not us.

 

Again, I'd publish the module as /cygdrive/c/Users/John, or even higher up the tree so can get whatever you like without adding more modules. e.g.:

 

If module uses                      rsync uses
/cygdrive/c/Users/John/Pictures     address::module
/cygdrive/c/Users/John              address::module/Pictures
/cygdrive/Users                     address::module/John/Pictures
...

 

If you get to the point of looping through your list of sources vs. hard-coding individual rsync lines, it's less confusing to have one point of origin.

 

rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/"whatever I want to call it but I need quotes if there are spaces"/

 

Yep, quotes or escape them with \.

Share this post


Link to post
Share on other sites

I've been watching this thread with interest. I've trolled Google and can't find anywhere that explains the syntax and usage of the --chmod switch. If you could point me towards a page or reference I'd be most grateful.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.