Transferring data from Synology to unRaid

[JoshFink]

Hi, I'm looking for some best practices for transferring data from my Synology to my newly built unRAID server. I need to transfer around 20TB. 

 

Would it be better to mount an UR share in the Synology and just transfer that way or is there a better way? 

 

Thanks

 

Josh

[ashman70]

From another computer I would mount a share for the synology then a share for the unRAID server and copy from the synology share to the unRAID share.

[JoshFink]

6 minutes ago, ashman70 said:

From another computer I would mount a share for the synology then a share for the unRAID server and copy from the synology share to the unRAID share.

Thanks

 

Wouldn't that slow down the transfer as it will then need to proxy through the computer? Maybe I'm mistaken.

[Squid]

2 minutes ago, ashman70 said:

From another computer I would mount a share for the synology then a share for the unRAID server and copy from the synology share to the unRAID share.

While it'll work, it'll be the slowest due to the 2 transfers over the network going on simultaneously.  But, it arguably is the easiest since nothing really needs to be done to get everything to work.

 

58 minutes ago, JoshFink said:

Would it be better to mount an UR share in the Synology and just transfer that way or is there a better way? 

 

What you want to do is do the transfer locally no matter what, and involve the network as little as possible.  If the synology allows you to move / copy files from one share to another without involving another desktop, then it'll work good.  Alternatively, you can mount the Synology share on unRaid using the Unassigned Devices plugin, and then use the Dolphin / Krusader apps to move from the Synology to the unRaid share.

 

The absolute best speed would be to ditch the network together, and physically attach the drives from the synology onto the unRaid server, use UD, Dolphin/Krusader to move everything.   No idea though if the synology drives have separate file systems on each drive, yada yada yada.

[Squid]

Either way, enable reconstruct write mode in Settings - Disk Settings.  All your drives will be spun up, but the writes will go ~3 times faster.  You can disable it after the transfer is done.

[JoshFink]

Just now, Squid said:

While it'll work, it'll be the slowest due to the 2 transfers over the network going on simultaneously.  But, it arguably is the easiest since nothing really needs to be done to get everything to work.

That's what I would have thought as well. Maybe as a last resort. 

 

Just now, Squid said:

 

What you want to do is do the transfer locally no matter what, and involve the network as little as possible.  If the synology allows you to move / copy files from one share to another without involving another desktop, then it'll work good.  Alternatively, you can mount the Synology share on unRaid using the Unassigned Devices plugin, and then use the Dolphin / Krusader apps to move from the Synology to the unRaid share.

I'll try from the Synology to unRaid first and see what the speeds are and then try it the other way around if they're poor. I'm not familiar with the plugins mentioned or the Dolphin/Krusader apps but I'll do some research. Thanks

 

Just now, Squid said:

The absolute best speed would be to ditch the network together, and physically attach the drives from the synology onto the unRaid server, use UD, Dolphin/Krusader to move everything.   No idea though if the synology drives have separate file systems on each drive, yada yada yada.

I don't think that the Synology file system would work that way. However, even if it did I think I would rather leave it up and working so no down time for the transfer. 

We'll see. Some great things to think of and I appreciate the direction.

[JoshFink]

1 minute ago, Squid said:

Either way, enable reconstruct write mode in Settings - Disk Settings.  All your drives will be spun up, but the writes will go ~3 times faster.  You can disable it after the transfer is done.

Awesome. I had no clue about this. Thanks

[Iormangund]

If you have enough free sata/sas connections you could connect the Synology drives to unraid and run xpenology in a VM. Then create a second nic using host only, mount the synology share in unraid (or other way round) and transfer via the host only nic. Found that so far to be the fastest way (no network bottleneck). I'm new to Unraid and migrating 40tb from my xpenology box to it and short of any better alternatives this is what I have been doing. Would add that you need to make sure you use the appropriate boot loader for your current synology version, if you are fully updated to 6.2 (or whatever the latest is) then this way isn't an option.

 

I also attempted mounting the raid array in a Ubuntu docker to take advantage of host share but without using pcie passthrough (current cpu has no vt-d) that didn't work. If you have vt-d you could possibly pass that to ubuntu then mount the raid array in ubuntu and transfer in that using vm share, that would be the very fastest way to do it.

 

Guide for mounting synology in ubuntu: https://www.synology.com/en-uk/knowledgebase/DSM/tutorial/Storage/How_can_I_recover_data_from_my_DiskStation_using_a_PC

Edited June 20, 2017 by Iormangund

[JoshFink]

Just now, Iormangund said:

If you have enough free sata/sas connections you could connect the Synology drives to unraid and run xpenology in a VM. Then create a second nic using host only, mount the synology share in unraid (or other way round) and transfer via the host only nic. Found that so far to be the fastest way (no network bottleneck). I'm new to Unraid and migrating 40tb from my xpenology box to it and short of any better alternatives this is what I have been doing. Would add that you need to make sure you use the appropriate boot loader for your current synology version, if you are fully updated to 6.2 (or whatever the latest is) then this way isn't an option.

 

I also attempted mounting the raid array in a Ubuntu docker to take advantage of host share but without using pcie passthrough (current cpu has no vt-d) that didn't work. If you have vt-d you could possibly pass that to ubuntu then mount the raid array in ubuntu and transfer in that using vm share, that would be the very fastest way to do it.

 

Thanks for the tip. Unfortunately I'm on the latest version (just updated today actually). 

 

Oh well, once preclear is done we'll see how this goes. 

[HellDiverUK]

I would use Unassigned Devices on the unRAID box to mount the shares from the Synology.  Then I'd run Putty on a Windows machine to ssh to the unRAID box and use MC to copy the data across.  That means it's a direct Synology to unRAID connection, the Windows box only controlling the transfer.

 

I use this method quite a lot, and it works great.  With turbo write I pretty much get full gigabit transfers (115-120MB/s).

[Iormangund]

Whatever way you are doing it, it's gonna take a long time. My rough maths show i've got over 2 weeks of transferring left (plus downtime for issues, like having to check parity for 20 hours after unclean shutdown).

 

If you are feeling brave, I would suggest having a go at doing the ubuntu mount and vm host share, will be the fastest way to do it and least likely to risk messing up any of your synology data.

[JoshFink]

Just now, Iormangund said:

Whatever way you are doing it, it's gonna take a long time. My rough maths show i've got over 2 weeks of transferring left (plus downtime for issues, like having to check parity for 20 hours after unclean shutdown).

 

If you are feeling brave, I would suggest having a go at doing the ubuntu mount and vm host share, will be the fastest way to do it and least likely to risk messing up any of your synology data.

Appreciate it. My family would kill me with that much downtime though with all their shows and such. I have time. I'd just like to do it once and make it as efficient as possible. 

 

4 minutes ago, HellDiverUK said:

I would use Unassigned Devices on the unRAID box to mount the shares from the Synology.  Then I'd run Putty on a Windows machine to ssh to the unRAID box and use MC to copy the data across.  That means it's a direct Synology to unRAID connection, the Windows box only controlling the transfer.

 

I use this method quite a lot, and it works great.  With turbo write I pretty much get full gigabit transfers (115-120MB/s).

Thanks.. I'll check this out. I'm not familiar with Midnight Commander but I'll do some research. 

[Iormangund]

Just now, JoshFink said:

Appreciate it. My family would kill me with that much downtime though with all their shows and such. I have time. I'd just like to do it once and make it as efficient as possible. 

 

Thanks.. I'll check this out. I'm not familiar with Midnight Commander but I'll do some research. 

MC is built into unraid. Though personally for transferring that much data I am using rsync, know it adds a little overhead but least you can resume and there is less risk of corrupted files.

[HellDiverUK]

2 weeks seems a very long time.  Last time I transferred 8TB it took about 2 days.  20TB is certainly doable in a week on gigabit.

[HellDiverUK]

Also consider using NFS - less overhead than SMB.

 

MC is a two-pane file transfer program, a bit like xtree Gold in DOS days if you're old enough to remember that.  Very useful program to get familiar with.

Edited June 20, 2017 by HellDiverUK

[Iormangund]

if you enable rsync server on the syno then you can just use a command from synology box like: rsync -havPW --stats /Sourcedir/ rsync://tower/Destinationdir/

[Iormangund]

4 minutes ago, HellDiverUK said:

2 weeks seems a very long time.  Last time I transferred 8TB it took about 2 days.  20TB is certainly doable in a week on gigabit.

I meant 2 weeks for my current migration (gone up now to 16 days with parity check added, sigh). Agree that a week should be fine for 20TB.

Edited June 20, 2017 by Iormangund

[shawnngtq]

It has been 4 years. I am facing the same thing now.

 

Here is some background, my Synology has 4 disks, SHR-2, 10 TB each. I want to transfer all 4 disks to unraid. Was wondering what's the best way to do that without buying any additional disks. The total data is 9 TB, would fit in a single 10 TB disk.

 

I am thinking

1. Remove 1 disk from Synology, shouldn't be a problem for SHR-2. Make it into an unraid data disk. Copy the data from Synology > External hard disk > Unraid data disk

2. Remove 2nd disk from Synology, make it into a unraid parity disk

 

However, the risk in this approach is that now, my Synology has 2 disks (0 backup), my Unraid has 2 disks (1 data, 1 parity building). I think the parity check would take a few days, in this timeframe, both server have 0 backup, if any 1 of the 2 disks failed, I will lose the data. Perhaps I can poweroff Synology with 2 disks, so that the chance of disk dying is lower?

 

Any suggestions?

Edited July 12, 2021 by shawnngtq

[Livio]

I want to replace my Synology DS 211+ with all its data (2x3TB on SHR) as I am no longer able to upgrade to DSM 7.xx. As an alternative to the DS 72x+ with all its worsing hardware constraints and limitations, I am looking for switching to UnRaid.

Any recommendation on how to easy and safely migrate the data to UnRaid these days?

[Vitalsignser]

Can I take my synology drive and mount it in my server using the unassigned drive plugin and copy the data over using krusader?