RFC: 6.2-rc2 experimental shfs hard-link support & appdata share discussion

[saarg]

Q1. When will the container specific images be mounted?

Q1a. Will they only be mounted when the Docker container is running?

Q1b. Will they be unmounted when the Docker container is stopped?

 

If the answer to Q1a and Q1b are both yes, then the system created quite a pickle. Imagine the file the user needs to edit can only be edited when the application (inside the Docker Container) is NOT running.

 

Maybe squid will make a plugin for exactly that 

[BobPhoenix]

Q1. When will the container specific images be mounted?

Q1a. Will they only be mounted when the Docker container is running?

Q1b. Will they be unmounted when the Docker container is stopped?

 

If the answer to Q1a and Q1b are both yes, then the system created quite a pickle. Imagine the file the user needs to edit can only be edited when the application (inside the Docker Container) is NOT running.

 

Which is the case for me.  I need to edit the properties files with the docker NOT running or what I change will just get replaced when the docker is shut down.  And the docker reads the properties file into memory when it first boots and doesn't reference the file again until the next boot.

[BRiT]

Q2. Will the user have a choice as to which filesystem to use inside the container specific images? Some users do not want to deal with BTRFS in any shape or form, and is why their cache drive is configured as XFS.

[RobJ]

I don't think this was presented as an all or nothing choice.  It is currently an option only, for more technical users.  Otherwise, everything behaves the same way it always did.  If it's going to be too much trouble or you don't need it, then don't turn it on.

 

Those who package apps that *require* it may need to adapt them for it, and/or add special instructions.  It might be good to hear from them, as to what specific things they would need in the implementation.

[BRiT]

RobJ, right, but my question is if you decide to use the container specific loopback image, can the user control what filesystem it uses internally?

[bobbintb]

Testing the hard linking and sometimes I am getting this error:

 

ln /mnt/user/downloads/test.mkv /mnt/user/downloads/test/new.mkv    
ln: failed to create hard link '/mnt/user/downloads/test/new.mkv' => '/mnt/user/downloads/test.mkv':
No such file or directory

 

The source is there and I can link it to other places. I can't find a commonality. Seems to just be that one folder named "test". Persmissions maybe? I gave it 777 permissions.

[daryl]

This is very exciting.  The lack of hard link support is one thing that has really been annoying me of late.

 

My use case is different to the one proposed.  I want to use unraid for incremental backups (like time machine) with things like rsnapshot which use hard links as part of the backup process.

 

I do not mind if I cannot use a cache drive with the share that I use for backups, but native hard links are a must. Please don't remove support   I don't think that using loopback images is a viable solution for this application.

 

Daryl

 

[Fireball3]

I second Daryl, along with some other users in this thread.

https://lime-technology.com/forum/index.php?topic=50078.0

 

Please keep the hardlinks working on user shares!

[tr0910]

I second Daryl, along with some other users in this thread.

https://lime-technology.com/forum/index.php?topic=50078.0

 

Please keep the hardlinks working on user shares!

I have been using hard links from v5 for years with rsnapshot.  But I have only needed to use disk links not user shares.

 

My documents share is limited to one disk and is about 500gb. It is snapshotted hourly to another disk. 

[limetech]

Testing the hard linking and sometimes I am getting this error:

 

ln /mnt/user/downloads/test.mkv /mnt/user/downloads/test/new.mkv    
ln: failed to create hard link '/mnt/user/downloads/test/new.mkv' => '/mnt/user/downloads/test.mkv':
No such file or directory

 

The source is there and I can link it to other places. I can't find a commonality. Seems to just be that one folder named "test". Persmissions maybe? I gave it 777 permissions.

 

Yeah that's a bug... fixed in next release.

[limetech]

Please keep the hardlinks working on user shares!

 

Well we decided that loopback images for containers is not a realistic option at this point.  So for -rc3 we went ahead and fixed up the 'mover' to preserve hard links when a share is moved between cache/array or array/cache.

[John_M]

Please keep the hardlinks working on user shares!

 

Well we decided that loopback images for containers is not a realistic option at this point.  So for -rc3 we went ahead and fixed up the 'mover' to preserve hard links when a share is moved between cache/array or array/cache.

 

Is there any chance you could fix the mover to work with symlinks too, please? At the moment they are ignored and left on the cache disk. That hasn't been a major problem because they are tiny and I can move them manually, but it would be nice if it could be fixed.

 

[bobbintb]

Testing the hard linking and sometimes I am getting this error:

 

ln /mnt/user/downloads/test.mkv /mnt/user/downloads/test/new.mkv    
ln: failed to create hard link '/mnt/user/downloads/test/new.mkv' => '/mnt/user/downloads/test.mkv':
No such file or directory

 

The source is there and I can link it to other places. I can't find a commonality. Seems to just be that one folder named "test". Persmissions maybe? I gave it 777 permissions.

 

Yeah that's a bug... fixed in next release.

 

Ah, good to hear, thanks. Keep up the good work. The developmental turn around time has greatly improved since the v5 days.

[limetech]

Is there any chance you could fix the mover to work with symlinks too, please? At the moment they are ignored and left on the cache disk. That hasn't been a major problem because they are tiny and I can move them manually, but it would be nice if it could be fixed.

 

Done.

 

[John_M]

Is there any chance you could fix the mover to work with symlinks too, please? At the moment they are ignored and left on the cache disk. That hasn't been a major problem because they are tiny and I can move them manually, but it would be nice if it could be fixed.

 

Done.

 

Fantastic! Thank you so much.

 

[jumperalex]

[gulp], that is a ... well ... big feature add in the RC phase isn't it? Enough time to test for corner cases or is it seemingly straight forward?

 

My ignorance makes me ask the easy questions so please be gentle

[daryl]

I second Daryl, along with some other users in this thread.

https://lime-technology.com/forum/index.php?topic=50078.0

 

Please keep the hardlinks working on user shares!

I have been using hard links from v5 for years with rsnapshot.  But I have only needed to use disk links not user shares.

 

My documents share is limited to one disk and is about 500gb. It is snapshotted hourly to another disk.

 

That doesn't work very well when you want to backup one unraid box to another unraid box

[limetech]

We decided to add "proper" hard link support at this time rather than assign a bunch of loopbacks.  Changes are in 6.2.0-rc3.

[Squid]

We decided to add "proper" hard link support at this time rather than assign a bunch of loopbacks.  Changes are in 6.2.0-rc3.

Is it safe to say that this is still experimental?

[limetech]

We decided to add "proper" hard link support at this time rather than assign a bunch of loopbacks.  Changes are in 6.2.0-rc3.

Is it safe to say that this is still experimental?

It should work.

[Squid]

We decided to add "proper" hard link support at this time rather than assign a bunch of loopbacks.  Changes are in 6.2.0-rc3.

Is it safe to say that this is still experimental?

It should work.

I'll do a test build of CA backup tonight allowing user shares as a destination and see how it handles it and let you know

[Ignitenz]

Do we still need to do the changes from Page 1 to get hardlinks working in rc3?

[bnevets27]

So now that this was listed in the 6.2 release as a feature is it safe to assume this is no longer experimental? Will this continue to stay as a feature in unRAID or is there a possibility of it being removed? It would be nice to have a definitive answer to this from limetech including an answer to Ignitenz post.

 

Do we still need to do the changes from Page 1 to get hardlinks working in rc3?

[limetech]

So now that this was listed in the 6.2 release as a feature is it safe to assume this is no longer experimental? Will this continue to stay as a feature in unRAID or is there a possibility of it being removed? It would be nice to have a definitive answer to this from limetech including an answer to Ignitenz post.

 

Do we still need to do the changes from Page 1 to get hardlinks working in rc3?

Good points.  Yes hard links are now supported.

[Squid]

So now that this was listed in the 6.2 release as a feature is it safe to assume this is no longer experimental? Will this continue to stay as a feature in unRAID or is there a possibility of it being removed? It would be nice to have a definitive answer to this from limetech including an answer to Ignitenz post.

 

Do we still need to do the changes from Page 1 to get hardlinks working in rc3?

Good points.  Yes hard links are now supported.

Side question while I'm at work and not really able to investigate fully...  What is the maximum # of hardlinks per file supported  (Right now I'm testing with ~10,000 hardlinks per file with no problems, but in the next day or so I want to do a test with ~1,000,000 hardlinks per file)...  My quick research seemed to indicate that it would max out at the number of inodes available that the fs was mounted with...