Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[Squid]

I'm going to talk out of both sides of my mouth here.

 

@NAS  This isn't a bug at all.  Rather its a design choice by @dlandon to have UD conform to industry standard norms on partitioning.  If you want to label something a bug, then the "bug" lies squarely on LT's shoulder's since they are not conforming to industry norms, and unRaid will trash a drive formatted outside of the array if partitioning isn't what it expects (regardless if a parity drive is installed or not)

 

That being said, this does have to get changed.  With UD now being slated for inclusion in the base OS, a reasonable expectation is that the partitioning scheme on a format should match the array so that the drive doesn't get repartitioned.  The better solution however is to have the OS recognize and not repartition anything that may not conform to its standards, and allow the drive to be added to the array (when parity isn't present) without losing all the data on it.

 

 

[NAS]

@Squid to be fair I specifically and intentionally never said it was a bug in UD for three reasons, 1. I didn't know enough to comment, 2. to the user it makes no difference anyway and 3. I really didn't want to start pointing as that only deflects from the issue.... but that plan kinda back fired lol

 

Anyways based on what I know so far I have to agree with the general sentiment that this must change and soon.

 

It is without question a requirement that there should be absolute compatibility between UD formatted disks and unRAID and if a new common ground needs agreed that better fits with norms then it needs to happen soon.

 

But I stand by my assertation that this is a bug, even if it is intentional and the code is sound; we simply cant have a situation where users can make a simple assumption/mistake and lose a disk of data with one button click.

 

Whatever you need from me you have it because this needs to go away before someone gets hurt.

 

 

 

 

[JonathanM]

1 hour ago, NAS said:

Whatever you need from me you have it because this needs to go away before someone gets hurt.

Since @limetech said they are willing and intend to fold UD into the core application, it's an issue that needs their direct involvement. At this point, I see 3 options, if you see other ways to solve it, chime in.

 

1. Change UD's handling of partitions for blank drives, instead of creating a partition automatically at the same time when formatting, offer to create a partition with either unraid or industry standard location. Create a different icon or legend for unraid compatible drives, so that only unraid compliant partitioned and formatted drives show up as array ready. Foreign drives wouldn't have the unraid icon so there is less chance of confusion.

 

This option is a LOT of work, and starts to resemble a poor imitation of gparted customized for unraid. Do we really need a graphical partitioning tool custom written for UD?

 

2. Change unraid's handling of partitions, so that industry standard partitions can be seamlessly mounted. Something in the back of my head is telling me this isn't as easy as it sounds.

 

3. Change unraid's behavior when probing drives to add them to an array so as not to clobber existing partitions that don't match its requirements.

 

Option 3 is somewhat safe, and probably needs to be done anyway at some point. At least the user is presented with a checkbox and a format action to blow away data, instead of automatically having to recreate a partition table if they accidentally assign the wrong drive to an array slot.

[dlandon]

I appreciate all the enthusiasm towards this issue, but there is a bit more drama here than is warranted.  This situation is no different than formatting an XFS disk on another Linux machine, adding data to the disk and then plugging it into unRAID and expecting it to go into the array without a format.

 

LT has committed to fold UD into unRAID and I think it best to not do anything right now that will distract them from that effort.  You all need to appreciate that UD has morphed into something that it was not originally designed for.  It was originally a USB disk mount and data transfer to/from a USB disk.   @gfjardim and I have done the best we can to make UD into what it is and @gfjardim has done a great job coordinating UD and preclear.  Now it's time to let LT handle the incorporation into unRAID and determine the features that make sense for their objectives.

 

UD is pretty complicated and LT will have to spend some time fitting it into unRAID.  It is best to let them decide how to handle the formatting of UD disks and the array format compatibility issue.  I don't want to interfere with their effort and make any changes at this point.

[JonathanM]

1 hour ago, dlandon said:

 This situation is no different than formatting an XFS disk on another Linux machine, adding data to the disk and then plugging it into unRAID and expecting it to go into the array without a format.

Yes, but one of the main selling points of unraid is that the reverse is fine. Data disks formatted with unraid work fine in another linux box, it's counter intuitive to not be able to go the other way. Not saying you need to do anything about it, quite the opposite. It's limetech's baby now, let them deal with it.

[NAS]

7 hours ago, jonathanm said:

2. Change unraid's handling of partitions, so that industry standard partitions can be seamlessly mounted. Something in the back of my head is telling me this isn't as easy as it sounds.

 

Agree. This would be ideal but it is disingenuous of me to assume that these wasnt a good reason that this scheme was used in the first place and the same complications still exist today.

 

7 hours ago, jonathanm said:

3. Change unraid's behavior when probing drives to add them to an array so as not to clobber existing partitions that don't match its requirements.

 

I think this is the one change that needs to happen soon. My focus on this topic is driven by a strong potential of user data loss and whilst there is an inelegance to users having two types of almost identical disks that are not fully compatible that is an issue that can be addressed in the longer term (much as it irks me about all the time I have wasted moving data to standardize on XFS disks only to now find they are still non standard)

 

4 hours ago, jonathanm said:

Yes, but one of the main selling points of unraid is that the reverse is fine. Data disks formatted with unraid work fine in another linux box, it's counter intuitive to not be able to go the other way. Not saying you need to do anything about it, quite the opposite. It's limetech's baby now, let them deal with it.

 

I agree here as well and this wording puts a finger on one point I was struggling to make. Also it is important to accept that there is a real difference between the mindset of a user bringing a disk from elsewhere (importing) and one plugging in a disk created on unRAID (native). The user that has stayed native during the whole story may not even think to consider that there may be issues and that is where things can go badly wrong.

 

 

[ken-ji]

8 hours ago, jonathanm said:

2. Change unraid's handling of partitions, so that industry standard partitions can be seamlessly mounted. Something in the back of my head is telling me this isn't as easy as it sounds.

 

I'm curious - which unraid partitioning scheme are we referring to?

the "old" start at 63 MBR?

the "new" start at 64 MBR?

 

(AFAIK gpt always starts at 64)

 

[unevent]

5 hours ago, ken-ji said:

 

I'm curious - which unraid partitioning scheme are we referring to?

the "old" start at 63 MBR?

the "new" start at 64 MBR?

 

(AFAIK gpt always starts at 64)

 

 

Default settings of some newer partition tools start at the 'Windows' 2048 (1MiB).  Some believe it matters to have the start sector divisible by 512 vs 8 and the 1MiB start also leaves room for bootloaders which unRAID has no use for.  IIRC, unRAID has used 64 (32KiB) since 2TB drives (512-byte sector and 512e 4Kn drives).

[JorgeB]

unRAID uses 64 as starting sector for all 4k aligned partitions, and because of how parity currently works I believe it won't be easy to change.

[TODDLT]

On 7/21/2017 at 6:33 AM, Squid said:

Anything executed in the "go" file is run before the array is started.  Mounting the drives is going to wind up spinning up the drives.  While I can't comment on why the drives won't spin down when mounted with UD, what you're going to want to do is either:

• Run your script with the user.scripts plugin to do the same thing.

 

I was able to set up the script in user.scrips using the same commands I had in my go script.  In doing so I figured out what at least part of my trouble was.

The command lines were 1st spinning down the drive, and then 2nd setting the spin down time frame used in case it ever got spun up for any reason.  The problem is line 2 was spinning the drive back up.  Now why it would not honor the spin down time of 10 minutes I don't know.

 

That being said, once I reversed the command lines in user.scripts, it all worked fine.  Thanks for the help.

 

This turns out not to be a UD issue or solution, so sorry for cluttering the page unnecessarily.  

Edited July 28, 2017 by TODDLT

[ken-ji]

Hey got a USB flash drive giving this in the UI. Its a Samsung Flash Drive Fit 128GB. exFAT formatted,  (I manually mounted it in /tmp/usb as UD was going to try to mount it to /) 

 

Any logs or command outputs you need?

UD's log says:

Quote

Jul 30 09:50:52 Disk with serial 'Flash_Drive_FIT_0374616040001351' is not set to auto mount and will not be mounted...

 

 

Tools - System Devices has this:

Quote

[0:0:0:0]    disk    Sandisk  USB Ultra        1100  /dev/sdb   8.05GB
[1:0:0:0]    disk    Samsung  Flash Drive FIT  1100  /dev/sda    128GB
[2:1:0:0]    disk    ATA      WDC WD30EZRX-00M 0A80  /dev/sdc   3.00TB
[2:2:0:0]    disk    ATA      WDC WD40EZRX-00S 0A80  /dev/sdd   4.00TB
[2:3:0:0]    disk    ATA      SAMSUNG HD103SI  1113  /dev/sde   1.00TB
[6:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdf   4.00TB
[6:0:1:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdg   4.00TB

 

Edited July 30, 2017 by ken-ji

[ufo56]

Is there any fix for this ? Its annoying to do hard power off over ipmi by turning power on or off because unraid can't to restart after that... Sometimes 10 times a day i have to do that.

Every time (almost)  when i open "Main" tab i have to restart my unraid box because everything hangs on here. Dockers stop's responding etc. Using only one windows share mount. Win 8.1.

[CHBMB]

@dlandon already answered this one @ufo56 and thought it was a problem on the client not UD.

[ufo56]

what client ? Problem on win 8 side ?

[CHBMB]

On 04/05/2017 at 6:08 PM, dlandon said:

That appears to be a SMB or network issue with the client computer.

 

@ufo56 

[dlandon]

7 hours ago, ken-ji said:

Hey got a USB flash drive giving this in the UI. Its a Samsung Flash Drive Fit 128GB. exFAT formatted,  (I manually mounted it in /tmp/usb as UD was going to try to mount it to /) 

 

Any logs or command outputs you need?

UD's log says:

 

Tools - System Devices has this:

 

Do you have the prelcear plugin installed?

[dlandon]

1 hour ago, ufo56 said:

Is there any fix for this ? Its annoying to do hard power off over ipmi by turning power on or off because unraid can't to restart after that... Sometimes 10 times a day i have to do that.

Every time (almost)  when i open "Main" tab i have to restart my unraid box because everything hangs on here. Dockers stop's responding etc. Using only one windows share mount. Win 8.1.

Is it a remote SMB mount?

[ken-ji]

10 hours ago, dlandon said:

Do you have the prelcear plugin installed?

Yep. Unfortunately I've already left the server behind for the week (This is at my relative's place and I have access during the weekends)

 

I can try to see if it can be replicated on my server.

 

[dlandon]

1 hour ago, ken-ji said:

Yep. Unfortunately I've already left the server behind for the week (This is at my relative's place and I have access during the weekends)

 

I can try to see if it can be replicated on my server.

 

Remove the preclear plugin and see if the problem persists.  Preclear runs a background daemon that UD uses to determine the disk information.  I suspect that preclear is getting it wrong.

[ufo56]

On 7/30/2017 at 2:53 PM, dlandon said:

Is it a remote SMB mount?

On same lan machine.

[dlandon]

5 hours ago, ufo56 said:

On same lan machine.

The SMB share is probably dropping off-line causing the hang.

[Squid]

16 minutes ago, dlandon said:

The SMB share is probably dropping off-line causing the hang.

Is it possible somehow to introduce timeouts or something.  I had the same thing happen when my internal network was messed up, but never bothered to ask about it.

[dlandon]

3 hours ago, Squid said:

Is it possible somehow to introduce timeouts or something.  I had the same thing happen when my internal network was messed up, but never bothered to ask about it.

I did a little research on this problem and it is a Linux design issue.  For example to find out the size of a mounted device, UD uses df on the mount point.  If df is run on a mounted device that is off-line it will hang (this is by design) waiting for the device to come back on-line.  I did what I could for the SMB shares by not doing things like a df on it if UD sees that it is off-line.  It would be very unwieldy to try to cover all the cases, and I don't have any interest in doing anything further with LT saying they will fold UD into unRAID.

 

My suggestion is to not mount SMB shares in unRAID for extended time periods, and if you do be sure the mount stays on-line.  Why mount an SMB share in unRAID to then just share it again as an SMB share?  If you are sharing SMB shares with NFS on unRAID, I'd suggest another answer.

[Squid]

I did a little research on this problem and it is a Linux design issue.  For example to find out the size of a mounted device, UD uses df on the mount point.  If df is run on a mounted device that is off-line it will hang (this is by design) waiting for the device to come back on-line.  I did what I could for the SMB shares by not doing things like a df on it if UD sees that it is off-line.  It would be very unwieldy to try to cover all the cases, and I don't have any interest in doing anything further with LT saying they will fold UD into unRAID.
 
My suggestion is to not mount SMB shares in unRAID for extended time periods, and if you do be sure the mount stays on-line.  Why mount an SMB share in unRAID to then just share it again as an SMB share?  If you are sharing SMB shares with NFS on unRAID, I'd suggest another answer.

Fair enough. My use case for smb mounts is to mount shares from a secondary server so that my docker apps can access the files

Sent from my LG-D852 using Tapatalk

[Videodr0me]

The last couple of versions of the unassinged devices plugin seem to forget the auto-share options. I rebooted server 3 to 4 times in the past 2 month and every time it forgot the mount and share settings (curiously not of all drives). As these were mainly system updates I did not think much of it, but it also happened after a normal reboot. After revisiting the unassinged devices tab and (again) enabling the mounts everything works fine. Is this normal or is something strange going on?