Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)


Recommended Posts

4 hours ago, trurl said:

Not sure what you are referring to here. Not a lot has changed about User Shares from 4.7 except some of the Use Cache settings. Were you not using User Shares before?

Trurl:

I was just reading Johnathanm comment regarding legacy issues. I had two Shares that was ans is set up one for music (P) and one for movies (S). I did this when 4.7 was set up and I initially had a few small drives. 

9 minutes ago, RobJ said:

I'll add more words to step 16 to display the message you saw ("All data on the parity drive will be erased when array is started"), then tell you to ignore it and click the checkbox to indicate "Parity is already valid". 

 

RobJ: I was not trying to be critical. I followed that step and I took the information literally. I am determined to learn all the ins and outs of unRaid and Linux over the coming years. I remember reading the Unix command descriptions (in the 80's while working with AT&T haven't used them 25 years) that were cryptic at best for a new user. When an event or message appears that I do not expect, nor a caveat what to do if such and such happens, then my first question is what did I screw up in the process.  If it appears that I did things correctly and now have some additional steps (fumbling around trying things) then it appears to be convoluted. I think the question for the process describe is does parity need to be maintained or not? I used this for media NAS for the last 6 years under 4.7.  I see that what you have been creating while I sat quite happy with my system cranking along without any hiccups until I needed to increase the size of my disk drives.  Back to the issue:

If parity needs to be maintained then follow method A:

If parity does not need to be maintained then method B:

Please remember when learning a new system it is the unexpected items that pop up causing end users to do stupid things. 

 

If you it would benefit you and other users I will gladly help with trying things and help finding items that may need further explanation or a reference to a detailed description.

It kind of like that old TV commercial "Let Mikey eat it he hates everything" suddenly Mikey is eating the cereal.

I could benefit greatly by becoming more knowledgeable and an asset to the forum and hopefully reduce some of the misunderstandings by users that create a lot of questions.

 

4 hours ago, jonathanm said:

Reference to use /mnt/user (shares only) instead of /mnt/diskX (individual disks) is an ongoing thing since the "user share copy bug" became a better known issue. If you know how unraid generates user shares and understand why copying between /mnt/diskX/share to /mnt/user/share can cause data loss, it's easy to avoid. For less savvy users, it's easier just to tell them to ignore the individual disks and let unraid work its magic with user shares.

 

As far as what you may have missed in the time between 4.7 and 6.3.2, there is no way for me to know what you have and haven't learned so far. So much has changed with the addition of notifications, dockers and VM's.

 

One major point is that plugins are less and less supported for applications, pretty much if you can do the task with a docker, you shouldn't be using a plugin.

 

OK, and I thought that the plugins were the new thing too. I think that the steps I need to do is first finish straightening out my data, make sure my basic NAS sytem is working with JRiver media center, (I would like to talk with you during this process to be sure I get everything setup to step into the brave new world of unRaid) then upgrade my mobo, cpu and ram. At least I will know what kind of an upgrade I will need to do.  I have looked into the new AMD Ryzen (low end) CPU that uses 65 watts, Asus mobo and DDR4 RAM 16MB and it turns out to be surprisingly affordable. I actually will not need to do some of the other hardware upgrades like power supply, high end cpu cooler etc. that brings the cost in-line with a FX-8350 (125 watts) based rig. Either way I will be out of pocket about $600 but If I wait a few months there may be some new less expensive options that may provide a more cost effective solution.  

Link to comment

No doubt I am somewhat repeating what has already been said, but sometimes different ways of saying the same thing can clarify.

2 hours ago, tunetyme said:

I think the question for the process describe is does parity need to be maintained or not?

The process in the wiki does maintain parity. The only thing that might make it seem that it doesn't is that message you get after New Config. That is simply because New Config tells unRAID to forget about the disk assignments and whether parity is valid or not. It would be better if the message went away after you check the box, though.

 

It would be possible to remove parity before beginning as I think RobJ mentioned, and that would speed up some of the disk writing, but then parity would need to be rebuilt at the end, which would take time, and you would be unprotected until it was rebuilt.

 

2 hours ago, tunetyme said:

OK, and I thought that the plugins were the new thing too.

Yes, plugins were introduced in V5. Before that we just had packages with no "official" way to manage them in the webUI. unMenu did have PKG Manager which helped with this but not many developers did any development with it.

 

The problem with plugins and packages was that they were not isolated from the core system, or indeed from each other, and so there could be conflicts which produced serious issues.

 

The recommended approach in V6, and a great improvement in many ways, is to use dockers unless your addin needs to modify the webUI or work directly with the core system in some way. Not only are dockers isolated from the core and each other, but they are easier to develop, as is shown by the incredible number of applications you can now run on your server. You can think of dockers as sort of minimalist VMs with only a single application installed in them.

Link to comment

As I have said, I have had my head in the sand for the last 6 years with the attitude that if it ain't broke don't fix it. (I won't make that mistake again). I am aware that I have just enough knowledge to do some real damage to my server. In hindsight, I think I should have asked for advice at the beginning of this process of upgrading and converting drives. I didn't realize that I needed help until I had finished installing the new drives. The replacement process went well with a few bumps in the road. I have 8 drives and I have replaced 6 of the 8. I replaced the Parity and cache first as they both require opening the server case and I could move both drives into the 2 open Icy Dock 4 in 3 slots. Cache was there but was blank but I keep it in the overall configuration. I was able to verify parity, so I unassigned the original parity drive rebuilt parity on the new 4TB drive, then formated the cache drive. Parity was maintained. The cache drive formated btrfs as the default while the global settings were set to xfs, so I assumed that was correct.

Copy method was as follows:

Parity 2TB to 4TB new

Cache 1TB to 1TB new btrfs

disk1 2Tb to 2TB new xfs (mv command)

disk2 2TB to 4TB new xfs rsync

disk3 2TB rfs to xfs 

disk4 2TB to 4TB new xfs (rebuild)

disk5 2 TB rfs to xfs

disk6 2Tb to 2TB new xfs (mv command)

 

My logic was to do the parity and cache first, then the 2 2TB NAS drive (disk 1 and 6). (these drives won't exceed 2 TB of data for several more years) the data on the drives is organized by genre and disk 6 was an overflow for the other 5. I have all drives set to fillup each drive before going to the overflow. It has worked well and this allows me to do NTFS backups by genre on a regular basis. This make it easy to find something on a backup as I can look at one disk not 6 to find anything. The process was to do the data drives requiring the least amount of time first then the big drives. Converting the rfs drives to xfs drives was an after thought. The events are as follows:

the 2 2TB drives I used the mv command from the console. This went smoothly.

the first 4TB drive (disk 2) I used the rsync command per the wiki this went smoothly Parity was maintained.

This is were the mistakes began on my part. I thought that perhaps it would be faster if I allow the system to rebuild a drive from parity instead of copying and this would be faster than rsync. No speed improvements but the new drive was now rfs instead of xfs. I understand why. At the time I became frustrated because I had erroneously reasoned that parity only tracked the actual data. As a result, I started down the path of preclearing one of the 2TB drives and formating it to xfs then copying the data using rsync then I may as well do the other 2 drives. 

 

I have repeatedly tried to just format the drives (still had data on them) to xfs. I spent hours trying to get the format button to pop up. So, I reasoned that I needed to preclear the drives to be able to format them. I am not really upset about this because it tells me that the drives are still in good condition and in my mind it was questionable before. As I write this the 4TB and the 2 TB drives are completing the preclear process. I thought that I was going to have this all done last week (actually mid February). I started this process back in early January with upgrading software and here we are looking for completion of this stage in early April (I hope). In terms of time I think I would have been much better off replacing all the drives one at a time.

 

7 hours ago, trurl said:

The process in the wiki does maintain parity. The only thing that might make it seem that it doesn't is that message you get after New Config. That is simply because New Config tells unRAID to forget about the disk assignments and whether parity is valid or not. It would be better if the message went away after you check the box, though.

   

This is what really confused me and caused my frustration. How would a user know to tell the system that parity is correct? I would and will always error to the system knows if it is correct or not. When I am trying something new and it doesn't turn out the way expected then I go back to the things I know work. I do think that this could be fixed.

 

As for plugins and dockers I will need to upgrade the rest of my system before I can leave the stone age and strive to deal with future shock.

 

I want to thank everyone for their patience in dealing with me and I really appreciate your help.

 

Barry

 

 

Link to comment

Share disk3 is identically named to a disk share

This is usually caused by moving the contents of one disk to another (XFS Conversion?) and an improperly placed slash. The solution is to move the contents of the user share named disk3 to be placed within a validly named share. Ask for assistance on the forums for guidance on doing this.

 

 

How can I correct this?

Link to comment
2 hours ago, tunetyme said:

Share disk3 is identically named to a disk share

This is usually caused by moving the contents of one disk to another (XFS Conversion?) and an improperly placed slash. The solution is to move the contents of the user share named disk3 to be placed within a validly named share. Ask for assistance on the forums for guidance on doing this.

 

 

How can I correct this?

One of your disks has a top level folder named disk3 that you accidentally created when trying to move the contents of disk3 to another disk. That top level folder is an "accidental" user share named disk3 because any top level folder on cache or array disks is a user share named for the folder.

 

Under that disk3 folder will be the folders of the "intentional" user shares (the contents from disk3) that you were trying to move from disk3 to that disk.

 

You just need to move those folders up one level so they are top level folders on the disk (which will then make them part of the user shares named for those folders).

 

Then when the disk3 folder is empty you can delete it from that disk.

 

Midnight Commander is a builtin file manager that is easy to learn, just type mc at the command line. I know it was available in unRAID 4.7 so maybe you are familiar with it already. Do you know how to use Midnight Commander?

Link to comment

tunetyme, would you mind checking the change I've made?  Here's the old Step 16:

  • You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box for Parity is already valid; IMPORTANT! click the check box, make sure it's checked to indicate that Parity is already valid or your Parity disk will be rebuilt! then click the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt

And here's the new version:

  • You should see all array disks with blue icons, and a warning (All data on the parity drive will be erased when array is started), and a check box for Parity is already valid. VERY IMPORTANT! Click the check box! Make sure that it's checked to indicate that Parity is already valid or your Parity disk will be rebuilt! Then click the Start button to start the array. It should start up without issue (and without erasing and rebuilding parity), and look almost identical to what it looked like before the swap, with no parity check needed. However the XFS disk is now online and its files are now being shared as they normally would. Check it all if you are in doubt.
    •  Before you click the Start button, you may still see the warning about the parity drive being erased, but if you have put a check mark in the checkbox for Parity is already valid, then the parity drive will NOT be erased or touched. It is considered to be already fully valid.
Link to comment

RobJ

 

1 hour ago, RobJ said:

And here's the new version:

  • You should see all array disks with blue icons, and a warning (All data on the parity drive will be erased when array is started), and a check box for Parity is already valid. VERY IMPORTANT! Click the check box! Make sure that it's checked to indicate that Parity is already valid or your Parity disk will be rebuilt! Then click the Start button to start the array. It should start up without issue (and without erasing and rebuilding parity), and look almost identical to what it looked like before the swap, with no parity check needed. However the XFS disk is now online and its files are now being shared as they normally would. Check it all if you are in doubt.
    •  Before you click the Start button, you may still see the warning about the parity drive being erased, but if you have put a check mark in the checkbox for Parity is already valid, then the parity drive will NOT be erased or touched. It is considered to be already fully valid.

 

I might add something at the end that would reassure the user if they don't click the check box.

 

"If you forget to click the Parity already valid and start the system it will rebuild parity and it will take awhile but no harm is done."

 

It is difficult for me as a user to have the confidence that everything on the new 4TB drive is identical to the old 2TB drive with a different format. So I wouldn't click that button unless I was absolutely sure that parity is valid. For me, if I could just verify that parity is correct one time then I could use this method with confidence.

 

In new users minds a command is absolute, get it wrong and you're screwed. All this changes over time after you use commands and see how they actually work.  One gets over their fears of mass destruction.  I know I have hand ripped well over 5,000 CD most required me to type in all the song info. So when I lost that hard disk and all the data I was upset to say the least. (Hence, I keep 2 backups) Now it's a breeze compared to what I went through 20 years ago.) I am willing to try any command to learn about it and how it works on my data as long as I have the confidence that I am not going to destroy the data and I know the consequences if I get it wrong and how to fix it. From what I've read on the forum many questions are the result of some action and a surprise result then the help! question shows up on the forum. 

 

Let me know if this is helpful.  As I have said I would be happy to try to help with the Wiki if you think this kind of perspective would be helpful.

 

 

Link to comment
8 minutes ago, tunetyme said:

The rsync command I used was

rsync -avPX /mnt/disk3 /mnt/disk7

 

That should be :  rsync -avPX /mnt/disk3/ /mnt/disk7

Note the slash after the 3.  Without that slash, you will end up with a disk3 folder on Disk 7 (/mnt/disk7/disk3).  With the slash added, you will end up with the entire contents of Disk 3 on Disk 7, and no disk3 folder.

Edited by RobJ
add sentence
Link to comment
14 minutes ago, tunetyme said:

I might add something at the end that would reassure the user if they don't click the check box.

 

"If you forget to click the Parity already valid and start the system it will rebuild parity and it will take awhile but no harm is done."

 

Done.

 

18 minutes ago, tunetyme said:

It is difficult for me as a user to have the confidence that everything on the new 4TB drive is identical to the old 2TB drive with a different format. So I wouldn't click that button unless I was absolutely sure that parity is valid. For me, if I could just verify that parity is correct one time then I could use this method with confidence.

 

That's why my very first step is to recommend a parity check, so that you know there are no drive problems to take care of, and that parity is good.  There's no reason it should not stay good throughout.

 

19 minutes ago, tunetyme said:

Let me know if this is helpful.  As I have said I would be happy to try to help with the Wiki if you think this kind of perspective would be helpful.

 

Keep it coming!   ;)

 

I have also added a summary of the method to begin it with, and a new Method section, with the various factors that are involved and comparative verbiage between the different methods. The methods are only summarized.  Will it be helpful?  Probably not, so many more words added...

Link to comment
1 hour ago, tunetyme said:

It is difficult for me as a user to have the confidence that everything on the new 4TB drive is identical to the old 2TB drive with a different format. So I wouldn't click that button unless I was absolutely sure that parity is valid. For me, if I could just verify that parity is correct one time then I could use this method with confidence.

I've probably said it before, probably even within the last few pages of this thread, but I thought I would explain why parity remains valid for anybody that might be following along.

 

unRAID parity is realtime. Any time there is a write to any disk in the array, the corresponding parity is recalculated and written at that time. All the moving/copying you do is just writing to the disks, which updates parity.

 

It's obvious that writing a file to a disk is writing, but deleting a file (such as when it is moved to another disk) is really just another write operation. Deleting a file writes the filesystem data that keeps track of files. As you are probably aware, deleting a file doesn't really erase anything, it's just that the filesystem no longer references the data.

 

The final piece of this puzzle is that formatting is also a write operation that updates parity. It writes an empty filesystem to the disk. Nothing is really erased (same as when deleting), just filesystem data is written. Format is faster than deleting all the folders/files though, and of course, the whole point of this exercise is to format to a different filesystem.

 

Link to comment
1 hour ago, RobJ said:

That's why my very first step is to recommend a parity check, so that you know there are no drive problems to take care of, and that parity is good.  There's no reason it should not stay good throughout.

 

I think you missed my point...The fact that you have two different size drives and different format the question in my mind is how can parity be valid when I swap out a 2TB RFS drive with 4TB xfs drive. I know it works out now but at the time it was a strech to comprehend this.

Link to comment
23 minutes ago, trurl said:

unRAID parity is realtime. Any time there is a write to any disk in the array, the corresponding parity is recalculated and written at that time. All the moving/copying you do is just writing to the disks, which updates parity.

 I think most users think parity is static in other words when I think of data stored on my drive it is 8 bits which is a group of ones and zeros These are at a very particular location on my drive when I make a copy it does not necessarily land at that exact same spot on the new drive it could be in any other sector. So without testing and seeing that is at the same exact location then there is a leap of faith that you are asking for that I have a hard time making. That's why I am suggesting that there is a method of doing a verify option to prove to the new user that it in fact works.

Edited by tunetyme
Link to comment
1 minute ago, tunetyme said:

 I think most users think parity is static in other words when I think of data stored on my drive it is 8 bits which is a group of ones and zeros These are at a very particular location on my drive when I make a copy it does not necessarily land at that exact same spot on the new drive it could be in any other sector. So without testing and seeing that is at the same exact location then there is a leap of faith that you are asking for that I have a hard time making. That's why I am suggesting that there is a method of doing a verify option to prove to the new userr that it in fact works.

It doesn't matter whether or not something lands in that exact same spot. In fact it is extremely unlikely that it will. And nothing about any of this requires anything to start or end up at any particular spot.

 

Parity is valid at the beginning of all this. Before anything is written, any particular spot is already reflected in parity.

 

If a particular spot gets written, parity is updated at the corresponding spot.

 

If a particular spot is never written, but somehow the filesystem comes to no longer reference it, such as the file that contained it is deleted, or the disk is formatted, again parity is updated, But remember, everything is a write. Parity is not updated at the spot that didn't get written, it is updated at the spot that corresponds to the filesystem data that was written to perform the delete or format.

 

So, parity is valid to start, and everything that happens is just a write that updates the corresponding parity, so parity remains valid throughout.

Link to comment
56 minutes ago, tunetyme said:
2 hours ago, RobJ said:

That's why my very first step is to recommend a parity check, so that you know there are no drive problems to take care of, and that parity is good.  There's no reason it should not stay good throughout.

 

I think you missed my point...The fact that you have two different size drives and different format the question in my mind is how can parity be valid when I swap out a 2TB RFS drive with 4TB xfs drive. I know it works out now but at the time it was a strech to comprehend this.

 

From a parity standpoint, size doesn't matter, format doesn't matter, data doesn't matter, nothing matters but the bits on every drive, whether you are using them or not.  From a parity standpoint, drives are all the same size, exactly as big as the parity drive.  They just have zeroes past the end of the physical drive.

 

Here are links explaining parity (the second has more links):

   Parity-Protected Array, from the Manual

   How does parity work?

Link to comment
12 minutes ago, tunetyme said:

Those of us who are old school think more along the lines of a parity bit used in each byte (9th bit) or data communications.

That's still the case, but instead of calculating a parity bit for a serial sequence of 8 bits along the drive, think instead of adding up all of the bits at a specific address on all the drives, and putting the resulting parity bit at that same address on the parity drive. That way you can always recreate any single bit on a specific drive by using all the other drives plus the parity drive.

 

The math is still the same as it ever was for parity calculation, just with an arbitrary number of drives in a column instead of a specific number of bits in a row.

Link to comment

Are you saying when you use rsync and copy a file drive one (2TB rfs to 4TB xfs) to drive two that each bit ends up at the exact same address?

 

If so, that's amazing and who ever wrote that command didn't have a life for quit awhile.

 

Again, if you are old school like me then "show me". That's why If I do verify parity before starting the process and then substitute the 4TB drive for the 2TB drive in the array I would want to see that parity is still valid. If I do it one time then I will trust it. Remember I started back in the day of punch cards,then Decwriters (tons of greenbar) and finally CRT's.  We were taught to be skeptical...GIGO (garbage in garbage out). I remember when they were trying to get the code rock solid for basic things like a keyboard. It may be that kids these days don't have to contend with this anymore but this was my training.

 

Still think the step 11 when newconfig turns format type back to auto is a mistake. If I unassign the drive, I can quickly see the format type so I am confident the info is there. Auto was most likely a short cut.

Edited by tunetyme
Link to comment
3 minutes ago, tunetyme said:

Any ideas on correcting the disk3 share?  

It shows up in shares but I can not find it on any disk in my system.

It must be there somewhere!!   You also need to check the cache drive, not just the array drives when looking for a folder corresponding to a user share.   Have you tried clicking the 'folder' icon at the right of the share name?   That may open up to show you the contents, and also what Drive they are on

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.