Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)


Recommended Posts

19 minutes ago, johnnie.black said:

 

Why not use copy/paste?

Hello,

I'm using a KVM switch to share my peripherals with my PC.

I can't copy and paste through that.

 

I was used to doing it that way for a long time in the past with other PC's long before the remote desktop connection became a part of windows.

 

I have only just started reading up about Putty and should probably retire the KVM and move ahead a couple of years in technology but the old man in me is a bit hesitant.

A question, please. If I open a putty session and start something in that console, will it keep running if I log out or close the window?

Do I need to keep a putty window open until the task is complete?

 

I kind of like the KVM because it is 2 button presses and I'm straight in and watching whatever is happening and it just keeps happening when I button out.

Link to comment
1 hour ago, CarloC said:

With the help of some community members here, and the instructions in the wiki that are based on this thread, I have completed the addition of a new drive, and replaced a RFS drive with an XFS drive.

I now have a drive excluded from the global shares as per the instructions.

 

At this point I could repeat this process as per the wiki for the remaining disks using the rsync command and the disk swaping of assignments etc.

 

However, is it possible for me to complete the transfer of the rest of the drives in this fashion:

1) Stop the array and put the excluded drive back into the global shares

2) Start the array - Will a parity check occur at this point? the drive was already included in the parity calculations even if it wasn't part of the user share, correct?

3) Use the Unbalance plugin to shift all the files from one of the RFS disks to the newly included and empty XFS disk.

4) Stop the array and change the file system on the newly emptied disk from RFS to XFS

5) Start the array and then format the drive. Will parity be maintained by the copying method used by Unbalance?

 

I'm hoping that this will then present me with another XFS formatted drive which is included in the array and I just proceed until all drives are done.

 

Any help with this is again appreciated.

 2 - no parity check would occur

 

5 - yes, parity is maintained

 

Parity is maintained at a very low level. Think of the disk as just a very long series of 1s and 0s. When the disk is formatted, some of those 1s and 0s get changed. Parity tracks those changes. When data is copied to the disk, some of those 1s and 0s are changed. Parity is maintained throughout.

 

The only way parity is not maintained is if you access a disk very directly so that unRaid has no way to detect a change to a disk. For example, if the disk is removed and put into another computer and updated.  Parity cannot be maintained. But it is not necessary to remove a disk to write to it outside of its array, but you have to work at it and no wiki instructions or posts are going to direct you to do those sorts of writes.

 

Enjoy your array! 

 

 

Link to comment
10 hours ago, CarloC said:

I was already a little sensitized to this issue because of an issue I am/was having with preclearing some drives.

In this post https://forums.lime-technology.com/topic/12391-re-preclear_disksh-a-new-utility-to-burn-in-and-pre-clear-disks-for-quick-add/?do=findComment&comment=506407 some corrections to the preclear_disk.sh file are presented.

I really struggled with getting the second more complex correction done and had to give up.

I will write a post to this topic talking about all the issues I have recently had with a preclear that would not go correctly.

 

Commands like that aren't made to be typed by hand!   ;)

 

I recommend using PuTTY, which does support copy and paste, although in an awkward way.  If nothing else, you can copy the command into a file, copy the file to the boot drive, then run the file from a terminal session.

 

10 hours ago, CarloC said:

The other information you shared has made me think of something. I'm not sure if this is the right place to bring this up.

It would seem to me that it is possible to have a second set of drives which are not a part of the user shares and could then act as backups to the drives in the array. The drives in the array could be public for use by anyone on the network, and the others could be secure and controlled by an administrator. Although not a true, offsite independent backup, they would be a set of redundantly protected backup disks holding copies of files.

Is this actually acheivable?

Does anyone do this?

 

Well yes they do, some using the Unassigned Devices plugin to add and control extra drives.  But a true backup should always be on a different machine, so most use other means for backups, like a second server, or cloud backups, etc.

Link to comment

Someone asked a question recently of how quick XFS was to delete files, remarking how painfully slow RFS is.

 

So I had a 1T data migration from one disk to another that I completed as a copy, and then timed the deletion of the source files.

 

It was a total of 2324 files, 194 folders, 1.03TB

 

Took 14 seconds!

 

XFS is quick to delete!

 

No idea how long RFS would have taken. Could have been 20 minutes or more.

 

Yet another reason ...

Link to comment
On 3/18/2017 at 4:36 PM, Harro said:

I will let everyone know tomorrow. Should have another drive empty tonight sometime. Ready for XFS .

Thank you all.

 Everything went smoothly and I must had had some excess gas with a lot of brain farts the other day. Have drive converted to XFS now and will continue with the rest. 

Thank you all once again.

Link to comment
1 hour ago, Harro said:

 Everything went smoothly and I must had had some excess gas with a lot of brain farts the other day. Have drive converted to XFS now and will continue with the rest. 

Thank you all once again.

Hello Harro,

 

I, too, thank everyone who took the time to read and respond to my questions and requests for help because, it turns out that, no more than a few hours after you asked a question about doing a disk conversion from RFS to XFS, I asked a very similar one. I also used Unbalance to empty a disk and then re-Format the empty disk. I then used Unbalance to move the next set of files and so on. I am doing my third disk now.

For me, it also appears to have worked seamlessly and with my array serving files the whole time except for just a few minutes. I am using Unraid 6.3.2

 

But this was after I did follow the file conversion wiki to do my first conversion.

 

To those who will know, I would ask why the wiki outlines, what seems to be, such a convoluted system of swapping drive assignments and new configurations while over-riding un-needed parity re-builds. At the same time, still very strongly cautioning about background processes that might be changing the files. It's fairly nerve-wracking for a newbie. 19 Steps is a daunting total.

 

Can I assume it is because my array is very simple? It is just a home media file server. One user share and no disk shares. No dockers or any other programs running. No cache drives and just one parity disk. In essence, I don't care where the actual files are located, I just need Unraid to keep track of the 1 user share.

 

If I had a more complex system, could I have brought the whole thing down by doing what I've done? Is that why keeping the files in the same disk location so important?

 

Not wishing to be a bother but just interested to learn.

 

Link to comment
22 minutes ago, CarloC said:

If I had a more complex system, could I have brought the whole thing down by doing what I've done? Is that why keeping the files in the same disk location so important?

Maybe the instructions were meant to cover everything that might be important to anyone. Not likely to bring the whole thing down, but it is possible to make things work differently than they did before depending on how you have things configured. And people can make mistakes if they aren't careful that would cause them to actually lose data.

 

My system is more complex than yours, but I did the conversion pretty early when this thread wasn't very long. If you know how parity works and how formatting works and how user shares work you can pretty much adlib it. The main idea is that changing the filesystem on a disk will format it, so its contents must be moved or it will be lost. Swapping disks, for example, is only important if you have something configured to use specific disks.

Link to comment
12 hours ago, CarloC said:

I, too, thank everyone who took the time to read and respond to my questions and requests for help because, it turns out that, no more than a few hours after you asked a question about doing a disk conversion from RFS to XFS, I asked a very similar one. I also used Unbalance to empty a disk and then re-Format the empty disk. I then used Unbalance to move the next set of files and so on. I am doing my third disk now.

For me, it also appears to have worked seamlessly and with my array serving files the whole time except for just a few minutes. I am using Unraid 6.3.2

 

But this was after I did follow the file conversion wiki to do my first conversion.

 

To those who will know, I would ask why the wiki outlines, what seems to be, such a convoluted system of swapping drive assignments and new configurations while over-riding un-needed parity re-builds. At the same time, still very strongly cautioning about background processes that might be changing the files. It's fairly nerve-wracking for a newbie. 19 Steps is a daunting total.

 

Can I assume it is because my array is very simple? It is just a home media file server. One user share and no disk shares. No dockers or any other programs running. No cache drives and just one parity disk. In essence, I don't care where the actual files are located, I just need Unraid to keep track of the 1 user share.

 

If I had a more complex system, could I have brought the whole thing down by doing what I've done? Is that why keeping the files in the same disk location so important?

 

I'll start by giving some background on the current wiki method, and then touch on a few things you may not have thought about.  I had noticed already several users using unBALANCE, and had been thinking about adding a method based on it, but I won't be able to completely flesh out the steps as I've never used unBALANCE, and can't because I don't use User Shares (which of course partly explains my own methodology).  I've always preferred knowing and controlling exactly where everything is, and I've a personal catalog file I keep open in an editor tab with all of my series and categories listed, with the drives they're on.  For example, I know that my Alias shows are on Disk 1, and all my PBS stuff is on Disk 4.  User Shares are convenient for many, but for me it just adds another layer to the access.  Off-topic though ...  But converting drives is a drive issue, only indirectly a User Share issue.  It's drives that are formatted, and that dominates the problem.


I don't know for sure, but I suspect if you wrote out the steps for doing the entire array with unBALANCE, it would be close to 19, or maybe even more.  A general method for users at large has to handle every contingency and every situational variation, as safely as possible for the least technical of users.  That always takes longer, and often more steps are involved, 'just in case'.  It inevitably means lots more words!

 

I do think my method is the most straightforward, safest, and fastest though, but then disk shares is what I work with and understand.  It does have an extra step or two to accommodate User Shares.  For User Share users, which is most or almost all unRAID users, unBALANCE sounds attractive, and I agree that it's a great plugin!  It uses basically the same rsync command, so should have the same speed and file verification checksumming for the individual transfers.  But in operation over multiple drives, it's likely to take longer, because it's moving the files off one drive to the others, then moving the files back when you do the next drive, and possibly moving some files multiple times, pushing them around the various drives to keep them out of the way of the next format operation.  With my method, files only move once.  The only way to make sure no files get moved more than once with unBALANCE is to carefully manage exactly where each file starts and where it ends up, and that sounds like something you wanted to avoid, if you don't want to care where the files are.

 

To me, my method seems just as seamless, and allows full operation just as much.  Both ways, you have to stop the array, make changes, then restart the array, just as many times.  When I did it with 6.1, I didn't have to use New Config, just adjust the assignments and restart the array.  Unfortunately in 6.2, something changed and it won't allow that, so we had to add the New Config step, which I agree does add a little extra hassle.  6.0 and 6.1 users don't have to do that.  We have requested that to be corrected, so we can just restart the array, but as far as I know, it hasn't happened yet (haven't checked it recently though).

 

The strong cautions though are just as important, for either method!  How do you know for sure that there were not files lost when you did it?  Because if there were any other processes (internal plugins, Dockers, scheduled scripts, or VM's, or externally scheduled backups) writing files to the array, they could very well have added files to the very drive you were emptying with unBALANCE.  In fact, the emptying of a shared drive makes it the most attractive target for writing new files to.  Either you make absolutely sure that nothing else could be writing to the array, or you should run unBALANCE a final time, just to make sure that nothing is left on the drive (written to it after unBALANCE had swept across it), before you format it.  Does unBALANCE after moving folders come back and check if they have reappeared on that drive, with more files in them?

 

So I'll add a general unBALANCE-based method, and I think many users may prefer it, as it does hide some of the detail, but it will take longer to perform.  And they will have to manually configure unBALANCE for each drive emptying.  The steps will have to have just as many warnings.  Users who aren't running anything else at all will obviously have a simpler time, can ignore the warnings, but that's a special case.

 

I really don't want any of this to be daunting or nerve-wracking, but each time we hear of a user doing something we hadn't thought of and getting into trouble, the word count increases.  The more words, the more daunting it appears ...  (just look at this post for instance!)

Edited by RobJ
add word
  • Upvote 2
Link to comment
11 hours ago, trurl said:

If you know how parity works and how formatting works and how user shares work you can pretty much adlib it. The main idea is that changing the filesystem on a disk will format it, so its contents must be moved or it will be lost. Swapping disks, for example, is only important if you have something configured to use specific disks.

 

^THIS. Hoping that people following the guide are taking the time to understand how and why the guide works, and synthesizing the basic truth that trurl so eloquently states.

 

7 minutes ago, RobJ said:

... and can't because I don't use User Shares (which of course partly explains my own methodology).  I've always preferred knowing and controlling exactly where everything is, and I've a personal catalog file I keep open in an editor tab with all of my series and categories listed, with the drives they're on.  For example, I know that my Alias shows are on Disk 1, and all my PBS stuff is on Disk 4.  User Shares are convenient for many, but for me it just adds another layer to the access.  ...

 

RobJ - this was me for 9 of my 10 years using unRAID. In the last year I've begun to embrace user shares, and finally decided they are a good thing. I tend to have some shares that are totally on a single disk, and others that spread out over several. I may combine the ones that fit on one disk together on a single disk (they may have been on separate disks when my disks were 1T, but now that they are 4T and even 8T, it just doesn't make sense to leave so much slack. But typically I dedicate whole disks to shares that span multiple disks.

 

I recently converted my considerable ISO library to MKV to use with Plex. And being able to specify a user share as the destination allowed it to run for many DAYS and overflow onto new drives without a hiccup. :) A beautiful thing.

 

8 minutes ago, RobJ said:

I really don't want any of this to be daunting or nerve-wracking, but each time we hear of a user doing something we hadn't thought of and getting into trouble, the word count increases.  The more words, the more daunting it appears ...  (look at this post for instance!)

 

This is the challenge! I am reminded of the saying that "It is impossible to make anything foolproof, because fools are so ingenious!"

Link to comment
47 minutes ago, bjp999 said:

In the last year I've begun to embrace user shares, and finally decided they are a good thing. I tend to have some shares that are totally on a single disk, and others that spread out over several. I may combine the ones that fit on one disk together on a single disk (they may have been on separate disks when my disks were 1T, but now that they are 4T and even 8T, it just doesn't make sense to leave so much slack. But typically I dedicate whole disks to shares that span multiple disks.

I just control things by setting each of my user shares to only include specific disks. I have several shares that are not very large and are all together on one disk. Only my Movies share spans (only 2 so far) disks. If it ever needs more I will just include another when needed (or more likely, upsize its disks). This is why I think user share settings gives you all the control you need. And my other user never has an opportunity to do the "User Share Copy Bug", since I don't share disks.

Link to comment
6 minutes ago, trurl said:

I just control things by setting each of my user shares to only include specific disks. I have several shares that are not very large and are all together on one disk. Only my Movies share spans (only 2 so far) disks. If it ever needs more I will just include another when needed (or more likely, upsize its disks). This is why I think user share settings gives you all the control you need. And my other user never has an opportunity to do the "User Share Copy Bug", since I don't share disks.

 

Exactly what I do.

 

I will say I had some questions and experimented - like if I move a file in a user share from one directory to another, will unRAID keep the file on the same physical disk even if I am breaking some split level rule. I have found (haven't tested extensively) that a move will  move the file onto the same disk if living within the same user share. And if you are moving a bunch of files, it will treat each file separately moving it to directories on the drive it lives on. Pretty slick!

Link to comment

I started my conversion of 16 drives to XFS and when I got to disk4 I got the following error

 

REISERFS warning (device md4): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.axBWPi) associated with [2 4 0x0 SD]

 

Any thoughts?

Link to comment
1 hour ago, mtruffa said:

I started my conversion of 16 drives to XFS and when I got to disk4 I got the following error

 

REISERFS warning (device md4): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.axBWPi) associated with [2 4 0x0 SD]

 

Any thoughts?

Post diagnostics of course

Link to comment
22 hours ago, RobJ said:

So I'll add a general unBALANCE-based method, and I think many users may prefer it, as it does hide some of the detail, but it will take longer to perform.  And they will have to manually configure unBALANCE for each drive emptying.  The steps will have to have just as many warnings.  Users who aren't running anything else at all will obviously have a simpler time, can ignore the warnings, but that's a special case.

 

Thanks Rob!  I  was planning to use this process to convert my server and it would be killer to have instructions  from you.  It's hard to keep track of everything in this thread.

Link to comment
42 minutes ago, mtruffa said:

here is the syslog and smart report on the extended test i ran. No errors found. I also ran reiserfsck --fix-fixable and no errors found. 

tower-syslog-20170321-1941.zip

tower-smart-20170321-1913.zip

Sorry, I meant go to Tools - Diagnostics and post complete zip. It includes syslog(s), SMART for all drives, lots more. Thought you might have already known about it. The standard for V6.

Link to comment
3 hours ago, mtruffa said:

I started my conversion of 16 drives to XFS and when I got to disk4 I got the following error

 

REISERFS warning (device md4): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.axBWPi) associated with [2 4 0x0 SD]

 

This looks like it might be an old issue returned, involving extended attributes and AFP.  I'm guessing you have accessed this drive over AFP at some point?  Read the discussions and fixes found in these threads:

 

   A "mover" issue?  (Mover uses rsync too)

   Running mover does not successfully move items

   Extended Attributes Fix

Link to comment
3 hours ago, mtruffa said:

I started my conversion of 16 drives to XFS and when I got to disk4 I got the following error

 

REISERFS warning (device md4): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.axBWPi) associated with [2 4 0x0 SD]

 

Any thoughts?

 

Also, there are some other things in your syslog:

Mar 21 19:43:59 Tower root: Fix Common Problems: Warning: Share Movies is set for both included (disk1,disk2,disk3,disk4,disk5,disk6,disk7,disk16) and excluded (disk17) disks ** Ignored
Mar 21 19:43:59 Tower root: Fix Common Problems: Warning: Share Music is set for both included (disk8) and excluded (disk17) disks ** Ignored
Mar 21 19:43:59 Tower root: Fix Common Problems: Warning: Share TV is set for both included (disk9,disk10,disk11,disk12,disk13,disk14,disk15) and excluded (disk17) disks ** Ignored

Why do you have it set to ignore these? They are there for a reason. Include means "include ONLY the listed disks", but Exclude means "ALL EXCEPT the listed disks". You should never set both include and exclude.

 

And then there's this:

Mar 21 19:44:23 Tower root: container "MineOS" hostpath "/mnt/appdata/minecraftos" does not exist

It should be /mnt/user/appdata/minecraftos or possibly /mnt/cache/appdata/minecraftos

Link to comment

Hi - not sure if anyone can help; I've had a read around but I haven't quite managed to answer my questions. I'm aiming to replace an existing 3tb ReiserFS data disk with a new 8tb XFS drive, then physically remove the existing 3tb drive (to leave space for a future large drive when needed). Rather than convert everything now, I'll just add larger replacement XFS drives as I need more space. I'm running version 6.2.1, with one parity drive.

 

I've followed the File System Conversion procedure, currently on step 8 (data is all copying with rsync to the new drive, about 80% done). I'd misunderstood what excluding the disk did (I thought it excluded it from parity, understand better now), so I can't simply remove the old disk and have parity stay valid.

 

I believe I still follow the procedure down to step 16 (restart the array with 'parity already valid' checked.). But what should I do next? I could, I think, unassign the old drive then rebuild parity, but ideally I'd like to keep protected. Should I follow the procedure to Shrink Array? That is, remove the share exclusion, format the disk (as XFS) to clear it quickly, run the 'Clear Me' user script, and follow the assorted other steps to remove the drive?

 

If that is the case, then my only remaining confusion would be whether to remove the exclusion, then format, or the other way round. But I'd also like to know if (a) there's a better/faster way of removing my now-unneeded duplicate data disk, and (b) since I'll be repeating the procedure at some future point, if my aim from the start is just to remove the original disk once the data is copied, is there a simpler overall method? Ideally I suppose to make a duplicate of the original disk, onto the new XFS file system, but without changing parity (so I could just copy the drive, swap assignments, and be done).

 

Thanks in advance for any help that can be offered,

 

TC

Link to comment
2 hours ago, Shamutanti said:

Should I follow the procedure to Shrink Array? That is, remove the share exclusion, format the disk (as XFS) to clear it quickly, run the 'Clear Me' user script, and follow the assorted other steps to remove the drive?

I think the reason for going to global share settings and taking the drive out of the exclude list is so you won't forget and later accidentally exclude another disk that eventually comes to occupy that disk slot (#).

 

The reason for formatting is because the script that clears the drive checks to make sure it doesn't have any files still on it. Formatting is quicker than deleting to get rid of all the files on it, but formatting doesn't actually clear anything. Formatting actually writes an empty filesystem to the disk. It is the Clear Me script that clears the drive by writing zeros to every bit of the disk, and since the disk is still in the array while it does this, parity is updated to reflect those zeros. Once the disk is all zeros, it can be removed without affecting parity.

Link to comment

I am adding a larger drive to my array and trying to convert the all drives to XFS from RFS. Currently I have:

 

Drive 1: 2TB RFS

Drive 2: 2TB RFS

Drive 3: 4TB XFS (excluded, empty)

 

If I copy the contents of drive 1 to drive 3, when I swap the drive assignments of Drive 1 and Drive 3, does parity remain valid or will I need to rebuild parity? In other words, can I do step 16 from the wiki or do I need to rebuild parity at that point?

 

Sorry if this has been answered previously but I wasn't able to find it in this thread.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.