All data just deleted in an instant


Recommended Posts

I have no idea what just happened. My entire array just vanished before my eyes in an instant when I was hash checking files in rtorrent. A file that is not on my array hash checked 0% and when I deleted the file showing zero bytes, the entire movie folder disappeared with every single file in it.

 

I can only assume I made a mistake somewhere but I have no idea what I did wrong and I don't want to make the same mistake again. 28TB of movie files just disappeared and if I can recover them without too much hassle, that would be incredible. I immediately stopped the array after I got several popup notifications informing me that the disks returned to normal utilization levels. So from the time that files were available and accidentally deleted to the array stopping was only a few minutes.

 

 

I have a syslog copy/paste ready but not sure how to format properly without a wall of text.

 

tower-diagnostics-20170620-1318.zip

Edited by TheRefugee
Link to comment

To try and clarify what was happening at the time of data loss:

 

rTorrent, Binhex-rtorrentVPN, was hash checking files after a large batch of torrent files were added. I knew some of them would return 0% because the files were no longer on the array. I had my Movie share open on my mac to confirm the torrent returning 0% was indeed not on my array. After confirming that the torrent file showed "Zero Bytes" on my mac, I "move to trash" the file showing "Zero Bytes" and then went back to the rtorrent webGUI to remove the torrent. I right clicked on the single torrent file and slected Delete Data (all) AFTER already deleting/moving to trash the corresponding file showing "Zero Bytes" on my SMB/Mac Finder window. After selecting the Delete Data (all) option, every single file in the share disappeared.

Link to comment

I've been very displeased with rTorrent, too. (Not the dockerization, mind you, just the fact that it slurpped up all my .torrent files and my Win machine uTorrent client could no longer find them.)

 

Without having looked through your diagnostics file, I'd be concerned that the option you selected said Delete Data (All). I'm by no means an rTorrent expert, but my immediate take is that "all" means "all", and that you just deleted all the downloaded data it knew about.

 

I believe that unless you have backups, your data is gone. unRAID will protect you against a single (or double with dual parity disks) disk failure, but it isn't a backup solution. A couple of possible thoughts:

 

1. Stop the array and don't let anything else write to it.

2. Identify which file system you have formatted your disks. (XFS or ReiserFS)

3. Do some Googling to see if there are any Linux based utilities that will allow you to undelete files. I know that when a file is "deleted" usually the OS just marks it deleted in the disk directory, and doesn't actually delete the file. There were many a time that I needed an undelete utility back in the DOS days before Windows implemented the Recycle Bin.

4. With some careful work, you may be able to go disk by disk and undelete all your movies.

 

Also, attaching the diagnostics is how you post the syslog without posting a massive wall of text, so you did the right thing there.

Link to comment
15 minutes ago, FreeMan said:

I've been very displeased with rTorrent, too. (Not the dockerization, mind you, just the fact that it slurpped up all my .torrent files and my Win machine uTorrent client could no longer find them.)

 

Without having looked through your diagnostics file, I'd be concerned that the option you selected said Delete Data (All). I'm by no means an rTorrent expert, but my immediate take is that "all" means "all", and that you just deleted all the downloaded data it knew about.

 

I believe that unless you have backups, your data is gone. unRAID will protect you against a single (or double with dual parity disks) disk failure, but it isn't a backup solution. A couple of possible thoughts:

 

1. Stop the array and don't let anything else write to it.

2. Identify which file system you have formatted your disks. (XFS or ReiserFS)

3. Do some Googling to see if there are any Linux based utilities that will allow you to undelete files. I know that when a file is "deleted" usually the OS just marks it deleted in the disk directory, and doesn't actually delete the file. There were many a time that I needed an undelete utility back in the DOS days before Windows implemented the Recycle Bin.

4. With some careful work, you may be able to go disk by disk and undelete all your movies.

 

Also, attaching the diagnostics is how you post the syslog without posting a massive wall of text, so you did the right thing there.

From my previous usage of ruTorrent, there are 2 options when removing a torrent. One only removes the torrent file and leaves the data and the other option removes the torrent file and the corresponding data of that single torrent file.

 

 

1. Array was stopped within minutes of the "delete Data (All)" operation. Tower still powered on though.

2. XFS

3.I am a total novice so I am not going to do anything on my own until further instruction.

4. I can only hope!

 

From my complete lack of knowledge and uneducated opinion, after manually deleting the "Zero Bytes" file from my mac/smb share finder window AND THEN returning to the rutorrent webGUI and selecting the "Delete Data (All)" option on a single torrent, the plugin couldn't see the file and then deleted the entire folder. FWIW, the file was not in it's own folder but placed directly in the share (/mnt/user/Movies/title.2016.mkv/".

 

To be perfectly clear, I don't want to blame binhex's rtorrent docker because I am admittedly a novice user and I find it hard to believe it wasn't complete user error.

Edited by TheRefugee
Link to comment
Just now, ashman70 said:

Do you have a backup of your data?

 

I have a little less than half of it backed up on a remote server and the other half can be recovered from third party sources but if I can recover the data locally that would be the vastly more convenient option. Nothing critical like family photos has been permanently lost. Backing up/re-downloading ~30TB of data at 3MB/s is something I would like to avoid if possible.

Link to comment
3 minutes ago, ashman70 said:

Do you have a backup of your data?

Assuming he doesn't since I mentioned backups and he didn't respond positively to that option... :(

 

The first Google result for "linux undelete xfs" pointed me to this which seems to have worked for this guy. Of course, this comes with NO GUARANTEE from me, I'm just trying to help point you in the right direction.

 

Maybe one of the local gurus will pop in shortly with additional suggestions and/or advice.

Link to comment
2 minutes ago, TheRefugee said:

 

I have a little less than half of it backed up on a remote server and the other half can be recovered from third party sources but if I can recover the data locally that would be the vastly more convenient option. Nothing critical like family photos has been permanently lost. Backing up/re-downloading ~30TB of data at 3MB/s is something I would like to avoid if possible.

In that case, you may be better off recovering from your backup and deciding how critical the remaining files are.

 

That link I provided indicated there was a reasonable amount of work involved, but that it did do a 95+% job of recovery.

Link to comment
3 minutes ago, FreeMan said:

In that case, you may be better off recovering from your backup and deciding how critical the remaining files are.

 

That link I provided indicated there was a reasonable amount of work involved, but that it did do a 95+% job of recovery.

 

Thanks for taking the time to try and push me in the right direction. I greatly appreciate it.

 

Regardless of how this shakes out, I will be reevaluating my storage solutions; time to have a local backup.

 

I'm mostly upset about my Plex library and the countless hours curating movie posters, backgrounds, and alphabetical/chronological series order. I have a backup of my Plex metadata so maybe the Plex forums will have some solutions for preserving that data once backup is complete....in 6 months lol

Link to comment

I really hope someone can help you recover your data, in any case, I know it doesn't help to hear this now, but have a complete backup of your data it an absolute must. I have a massive plex library and its duplicated onsite. It would take me years to reconstitute the library if I even could, so I don't take any chances.

Link to comment

Hi Refugee,

 

Since you e-mailed our support address as well, I thought I'd chime in with some feedback, though I think you're already on the right track with the guys in here providing advice.  In short, this definitely sounds like a application caused the loss of data, not an issue with the OS or any of the filesystems we support.

 

In the future, we plan to implement mechanisms in the OS to allow you to easily recover from scenarios like this within a few clicks, but it will take some time for us to implement them properly.  In the meantime, we advise users to be very cautious about applications they install and actions they take within those apps, especially whenever a button has the word "Delete" in it.

Link to comment
14 hours ago, jonp said:

this definitely sounds like a application caused the loss of data, not an issue with the OS or any of the filesystems we support.

 

i would agree with this (docker image maintained by me), please see my response to his post:-

 

 

14 hours ago, jonp said:

n the future, we plan to implement mechanisms in the OS to allow you to easily recover from scenarios like this within a few clicks

 

That is REALLY good news, and a very valuable feature for unRAID, although i hope never to use it :-).

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.