itimpi

unRAIDFindDuplicates.sh

Recommended Posts

Is it possible the size being compared is the actual size on disk? If so, a sparse file will be incorrectly reported as failing the size comparison, when the content of the file if read out or checksummed is identical.

Possible.  The size is being obtained using a command of the form

du -s filename

so you could see if they are reported the same on your system?  If that is the case then an alternative command could be used.

Try
du -sb filename

Share this post


Link to post
Share on other sites

Is it possible the size being compared is the actual size on disk? If so, a sparse file will be incorrectly reported as failing the size comparison, when the content of the file if read out or checksummed is identical.

Possible.  The size is being obtained using a command of the form

du -s filename

so you could see if they are reported the same on your system?  If that is the case then an alternative command could be used.

Try
du -sb filename

If you change the script on your system (it should be easy enough to find the du -s command in the script) does it fix the issue on your system?    As I currently do not have an example of it going wrong then it is a bit harder to test here.    Actually looking at the du options I am not sure why I used the -s option - it looks as if simply using the -b option instead would work?

 

I find it intriguing that the script has been available for over a year and this is the first time this issue has come up.  It is  just shows how difficult it can be to allow for all the edge cases.

Share this post


Link to post
Share on other sites

du -b

 

That seems to solve the issue I was having with this script. Thank you!

Share this post


Link to post
Share on other sites

du -b

 

That seems to solve the issue I was having with this script. Thank you!

Thanks for confirming that works.

 

I will do a bit of testing at my end and assuming nothing shows up I will upload an updated version of the script to the first post in this thread.

Share this post


Link to post
Share on other sites

First, thanks for this script :)

 

I found this because I stumbled upon an empty directory in one of my shares and was looking for something to locate those.

 

The good news: I found about 250GB of duplicate files I had no idea existed. So again, thank you.

 

The bad news: I cannot for the life of me get the -f/-F args to produce anything.  I created a duplicate folder on the chance that the one I stumbled on was the only one, but it is not being reported by the script.

 

Any help appreciated,

 

Thanks!

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

 

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

 

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

Not sure if the script is still being maintained but fix common problems plugin on the extended tests also checks for dupes

 

Sent from my LG-D852 using Tapatalk

 

 

Share this post


Link to post
Share on other sites

Yeah I guess it's not. I have fix common problems but my issue was duplicate empty directories on random disks which this script picked up very well and I manually deleted. I had no duplicate files.

 

Sent from my SM-G930F using Tapatalk

 

 

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

 

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

i'll look to see if this can be fixed simply and if so I will apply an update.    However if it is going to be hard to do without major changes to the script I probably will not bother.

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

 

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

i'll look to see if this can be fixed simply and if so I will apply an update.    However if it is going to be hard to do without major changes to the script I probably will not bother.

No worries. I didn't expect anything I just thought it was worth mentioning in case others were having issues now or in the future. Thanks itimpi and great script!

 

Sent from my SM-G930F using Tapatalk

 

 

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

i'll look to see if this can be fixed simply and if so I will apply an update.    However if it is going to be hard to do without major changes to the script I probably will not bother.

No worries. I didn't expect anything I just thought it was worth mentioning in case others were having issues now or in the future. Thanks itimpi and great script!

 

Sent from my SM-G930F using Tapatalk

I've worked out what line the message comes from.  It appears to be a quirk of the way bash handles wildcard expansion.  I will have to look up my bash special character handling to see if I can see a way of avoiding the issue.

 

Share this post


Link to post
Share on other sites

Sorry for reviving an old thread but just tried this out today and realised the script doesn't like apostrophes in file names.

 

Not sure if it's worth fixing or not?

 

ls: cannot access '/mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example': No such file or directory

The full directory is /mnt/disk*/STORAGE/laptop backup/Dropbox/EMPLOYEES/example's/

i'll look to see if this can be fixed simply and if so I will apply an update.    However if it is going to be hard to do without major changes to the script I probably will not bother.

No worries. I didn't expect anything I just thought it was worth mentioning in case others were having issues now or in the future. Thanks itimpi and great script!

 

Sent from my SM-G930F using Tapatalk

I've worked out what line the message comes from.  It appears to be a quirk of the way bash handles wildcard expansion.  I will have to look up my bash special character handling to see if I can see a way of avoiding the issue.

Thanks itimpi. Please don't waste your time if it's to much work though. It's only my OCD that makes me want a clean error free log file. Cheers

 

Sent from my SM-G930F using Tapatalk

 

 

Share this post


Link to post
Share on other sites

Does anyone know a good way to automatically delete the duplicate files that this script finds?

 

I had a 6TB drive become unmountable, recovered over 5TB of files from the emulated drive, then realized I hadn't tried repairing the filesystem, and got my drive back, so now I've got 5+TB of duplicate files, spread all across my array.

Share this post


Link to post
Share on other sites

when trying to run this script I get permission denied? I am logged on as root?

 

image.png.cb261133bdfcf1afbad546a0f9243ed0.png

Share this post


Link to post
Share on other sites

Have you given the script execute permission?     If you downloaded it to the flash drive this should be automatic (because it is FAT32 format) but will not be the case if put elsewhere.  Alternatively run it using the ‘sh’ command which does not require the script to have ‘execute’ permission.

Edited by itimpi
  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.