April 09, 2020, 08:29:21 am
News: If someone gives you crap then give them a Bad Star
Pages: [1] 2   Go Down
Author Topic: Duplicate File Finder  (Read 9480 times)
wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« on: June 11, 2014, 05:25:58 am »

G'day smile
 
I have been thinking of doing this for a very long time. I wanted to write a small utility which I can use to find duplicate files from a drive. I am sure you can find one online too but for me this is a fun project.
 
You can grab the latest version from http://www.codelake.com/downloads/DuplicateFileFinder.zip
 
Created in Visual Studio 2012, .NET Framework 4.5.1 using C#
 
I will keep adding more features to it as I find time. Please let me know if you have anything specific in mind and I shall try to fit it in in the development schedule.
 
Cheers smile
« Last Edit: June 19, 2014, 06:22:12 am by wrack » Logged

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4532


Pure Evil Administrator


WWW
« Reply #1 on: June 11, 2014, 05:53:19 pm »

Cool! Are you actually comparing content or are you just doing a hash check?
Logged

-- signature --

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #2 on: June 11, 2014, 08:24:43 pm »

Cool! Are you actually comparing content or are you just doing a hash check?
MD5 Grin
Logged

simmo
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8547



« Reply #3 on: June 11, 2014, 08:31:10 pm »

I've been meaning to run something like this on my "backup" drive as I suspect there are a couple folders I've saved twice (and have a few bookmarked, just never tried). I ran your app and it says no matches, so maybe I don't have any :o

(also, the popup that said no matches was no where to be found, only on task bar and I could not do anything with either window except right click on taskbar and "close window.")
Logged

                 ..... .....

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4532


Pure Evil Administrator


WWW
« Reply #4 on: June 11, 2014, 09:17:19 pm »

I use Linux so I only need to open a terminal session and enter
Code:
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
Logged

-- signature --

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #5 on: June 12, 2014, 12:04:51 am »

I've been meaning to run something like this on my "backup" drive as I suspect there are a couple folders I've saved twice (and have a few bookmarked, just never tried). I ran your app and it says no matches, so maybe I don't have any :o

(also, the popup that said no matches was no where to be found, only on task bar and I could not do anything with either window except right click on taskbar and "close window.")

Did you run a simple search or advanced search?

Weird behaviour with the dialog. I can't replicate it here. How many files are we talking about? I ran advanced search on a 36.9GB folder with 436639 files and 61814 subfolder and it took 1 hour to find me all duplicate files. Working on an updated version. Will post it soon.
Logged

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #6 on: June 12, 2014, 12:05:25 am »

I use Linux so I only need to open a terminal session and enter
Code:
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
Show off horse
Logged

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #7 on: June 12, 2014, 01:58:28 am »

Ok new version is up. Get it from http://www.codelake.com/downloads/DuplicateFileFinder.1.2014.0612.0138.zip
Logged

simmo
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8547



« Reply #8 on: June 13, 2014, 02:16:20 pm »

Did you run a simple search or advanced search?
Simple.

Weird behaviour with the dialog. I can't replicate it here.
It might have to do with it being minimized while running. I typically have a LOT of things open at once (20 at the current moment) so when I "switch tasks" I typically hit [win key] + m to minimize errything then select what I want from the task bar, so my view isn't cluttered with windows I'm not currently working with.

How many files are we talking about? I ran advanced search on a 36.9GB folder with 436639 files and 61814 subfolder and it took 1 hour to find me all duplicate files.
About 65-7gb, your app says 743,610 files.

Working on an updated version. Will post it soon.
Trying it out now, will let you know yes


[edit1]
Simple search found no dups (which may be correct), same behavior with the minimized (or off screen?) dialog box.

Running adv search now.
[/edit1]

[edit2]
Adv search found over 65k duplicates laugh1 same behavior with the minimized (or off screen?) dialog box.

[/edit2]

« Last Edit: June 13, 2014, 06:04:13 pm by msimmons » Logged

                 ..... .....

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #9 on: June 13, 2014, 10:28:30 pm »

Ok. I am not able to fix the dialog issue but all that dialog says is how many files were found and you just have to click ok. If you have minimised the application and if the dialog box was opened then try to right click on the dialog form from the taskbar and click close.

Working on to fixing the issue. May have to come up with something better.
Logged

simmo
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8547



« Reply #10 on: June 14, 2014, 01:56:53 am »

If you have minimised the application and if the dialog box was opened then try to right click on the dialog form from the taskbar and click close.
I could not do anything with either window except right click on taskbar and "close window."
jamie

One thing you might consider is excluding "recycler" (or whatever that hidden recycle bin is called) a lot of my matches were in there.
Logged

                 ..... .....

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #11 on: June 14, 2014, 04:55:40 am »

If you have minimised the application and if the dialog box was opened then try to right click on the dialog form from the taskbar and click close.
I could not do anything with either window except right click on taskbar and "close window."
jamie

One thing you might consider is excluding "recycler" (or whatever that hidden recycle bin is called) a lot of my matches were in there.
Yeah I found that too. I will modify the app to do just that.
Logged

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #12 on: June 17, 2014, 12:28:05 am »

New version (1.2014.0617.0015) is up. Get it at http://www.codelake.com/downloads/DuplicateFileFinder.zip


* Make the minimum size of the folder tree smaller.
* Removed the final message dialog and now showing information in the status bar.
* Exclude Recycle Bin and System Volume Information folders while scanning.
« Last Edit: June 17, 2014, 12:29:43 am by wrack » Logged

simmo
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8547



« Reply #13 on: June 17, 2014, 06:25:20 pm »

Thanks! I'll try to get around to running it in the next few days and let you know how it goes  jamie
Logged

                 ..... .....

wrack
Master of TMEOotAP
*

Stars: +3/-0
Offline Offline

Posts: 11116


Don't look back. Look in the mirror to look back.


WWW
« Reply #14 on: June 17, 2014, 11:28:48 pm »

No problems smile

Let me know if you find anything nasty!
Logged

Pages: [1] 2   Go Up
Print
 
Jump to: