Everything Context menu find duplicates in selected folder

Discussion related to "Everything" 1.5 Alpha.
Post Reply
eswul62
Posts: 46
Joined: Wed Jul 31, 2013 6:07 am

Everything Context menu find duplicates in selected folder

Post by eswul62 » Tue Jan 11, 2022 1:15 pm

Using Windows Explorer or Directory Opus I would like to :

r-click a folder (in the folder tree)

and have Everything searched that folder for duplicates with parameters dupe: dmdupe: sizedupe:

How to do that?

Thanks.

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Tue Jan 11, 2022 6:31 pm

Everything 1.5 Alpha introduces a new method of duplicate finding. Column-data duplicates. This effectively replaces the dupe:, dmdupe:, sizedupe: commands because those only work against the entire filesystem Index and cannot be refined to a specific drive or folder or search results.

For example:

dupe: is now superseded by the combination of sort:name find-dupes:

sizedupe: is now superseded by the combination of sort:size find-dupes:

dmdupe: is now superseded by the combination of sort:dm find-dupes:

You can stack these and include them with other keyword searches, and they will be acted upon from left to right.

Changing how your windows context menus work is beyond the scope of my help. I recommend using bookmarks within Everything for workflow.

NotNull
Posts: 3648
Joined: Wed May 24, 2017 9:22 pm

Re: Everything Context menu find duplicates in selected folder

Post by NotNull » Tue Jan 11, 2022 7:16 pm

As @raccoon already mentioned: the dupe: functions work against the entire Everything database.
that means that you will get unexpected results. Fore example, if you search for: exact:abc.txt dupe: dmdupe: sizedupe:,
it will find all 3 abc.txt files with size 123 and date "yesterday", but also all 5 abc.txt files with size 456 and date "last year".


An alternative: import the following REG file and then you can choose "Search for exact duplicates" in the context menu.
(Check if the path in the regfile corresponds with the location of your Everything and edit the reg-file accordingly.)


Note this can be accomplished in a better way by using the Search Preprocessor functions, but I consider those "too alpha" to rely on at this stage of development.



ExactDupes.reg

Code: Select all

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Classes\*\Shell\SearchDupes]
"MUIVerb"="Search for exact duplicates"

[HKEY_CURRENT_USER\SOFTWARE\Classes\*\Shell\SearchDupes\Command]
@="cmd.exe /c for %%F in (\"%1\") do start \"\"  \"C:\\Program Files\\Everything 1.5a\\Everything64.exe\"    -search \"  exact:\"\"\"%%~nxF\"\"\"     size:%%~zF    dm:\"\"\"%%~tF\"\"\"      \""


NotNull
Posts: 3648
Joined: Wed May 24, 2017 9:22 pm

Re: Everything Context menu find duplicates in selected folder

Post by NotNull » Tue Jan 11, 2022 7:27 pm

OK, I completely overlookes the "r-click a folder (in the folder tree) and have Everything searched that folder for duplicates" part....
What I wrote is for right-clicking a file and find exact duplicates (based on name, size, date) of that file anywhere.
Don't think that is possible with Everything (but will think about it).

You will be better off with a dedicated duplicate finder utility like AllDup.

therube
Posts: 3423
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything Context menu find duplicates in selected folder

Post by therube » Tue Jan 11, 2022 7:31 pm

I'm probably not understanding, but...
You can stack these and include them with other keyword searches, and they will be acted upon from left to right.
Explain that a bit?
sort:name find-dupes: sort:size find-dupes:

Should that find name dupes that are only [also] the same size?

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Tue Jan 11, 2022 8:47 pm

yes

therube
Posts: 3423
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything Context menu find duplicates in selected folder

Post by therube » Tue Jan 11, 2022 8:57 pm

But that is not the case.
.
.
Everything - sort by name by size.png
Everything - sort by name by size.png (19.87 KiB) Viewed 4863 times

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Wed Jan 12, 2022 1:11 am

Unfortunately, dupe: dmdupe: sizedupe: will not work with a folder search as these search functions find duplicates in the ENTIRE index.

With Everything 1.4, please try the following:
  • In the Registry Editor (Start -> regedit), navigate to:
    HKEY_CLASSES_ROOT\Directory\shell
  • Create the following key:
    Find dupes in Everything...
  • Create the following key:
    command
  • Set the (Default) data to:
    "C:\Program Files\Everything\Everything.exe" -path "%1" -search sizedupe:
Please note: this will find files with a duplicated size anywhere in the entire INDEX and limit the results to the specified path.





For Everything 1.5, please try the following:
  • In the Registry Editor (Start -> regedit), navigate to:
    HKEY_CLASSES_ROOT\Directory\shell
  • Create the following key:
    Find dupes in Everything 1.5...
  • Create the following key:
    command
  • Set the (Default) data to:
    "C:\Program Files\Everything 1.5a\Everything64.exe" -path "%1" -search "find-dupes: sort:size"



sort:name find-dupes: sort:size find-dupes:
Sorting and find-dupes are applied after the search completes.
Only the right-most sort: and find-dupes: command is used.

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Wed Jan 12, 2022 3:37 am

void wrote:
Wed Jan 12, 2022 1:11 am
sort:name find-dupes: sort:size find-dupes:
Sorting and find-dupes are applied after the search completes.
Only the right-most sort: and find-dupes: command is used.
They didn't stack before? Should they be stacked? The same way that using column header right-click "Find xxx Duplicates" can be stacked?

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Wed Jan 12, 2022 6:46 am

They didn't stack before?


No.


Should they be stacked?
The same way that using column header right-click "Find xxx Duplicates" can be stacked?
Probably, yes.


For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment?
I will trial stacking dupe search commands in the search box.
Thanks for the suggestion.

eswul62
Posts: 46
Joined: Wed Jul 31, 2013 6:07 am

Re: Everything Context menu find duplicates in selected folder

Post by eswul62 » Wed Jan 12, 2022 1:17 pm

raccoon wrote:
Tue Jan 11, 2022 6:31 pm
Everything 1.5 Alpha introduces a new method of duplicate finding. Column-data duplicates. This effectively replaces the dupe:, dmdupe:, sizedupe: commands because those only work against the entire filesystem Index and cannot be refined to a specific drive or folder or search results.

For example:

dupe: is now superseded by the combination of sort:name find-dupes:

sizedupe: is now superseded by the combination of sort:size find-dupes:

dmdupe: is now superseded by the combination of sort:dm find-dupes:

You can stack these and include them with other keyword searches, and they will be acted upon from left to right.

Changing how your windows context menus work is beyond the scope of my help. I recommend using bookmarks within Everything for workflow.
Thank you so much!
Ahum . . .
I was not aware of this to be honest.

I do have a separate tool to check for duplicates (purchase), however, I use it only in case of checking for duplicates within multiple folders on multiple drives, such scenarios.
It takes quite a lot clicks selecting folders, blah blah and... waiting for the results and so.

For the fast results I'd like to use Everything.

Yesterday I was working within 1 folder, with quite a number of duplicates, with different names though.
Sort on size and then judge/select and delete them one by one, leaving 1 in place.

Again thanks for the above.

Note sure how to combine them though, meaning: show ONLY files that are duplicates both size AND date (at the same time).

As far as I can see:
sort:size find-dupes: sort:dm find-dupes:

The top -say- 15/16 rows are duplicate modified date wise, I am seeking a way to find the ones like the below 6 (same size/date)

SnagIt-12012022 140905.png
SnagIt-12012022 140905.png (31.49 KiB) Viewed 4810 times

Any suggestions?

eswul62
Posts: 46
Joined: Wed Jul 31, 2013 6:07 am

Re: Everything Context menu find duplicates in selected folder

Post by eswul62 » Wed Jan 12, 2022 1:57 pm

@void - thank you very much!
This may work for the installed version, I am also running a portable version, side by side.
So I am trying to find an alternative way, that I may use in future cases.

Whereas I initially thought it was limited to 1 folder, I just now noticed it involves more folders.

Feel embarrassed to confess that these 'sort:size find-dupes:' an d 'sort:dm find-dupes:'
were new to me. Ahum...

But I believe I've pretty much covered the matter now by using:

[example]

X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" sort:size find-dupes: sort:dm find-dupes:
or
X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" France find-dupes:
(and sort manually)

therube
Posts: 3423
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything Context menu find duplicates in selected folder

Post by therube » Wed Jan 12, 2022 3:57 pm

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.
I'm totally against that.
Cause IMO all that would do is to clutter up the search bar & more importantly history.
If manually typed in, I'm OK with it ending up in history, but if it is automatically entered only cause you selected a menu/context menu item, I wouldn't care for that.

(Even the same way that, Search for {Name|Size|Date...} does, I don't care for that either.)

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Thu Jan 20, 2022 6:10 am

Everything 1.5.0.1296a makes some changes to dupe: and finddupes:

You can now specify a property with dupe: search function:

dupe:<property-name>

For example:

dupe:size
dupe:date-modified
dupe:dm (dm is an alias for date modified)

The old behavior of dupe: stays the same, that is dupe: (without a property-name) finds files/folders where another file/folder with the exact same name exists in the index.



Like dupe:, you can now specify a property with finddupes: search function:

finddupes:<property-name>

For example:

finddupes:size
finddupes:dm


This will avoid the messy sort:<property-name> finddupes: syntax.
However, the old behavior of sort:<property-name> finddupes: will still work when no property-name is specified with finddupes:



You can now specify multiple dupe:<property-name> and finddupes:<property-name> in a search.

duplicates are found in order from left to right.

For example:

dupe:size dupe:dm dupe:name



You can now also specify the sort order without affecting the dupe command:

For example:

dupe:size dupe:dm dupe:name sort:path


For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.
I did try this, but it felt slow and clunky.
Each time you add a dupe: to the search, Everything must do a full requery (slow).
whereas with the column header -> right click -> find dupes, Everything only needs to find the duplicates in the current view without a requery.
The search edit pollution was noticeable as therube predicted.


(Even the same way that, Search for {Name|Size|Date...} does, I don't care for that either.)
Ideally, a filter for each column would be nice.
It is on my TODO list (column header split buttons).


X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" sort:size find-dupes: sort:dm find-dupes:
The following should now give the desired results:
X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" dupe:size dupe:dm

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Thu Jan 20, 2022 7:57 am

Awesome new shiny tools, @void! I can't wait to try them out after I've finished playing with the other new shiny tools.

A common request around duplicate finding is, the ability to make each object conforms to multiple <property-name> matches, and not that some objects have duplicate names with another object, while also having duplicate sizes with entirely different objects of different names, etc. I guess this type of request would have to be called a serial or cascading duplicate search. Right now, object filtering happens in parallel and without an attention span for the previous dupe matches.

Is this something you'd consider poking at?

dupe:name;size;dm

EricB
Posts: 40
Joined: Wed Jun 26, 2013 8:56 am

Re: Everything Context menu find duplicates in selected folder

Post by EricB » Thu Jan 20, 2022 8:12 am

I would second that! A duplicate subsearch for drilling down would be awesome.

Another thought: often I'm looking for Name duplicates for ebooks (epub format, which is actually a zipfile). Sometimes only a single epub property is different (the .opf file in the epub, which contains the book description/summary). This leads to minimal file size differences, most of time only 0,5 - 1k. Would it be possible to add a modifier to a duplicate search to allow for such a defined file size difference?

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Thu Jan 20, 2022 9:35 am

From what you described, I don't think that it would be any more revealing than a simple name duplicate search. There's no way to tell whether that 1,000 bytes difference is a change in the book description/metadata, or the book itself.

What I would do instead is bulk-extract all of your .epub files using 7-zip -> Extract to "*\" so that the contents of each book is extracted to its own individual folders. Then use Everything to locate exact duplicates. This is revealed by the same filenames, filesizes, between the different folder Paths. You can also add a SHA-1 hash column to spot exact content duplicates without relying on names and sizes alone.

EricB
Posts: 40
Joined: Wed Jun 26, 2013 8:56 am

Re: Everything Context menu find duplicates in selected folder

Post by EricB » Thu Jan 20, 2022 10:53 am

Well, I know from experience that if the size difference is only so little, the difference in content of the zip would be negligible (for this specific case of ebook comparison). It would make it easier to inspect name duplicates; whenever there is such a small size difference, it is safe (in my experience) to consider it as a duplicate and remove it. If the size difference is more than that, further inspection is needed. I use Total Commander for that, you can easily compare the contents of two zip files using the SyncDirs feature. No need to unpack. Works for many archive types.

I already implemented the hash method you describe, albeit using MD5. I added it as a property to the index, so it is immediately available at all times. This indeed allows me to filter out files that are the same in content but have a different name.

Further explanation: the small size difference for ebooks use case stems from previous use of Calibre. It updates some of the ebook properties, causing slight changes in the original epub file. A tad clumsy when you are in the process of deduplicating and ordering your collections.... I found that the combined use of Everything and Total Commander works better in that respect. I might import everything again in Calibre after all that work is done (it is a good library program for sure).

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Thu Jan 20, 2022 10:54 am

dupe:name;size;dm
I will consider this.
Thank you for the suggestion.

therube
Posts: 3423
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything Context menu find duplicates in selected folder

Post by therube » Thu Jan 20, 2022 9:38 pm

bulk-extract all of your .epub files using 7-zip -> Extract to "*\"
Something similar from a command line:

fasti7.bat:

Code: Select all

777 x %* -o*  %2 %3 %4


:: with %2 %3, you can do something like
:: 777 x *.zip  -o*  *.exe  *.chm
:: which would ONLY extract .exe & .chm
:: files from the .zip, rather then (by
:: default) ALL files...

:: like for with sysinternal files, you'd
:: want the .exe & .chm, but nothing else
("777" is simply a renamed 7z.exe)
("fasti7" is named based on an old DOS tool, FASTI - Easy ZIP, ARJ, LZH unpacker creates directories for contents of each archive.)

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Fri Jan 21, 2022 6:14 am

Everything 1.5.0.1297a adds support for dupe:<property1;property2;property3>

For example:

dupe:size;dm;name
dupe:size;sha-256

Everything will find duplicates for the first property before gathering any other properties. (so the sha-256 property will only be loaded for files with the same size)
Up to a maximum of 3 properties can be specified.


dupe:

therube
Posts: 3423
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything Context menu find duplicates in selected folder

Post by therube » Fri Jan 21, 2022 6:55 pm

dupe:size;dm;name
That is a global scope (unlike say, Find Name Duplicates)?

Maybe not, but...

NOT'ing is not allowed?

Code: Select all

dupe:size;dm;!name
---

But...

OK, yes it is, I think.
But then you can do something like 'Find Size Duplicates' to filter out the global-ness.
(Depending what you do, how you go about it, it can be a bit confusing...)

wrisco
Posts: 7
Joined: Wed Jan 19, 2022 6:43 pm

Re: Everything Context menu find duplicates in selected folder

Post by wrisco » Fri Jan 21, 2022 7:19 pm

So fast! This works great: dupe:size;dm
therube wrote:
Fri Jan 21, 2022 6:55 pm
NOT'ing is not allowed?
Yes, I noticed that too, would be great.
I'm thinking of a way to find duplicates of size and date modified, excluding the ones that are duplicated on the same drive.

Code: Select all

regex:path:^([a-z]) dupe:size;dm !dupe:size;dm;regular-expression-match-1
or like @therube said:

Code: Select all

regex:path:^([a-z]) dupe:size;dm;!regular-expression-match-1
I wonder if this would be possible..?

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Sat Jan 22, 2022 1:55 am

Thank you for the suggestions!

Everything 1.5.0.1298a adds support for !

For example:
dupe:size;name;!dm
(Find files with the same size and name, but with a different date modified)



I also fixed the sorting after using dupe:xxx;yyy;zzz
Previous version were resorting by xxx only.
Everything will now sort by xxx, then yyy and then zzz.

I also fixed the case: search modifier.
This was previously not working at all.
By default Everything will perform a case-insensitive dupe search.



You can now also specify the sort order with -ascending/-descending suffixes, for example:

dupe:size-ascending;name-descending
The order does not affect which results are returned, only the order in which the results are displayed.



regex:path:.:(.*) dupe:size;dm;!regular-expression-match-1
will now work as expected.



Currently, Only one dupe:xxx;yyy;zzz command can be used.
I have put on my TODO list to allow multiple dupe:xxx;yyy;zzz commands.

The right click column header -> Find duplicates does not affect the initial query, this simply applies the dupe command on the current results.



dupe:

jimspoon
Posts: 72
Joined: Tue Apr 26, 2011 11:39 pm

Re: Everything Context menu find duplicates in selected folder

Post by jimspoon » Sat Jan 22, 2022 1:07 pm

It would be great to have the ability to limit the results to files in a certain directory tree, plus all their dupes whereever else they may be, as when we were discussing "dupesizewithpath:" Maybe have a modifier with a name like "dupewithpath:" or "dupeofpath:" or "comparetopath:".

wrisco
Posts: 7
Joined: Wed Jan 19, 2022 6:43 pm

Re: Everything Context menu find duplicates in selected folder

Post by wrisco » Sat Jan 22, 2022 11:47 pm

void wrote:
Sat Jan 22, 2022 1:55 am
Everything 1.5.0.1298a adds support for !
I also fixed the sorting after using dupe:xxx;yyy;zzz
This is great! Works really well and it's super useful!

Code: Select all

regex:path:^([a-z]) dupe:!size;!dm;!regular-expression-match-1
This seems to do what I want; show files and folders that are not duplicated on another drive.

eswul62
Posts: 46
Joined: Wed Jul 31, 2013 6:07 am

Re: Everything Context menu find duplicates in selected folder

Post by eswul62 » Fri Jan 28, 2022 3:15 pm

Super!

As always, thank you very much indeed.

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Thu Feb 03, 2022 7:08 am

void wrote:
Sat Jan 22, 2022 1:55 am
Currently, Only one dupe:xxx;yyy;zzz command can be used.
I have put on my TODO list to allow multiple dupe:xxx;yyy;zzz commands.
milestone. 1299a - added support for multiple dupe:xxx;yyy;zzz commands.

@void, Question, what is the correct way to break out of "dupe mode" back to the query? The method I've been using is toggling Search -> Match Case both on/off again.

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Thu Feb 03, 2022 7:14 am

If you are using Column Header -> Find Duplicates..., double click DUPE in the status bar to clear the dupe command.

When using dupe: in your query, I usually just add a space to the end of the query string to update the results.

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Thu Feb 03, 2022 7:28 am

Thanks! :D

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Tue Feb 22, 2022 6:24 am

Everything 1.5.0.1301a removes the old dupe:, sizedupe: behavior of searching the entire index.

These functions will now find duplicates in the current results.
I don't think anyone will miss the old behavior.
-If you want the old behavior just use dupe:/sizedupe: on its own without any other search terms or filters.

Added name-dupe: search function.
Added path-dupe: search function.
Improved finding duplicates from the advanced search dialog.

namedupe: and sizedupe: will now combine into dupe:name;size

raccoon
Posts: 683
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything Context menu find duplicates in selected folder

Post by raccoon » Tue Feb 22, 2022 6:49 am

clarity. is the old behavior of dupe: and sizedupe: removed or just deprecated from documentation? Or when you say "on its own" do you mean without any other search terms (path filtering)?

void
David Carpenter (Developer)
Posts: 9370
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything Context menu find duplicates in selected folder

Post by void » Tue Feb 22, 2022 6:56 am

In other words, the old behavior is removed.

The old behavior was to search for duplicates in the ENTIRE index, not the current results.

So a search like:

C:\windows\ dupe:

would return unexpected results, because the 'duplicated' results would be from the entire index, not under C:\windows.



Now, the behavior is to find the duplicates within the current results.

If you specify dupe: by itself, with no other search terms or filters, it will give the old behavior, that is name duplicates in the ENTIRE index.

Post Reply