Efficiency of Property Index on Large Datasets

[twood]

Hihi, I'm wondering how to how keep metadata indexing consistently efficient over large filesets. Huge preface of "I know I'm making a lot of choices that will make it extremely slow", my issue is about relative speed, not absolute speed.

What I'm Doing:
Property index of all non-C: files for Audio Format, Video Format, Owner, and SHA512

Specs:
16 GB RAM (Everything sits around 15-20% usage)
~3.8 million files
~30tb data
Other than 300k local files (which are not being metadata indexed), all others are on network drives (see my preface)
2.5 million have successfully had their properties indexed and written to disk, 1 million have not and need to, and 300k are excluded.

Why:
Deduplication and archiving, mostly.

Perceived Problem:
Indexing starts out relatively fast (A few hundred files a second at outset, down to 15% progress daily over the next five or so days), but around the 70% mark drops off considerably down to 1-3% daily. Average filesize of indexed and unindexed files does not differ significantly, nor has my machine maxed out its RAM.
Among other things, I wonder how much is because of machine uptime without a restart? I don't restart while the index is running as it's liable to lose considerable (~30%) progress when I do. Even under optimal circumstances (no loss) once the slowdown hits the index could take up to 1-1.5 months of continuous run to complete.

Questions Towards a Solution:
- Acknowledging that what I am doing will necessarily be very slow in general, are there steps I can take that will prevent the considerable dropoff in pace? (I do not have access to the file server to eliminate the network bottleneck, unfortunately)
- Is there a way to manually instruct Everything to index ONLY files missing a value for the given property, instead of checking the property for all files? E.g. I only want it spending resources to check the 1 million that haven't been covered yet, but it's burning significant time on the 2.5 mil that have.
- Is there a way to force Everything to write partial index progress to disk, similar to `/save-db`? If I could trust that partial progress was truly saved before closing/restarting/whatever else, would alleviate a lot of issues.

Thanks a ton

[void]

- Acknowledging that what I am doing will necessarily be very slow in general, are there steps I can take that will prevent the considerable dropoff in pace?

Indexing SHA512 will be slow and inefficient to maintain.
Avoid indexing SHA512 in Everything, particularly on system drives and volumes that change often.
Use sidecar files or .metadata.efu to store SHA512.

Indexing Audio Format, Video Format, and Owner will be fast and efficient.

- Is there a way to manually instruct Everything to index ONLY files missing a value for the given property, instead of checking the property for all files?

There's no way to tell Everything to stop or cancel indexing properties for certain files.
To re-index properties values for files that have missing values:
Search for these files, eg: !sha512:
Select all files and press Ctrl+F5 to reindex properties for the current selection.

Everything will index properties from Ctrl+F5 with higher priority.

- Is there a way to force Everything to write partial index progress to disk, similar to `/save-db`?

The property index is stored in your database, so saving the database to disk will save property indexing progress.
Besides

/save-db

, exit Everything (File -> Exit) and restart Everything.

[twood]

Thanks for the quick reply!

Avoid indexing SHA512 in Everything, particularly on system drives and volumes that change often.
Use sidecar files or .metadata.efu to store SHA512.

For long-term integrity checking and similar archival purposes, sidecar files are indeed the intended direction long-term.With the resources and time available to me now, and again acknowledging that I know it will make the index slow in general, SHA512 indexing in Everything is my most reliable means of perpetually and carefully deduplicating across the fileset. We have many files with extremely minor differences, deduping by name/size/extension is not specific enough for our usecases, so the linear slowdown is a necessary concession. My curiosity is less about search velocity and more about deceleration (that 15%/day to 1-3%/day behavior, if there's any insight there)

Everything will index properties from Ctrl+F5 with higher priority.

This doesn't seem to be behaving as expected? It appears to be doing *something* (index progress now occasionally jumping a significant percentage very quickly, which has not been observed before), but run against some missing-property files for testing they definitely don't appear to be prioritized. If you'd prefer, I can open a separate forum thread for that, as it is separable from the topic of the original question.

[twood]

Adjustment to my prior statement about prioritization not working—it did get there eventually, but I'm uncertain on how the feature works. My understanding was that Ctrl+F5 would prioritize selected files over others, but still index them all? The behavior seems to be that it skipped over unselected files (updating the progress bar in the process), till selected files were all that was left (while still taking its time on some subset of unselected files on the way). Very curious to better understand what expected behavior looks like?

[void]

There's two indexed property request queues:

Background queue
The initial requests go here.

Foreground queue
Ctrl + F5 requests get added to the end of this queue.
Modified files get added to the end of this queue.

It's possible there was modified files, which will update first.
It's possible there was an older Ctrl + F5 request, which will be processed first.



The next update will add Ctrl + F5 requests to the start of the foreground queue.