Using network share index forces refresh

[TVOps]

I have an extremely large (500TB) network share mapped and use Everything to index it. I also use the HTTP server to allow users to search the share. I have the index re-fresh once a day at midnight (since while Everything updates, search is unavailable).

My problem is that occasionally, Everything or the system itself will crash and upon restarting Everything, it refreshes the index, meaning the HTTP search is offline for an hour or so while the index refreshes. Is it possible to turn off index refreshing EXCEPT for the update time set in Options -Indexes - Folders?

[void]

Please try starting Everything and let it index the folders.
Once it has completed indexing the folders, exit Everything completely.
This will force Everything to save the Everything.db to disk so the next time you run Everything, Everything will try to load the database from disk instead of re-indexing everything.

I would recommend removing any NTFS indexes from Tools -> Options -> NTFS as these indexes could be causing Everything to rebuild the entire database.

The scheduled updates in Tools -> Options -> Folders occur in the background.
You should be able to use the HTTP server while this occurs.
Starting Everything after a crash may start one of these scheduled updates.

[TVOps]

I have tried letting Everything load and index, then exit. Upon restart, everything does load from the .db file. However, almost any change I make to the Indexs -> Folders tab results in an entire database refresh.

For example, I have two network drives mapped:
\\ddn-cifs\500TB
\\filer\100GB

Attempt to monitor changes is off, schedule is set to update at midnight for both folders.

If I add a new share or remove \\filer\100GB, the entire index refreshes, meaning it has to rescan that 500TB folder.
As for those scans occurring in the background, the HTTP server does continue to serve the page, but it only serves the custom everything.gif I have at the top and a blank search field which displays nothing. A search of * returns 0 Results until the index finishes refreshing.

I have had "Include in database" unselected from NTFS tools page, is that the only way to remove the local index?

[void]

I have tried letting Everything load and index, then exit. Upon restart, everything does load from the .db file. However, almost any change I make to the Indexs -> Folders tab results in an entire database refresh.

This is the expected behavior.
I hope to make adding and removing folder indexes nearly instant for the next update.

I have had "Include in database" unselected from NTFS tools page, is that the only way to remove the local index?

Yes.
Please make sure auto include new fixed volumes and auto include new removable volumes are disable too.

[zybexXL]

I'm testing Everything on our SAN storage... congratulations, this tool is excellent
I also have the same issue as the OP. I'm indexing several SAN units with a total of about 100 TB, and about 10 million files. This index takes about 2 hours (don't ask). I have a scheduled update for 4am, but I'd like the current index to be available while a new one is built.

Changing any indexing options also causes the index to be rebuilt, which again makes it unavailable for 2 hours. All NTFS volumes are disabled, it's just network shares.

RAM usage after indexing is about 1.2 GB, which is fine. I'm using 1.4.1.809b x64.

Thank you!

[zybexXL]

May I also suggest an alternative/faster mode for network shares scanning ? (perhaps a new option "use fast Network folder scans" or something).
Enumerating folders with thousand of files is very slow over CIFS/SMB protocols. Windows does many I/O calls to get info for each file in a given network folder, making it extremely slow. To work around this, I implemented an algorithm in some of the Perl/Python scripts we use to cache folder contents based on the modified date of the folder:

Foreach folder:
- read folder MTIME
- if we have the folder in DB, compare with stored MTIME; if it's same, then contents are the same (POSIX requirement) - no need to update!
- if MTIME changed, then read new contents from network as usual, and update DB
- recurse into ALL subfolders (regardless of contents having come from DB or not). Same MTIME guarrantees that content of a folder is same, but tells nothing about subfolders, so we still have to recurse.

This improves scanning time of our volumes from 2 or 3 hours to just a few minutes, and greatly reduces I/O on the SANs.

[zybexXL]

... I just realized I woke up a very old thread, sorry about that

[void]

I've added the ini option folder_update_rescan_asap to Everything 851b or later.

Setting this option to non-zero will rescan as soon as possible after a scheduled rescan is missed.
Setting this option to zero will only rescan at the scheduled time.

You can toggle this function within Everything, however you will need to restart Everything for it to take effect.

To disable rescanning on startup:

• In Everything, type in the following search and press ENTER:
  /folder_update_rescan_asap=0
• Restart Everything.

May I also suggest an alternative/faster mode for network shares scanning ?

I'll look into improving the performance of folder indexing in Everything 1.5.

[crpig]

Hi void,

I'm already setting add 「folder_update_rescan_asap=0」 to Everything.ini.
But every time I open everything first time, it will force rescan all my folders.

[void]

Please make sure Everything is storing your settings and data:

• In Everything, from the Tools menu, click Options.
• Click the General tab on the left.
• Check Store settings and data in %APPDATA%\Everything.
• Click OK.

[crpig]

Hi void,

>> Check Store settings and data in %APPDATA%\Everything.
This is already checked.

Here is my 'Everything.ini' in %APPDATA%\Everything.
Every time I restart everything, it will re-scan folder list which is in Line 446 of Everything.ini.

[NotNull]

You seem quite knowledgeable, but just to make sure:
Isn't this the the NTFS volumes - "C:","D:","F:","H:" in your case - updating?

When Everything isn't running, it also has to update it's index with the changes made to these NTFS volumes.

Your Folder Indexing settings look OK.


BTW: I assume you replaced your ETP Server password with a fake one? If not, now you will have to change it

[crpig]

Hi NotNull,

>> BTW: I assume you replaced your ETP Server password with a fake one? If not, now you will have to change it
Yes, I replace it with '12345678' before I upload my settings. Anyway, thank you for your kindly reminding.


>> Isn't this the the NTFS volumes - "C:","D:","F:","H:" in your case - updating?
>> When Everything isn't running, it also has to update it's index with the changes made to these NTFS volumes.
>> Your Folder Indexing settings look OK.
No, my question is the folders here:
==> folders="\\\\uranus\\Speed","X:\\"

Every time I restart everything after I restart my computer, It will re-scan folder '\\uranus\Speed', which is a quite large network filesystem.(samba)
Since this will take a lot of time before I can use everything.
So I'm wondering How can I disable the startup scan, cause I only need these folders to be re-scanned at 4:00am everyday.

[NotNull]

>> Isn't this the the NTFS volumes - "C:","D:","F:","H:" in your case - updating?
[..]
No, my question is the folders here:

OK; this was just to eliminate the obvious (didn't expect the answer to be Yes)


Please don't make changes to your Everything configuration (yet) as that will make troubleshooting much harder.

Are you running Everything with a different account?
(For example: you enabled Menu:Tools > Options > General > Run as administrator)
In that case Everything.ini is probably in a folder different from %APPDATA%.

You can check the currently active rescan setting with:

• Enable the statusbar (if needed): Menu:View > status bar
• Enter
  /folder_update_rescan_asap
  in the Everything search bar and press ENTER
  The statusbar will show the current state for a couple of seconds on the left (maybe right in the Chinese version?)

- or -

• Enter
  about:config
  in the Everything search bar and press ENTER
  This will show you the currently active INI file entries
• Select Menu:File > Save As to show the path where the INI comes from.
  Note: saving the file has no use, as it will be overwritten.

Some questions

Every time I restart everything after I restart my computer, It will re-scan folder '\\uranus\Speed'

1. Not your X:-drive? (or is X: offline?)

A test:

• Let Everything complete the scanning of \\uranus\Speed
  Everything has now completed it's (scheduled) scan and shouldn't scan again until 04:00
• Exit Everything completely (system tray: right-click Everything icon and choose Exit
• Start Everything again

2. Does Everything now scan \\uranus\Speed ?
(don't test this at 4 AM in the morning! )

3. Does Everything behave different when you reboot after exiting Everything in the test above?

[void]

Everything 1.5 alpha adds support for fast network rescanning.