High DB File and Failure at the end of Indexing

Discussion related to "Everything" 1.5 Alpha.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

High DB File and Failure at the end of Indexing

Post by nesedenyana »

Hi,
The db file increases in size exceptionally and at the end of indexing (with contents) it fails with out of memory file. My PC is 16 GB Ram and 500 GB SSD.
https://prnt.sc/Fc22NylBHbZD
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Everything stores content in RAM.

I recommend not using content indexing, storing your content on a good NVMe drive and just searching the files from disk with content:
Everything will max out your NVMe SSD read speeds (3000+ MB/s)



Please try reducing the number of files you are content indexing:
I recommend indexing only user documents.
  • In Everything 1.5, from the Tools menu, click Options.
  • Click the Content tab on the left.
  • Change the Include only folders to:
    %userprofile%\Documents
  • Change include only files to:
    *.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx
  • Click OK.
To check how much memory Everything is using to store file content:
  • In Everything, from the Tools menu, under the Debug submenu, click Statistics.
  • The total content size is included in Database -> File data size.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Hi,
I am not sure if these solves my problem. I have a 500 GB PC that' half of it PDFs. I had already the PDF, DOC, XLSX filter on. I have also changed it to index only the user profile. But, already all my files are there.

With my current PC conditions: Is there any other settings that I can tweak aside from reducing the number of files?
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Please see that the software doesn't obey the folders selected.

https://prnt.sc/uwkFyyYtKuF4
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Also, it doesn't obey the "Not Index" order :)
https://prnt.sc/sGdiV4lb1tQ-
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Everything will show the current file being property-indexed at the top right.
Hovering over this indexing text will show the full path and filename.

This tip will also show the current file being property indexed (not just content indexed).
Are you indexing any properties under Tools -> Options -> Properties?

Technically, content is just a "property" to Everything and all properties are listed here in this tip.



I recommend not using content indexing, storing your content on your SSD and searching these PDF files with the content: search function.

Indexing all these PDF files, then storing this index on disk is not going to offer any real performance increase.



Please consider indexing these PDF files with Windows Search and searching your Windows index with si:
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I might be misunderstanding how Everything works: So, can I use content:"XXX" search methodology without content indexing? I thought that content indexing was already for this? What it adds?
I already use content:"xxx" or content:<aaa bbb>
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Yes, you can use content: without content indexing.

With no content indexing:
Using content: will just go straight to disk to search your content.



With content indexing:
Everything will store content in memory for instant content searching.
You'll need a bucket load of RAM to enable content indexing for a lot of files.
Everything is only really designed to index a maximum of 1GB of text.
Only indexed content is searched when using content:
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

This is the property page: https://prnt.sc/Nh2sInf0oQkW
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I understand, but why content indexing is not working for me?
With a 500 GB SSD and 16 GB Ram, it doesn't work, should it only work in the gamer PCs?
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Everything will be trying to read the # property for every single file.

The current file being indexed for # will also be shown under the content page (top right).
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I don't remember putting # there, but now removed. And trying to content index again,at least PC seems calmer during indexing. I will inform how it goes without properties.
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

I understand, but why content indexing is not working for me?
With a 500 GB SSD and 16 GB Ram, it doesn't work, should it only work in the gamer PCs?
From my limited understanding of your setup, it sounds like you are trying to index 250GB of PDFs with only 16 GB RAM.
You would need about 250GB of RAM to index that much PDF content with Everything.
Granted, your PDFs will not be text only and contain images, the memory usage will still be insanely high.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

It's the same thing again.

I have tried to content index a folder with 72 GB of content (PDFs).

During indexing: https://prnt.sc/pSyER5zo9PoE

After some point, the out of memory failure. I couldn't get a screenshot in each time, because it fails itself and closes, (and somehow Teams too.)

Do you have any idea? Because I was happily using the content search function for now. I am ready to submit any data you need.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

You would need about 250GB of RAM to index that much PDF content with Everything.
Really? Is this how it works? So, for a typical PC with 16-32 GB ram, content index will only work for 16-32 GB of files?
Also: It worked before!!
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Yes, index content in Everything is stored in RAM.
Also: It worked before!!
Content indexing worked before? or content searching worked before?

Maybe you have a PDF file that is returning bogus content causing Everything to read a massive amount of text content that isn't there.
Does Everything appear to get stuck on a particular PDF?

16GB is a very large amount of text.. I would assume that these PDFs are not just all text and contain images.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Of course, they are technical books and reports. With many many images.
Content indexing worked before. Before, I wasn't specifying a specific folder, it was indexing everything that are present in the search folders. It worked smoothly.
I cannot understand why it fails now. Maybe I can go back to an older version. I can't understand what changed.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Another clue: I have 500 GB PC but it says 1.35 TB here, I don't know why. No external SSD is connected.
https://prnt.sc/mx-rUy9jeHxT
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Another note: Without content indexing, content:"xxx" search results in downloading online files of Onedrive.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I might have underestimated the statement that Everything indexes the files in RAM. Because, what I can see is the ram used by Everything increases as the indexing continues, and when the file size reaches to something close to my ram capacity, everything starts to fail.

So, this is really interesting. Do we say that Everything will store all the files in RAM always? Don't we have any other option? This means that we can content index only a small number of files. It's not very relevant in that case, right?

Searching without content indexing: I didn't understand what you propose, because if I search without content indexing, Everything just starts to download all the online files not stored in my PC (Onedrive). So, I cannot use that, and I cannot imagine how effective it can be.
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

I suspect there's an issue with the PDF iFilter and Everything is treating some PDF files as text/plain.

The next alpha update will never treat PDF files as text/plain when content indexing.



Could you please check the largest indexed content size for some files:
Copy and paste the following into your Everything search box:
addcol:col1 col1:=len($content:) sort:col1-descending

This will list all files by indexed-content size from largest to smallest.
Please compare the size shown in the Column 1 column with the size of the file.
The column 1 size should be much smaller than the size of the file.



Please make sure Everything is installed correctly and running as a standard user:
  • In Everything, from the Tools menu, click Options.
  • Click the General tab on the left.
  • Check Store settings and data in %APPDATA%\Everything.
  • Uncheck Run as administrator.
  • Check Everything Service. (Please make sure this is tick-checked and not square-checked)
  • Click OK.
  • Exit Everything (right click the Everything tray icon and click Exit).
  • Restart Everything.

So, this is really interesting. Do we say that Everything will store all the files in RAM always? Don't we have any other option? This means that we can content index only a small number of files. It's not very relevant in that case, right?
Everything will store all the file content text in RAM.
An alternative to content indexing is to just search the content from disk when needed with content:
Indexing over 1 GB of text is not really recommended with Everything.
1 GB of text is a lot of text.


if I search without content indexing, Everything just starts to download all the online files not stored in my PC (Onedrive). So, I cannot use that, and I cannot imagine how effective it can be.
Please try excluding your OneDrive folder by including the following in your search:
!C:\Users\*\OneDrive\**

If you find this useful, please consider creating a filter from Search -> Add to filters.

Filters can be activated from the Search menu, Filter bar (View -> Filters), right clicking the status bar, filter macro or filter keyboard shortcut.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Please compare the size shown in the Column 1 column with the size of the file.
You can see that the sizes are exactly same. (The left is normal sizes, right is the code you sent)
https://i.imgur.com/msaOwkK.png
Please make sure Everything is installed correctly and running as a standard user:
I maded the checks. Everything as you said.
Indexing over 1 GB of text is not really recommended with Everything.
This is different. I have 100-200 GB PDFs, but of course I don't know how much text does that mean. But I am 100% sure it was working before, and now it doesn't. It might be a broken PDF as you said, but the figure above shows that Everything just uses the same file size.
Please try excluding your OneDrive folder by including the following in your search:
!C:\Users\*\OneDrive\**
That doesn't help me. I need to include Onedrive.
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Thanks for the screenshot.

The screenshot appears to be missing the "Column 1" column.
The new "Column 1" column should be added to the right. You may need to scroll it into view.

Please compare the size column with the "Column 1" column, with the following search:

addcolumn:column1 column1:=len($content:) sort:column1-descending

Are the values huge in column 1? (equal to or larger than the file size in bytes?)
-If so, this would indicate an issue with the PDF iFilter.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

See the screenshot. Does this give us hope about future update?
https://i.imgur.com/cQrntOd.png
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Thank you for the screenshot.

This confirms Everything is treating PDFs as text/plain.

I'm still investigating why this is suddenly occurring..
(The system is basically saying there's no PDF iFilter)

The next alpha update will never treat PDFs as text/plain.

I'll have a fix soon.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Thanks a lot! Waiting impatiently. In these short months content searching became crucial part of my daily work as a design engineer.
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

What PDF viewer are you using nesedenyana? Have you changed it recently?
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Oh, does that affect? Because, I changed a lot.
In the recent months, I have changed between Adobe Pro, PDF X-Change and Sumatra PDF. I changed it back from Sumatra to Adobe couple days ago.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I noticed Windows search also doesn't search inside the PDFs now. It just returns the results from Words and Excels. Will report back.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Here is the problematic setting, I believe. But, I am 100% sure that it wasn't the case before. It can be something I have done at last couple of weeks that changed this settings.
Now, I will wait for couple of hours for Windows to index and see if it affects the Everything.
https://prnt.sc/VR_5GD1F30Q-
horst.epp
Posts: 1453
Joined: Fri Apr 04, 2014 3:24 pm

Re: High DB File and Failure at the end of Indexing

Post by horst.epp »

nesedenyana wrote: Sun Jan 22, 2023 1:04 pm Here is the problematic setting, I believe. But, I am 100% sure that it wasn't the case before. It can be something I have done at last couple of weeks that changed this settings.
Now, I will wait for couple of hours for Windows to index and see if it affects the Everything.
https://prnt.sc/VR_5GD1F30Q-
That doesn't effect the Everything index
as long as you don't use si: to query the Windows index.
Everything uses the installed PDF iFilter to get the PDF content for its own index.
What PDF search filter do you have installed ?
You can check it with the NirSoft tool SearchFilterView
https://www.nirsoft.net/utils/search_filter_view.html
It look like this:
Screenshot - 22.01.2023 , 14_55_05.png
Screenshot - 22.01.2023 , 14_55_05.png (55.73 KiB) Viewed 51123 times
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Thanks for that.
Very interesting, I have around 5 PDF viewer including the paid Adobe Pro, and no iFilter.
https://prnt.sc/XFAVs068nVfP
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Another interesting observation I think: When I checked the registry key in HKEY_CLASSES_ROOT\.pdf\PersistentHandler as recommended by Adobe, the key that I found is explained in the link as plain text.
{5E941D80-BF96-11CD-B579-08002B30BFEB}

https://strontic.github.io/xcyclopedia/ ... 0bfeb.html
horst.epp
Posts: 1453
Joined: Fri Apr 04, 2014 3:24 pm

Re: High DB File and Failure at the end of Indexing

Post by horst.epp »

nesedenyana wrote: Sun Jan 22, 2023 3:15 pm Thanks for that.
Very interesting, I have around 5 PDF viewer including the paid Adobe Pro, and no iFilter.
https://prnt.sc/XFAVs068nVfP
Using the Nirsoft tool you can set the required PDF iFilter
by removing the extension from the current one
and add the extension to another one.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Thanks..
I have added pdf to the PDF-XChange's iFilter. I don't know why, but even though I reinstalled the Adobe Pro, I cannot see their iFilter here.
https://prnt.sc/hpUFtTn0weyH
And also I don't know why once installed and now gone ilovepdf is still shown in the pdfs as TypeName.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

This is good news, right?
Image
horst.epp
Posts: 1453
Joined: Fri Apr 04, 2014 3:24 pm

Re: High DB File and Failure at the end of Indexing

Post by horst.epp »

nesedenyana wrote: Sun Jan 22, 2023 4:32 pm Thanks..
I have added pdf to the PDF-XChange's iFilter. I don't know why, but even though I reinstalled the Adobe Pro, I cannot see their iFilter here.
https://prnt.sc/hpUFtTn0weyH
And also I don't know why once installed and now gone ilovepdf is still shown in the pdfs as TypeName.
The TypeName has no relevance at all.
Its a relict from the last PDF software installation
and should be changed when you assing another software to open PDFs.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Hi again.
All problems are solved after I made sure iFilter is added using NirSoft. It must be a mistake due to some PDF viewer installations that my iFilter was set to plain text. After sorting out the iFilter issue, everything worked great in Everything. I have over 13000 documents indexed and works flawlessly.

So, for the next release, should there be a automatic check for the iFilter?
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Everything 1.5.0.1336a fixes an issue with Everything treating PDF files as text/plain when no iFilter is installed.

I will consider falling back to known iFilters. (even when they are not installed correctly)

For now, Everything will use the system installed iFilter.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Indexing significantly slowed down after this update. I am waiting to test the new update, but it's been a half-day since I started while the same number of files were done in half an hour before the update.
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

As you can see, Everything is indexing files that shouldn't index (content)
Image
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

As you can see, Everything is indexing files that shouldn't index (content)
Where do you see that?

The column 1 value will be 0 for files that Everything doesn't content index.

Everything will show content and properties being indexed in the top right status text.
(Indexed properties are found under Tools -> Options -> Properties)
(Technically, content is just a property to Everything)
It looks like Everything should be indexing PDF files from your include filters.

Perhaps the change in PDF iFilter is making Everything take longer to index.
Does Everything appear stuck on a particular PDF file? or is the PDF indexing just slow?
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Oh yeah, content is 0, understood.

No, the process moves on, but much much slowly. Since I close my PC at the end of day, I havent been able to complete one single indexing yet. Every day, we start from scratch.
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

What is reported under Tools -> Debug -> Statistics -> Build?

and

What is reported under Tools -> Debug -> Statistics -> Save?
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Image
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Taken from another time where content indexing is at 61%.
Image
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Thank you for the statistics.

The stats show Everything is successfully saving the database to disk and loading the database from disk.
There could be an issue with a volume being flagged as out of date.

The next time Everything performs a reindex of your content, could you please:
  • Copy and paste the following into your Everything search box:
    /restart-now-debug-log
  • Press ENTER in your Everything search box.
  • Everything will restart immediately without saving your database to disk and restart in debug logging mode.
  • Once Everything starts reindexing your content, could you please:
  • From the Tools menu, under the Debug submenu, click Stop Debug Logging.
    ---This will open your %TEMP%\Everything Debug Log.txt in Notepad---
  • Could you please send this file to: support@voidtools.com
Privacy
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

Sent
void
Developer
Posts: 16974
Joined: Fri Oct 16, 2009 11:31 pm

Re: High DB File and Failure at the end of Indexing

Post by void »

Thank you for the logs.

The debug logs show the USN Journal on your C: drive is filled between closing Everything and restarting Everything.
This will cause Everything to see the C: drive as a new drive and thus rescan all your properties/content.



What are the USN Journal settings for your C: ?
-To view these settings:
In Everything, from the Tools menu, click Options.
Click the NTFS tab on the left.
Select your C: drive.
What is shown in this screen?
-Please try increasing your USN Journal maximum size to: 131072 KB



Please make sure you don't have any stuck Windows updates.
-I've seen stuck updates make millions of file system changes on reboot.



-Please check your Index Journal in Everything.
This will show all changes to your file systems.
Is there something spamming your file system?
nesedenyana
Posts: 58
Joined: Mon Sep 19, 2022 10:38 am

Re: High DB File and Failure at the end of Indexing

Post by nesedenyana »

I have Sharepoint connected to my system which is also indexed by Everything. It gets updated quite a lot by my colleagues etc.
You can see the screenshot below, I increased the size. I don't know where I canssee the journal.
Image
Post Reply