Reparse Target not getting populated in some cases

[kyle]

I added the Reparse Target and Reparse Tag properties and even rebuilt the index, but it still doesn't populate the Reparse Target in some cases.

If I use reparse-tag: as the search, it shows 556 results but only about half of those (287) have a Reparse Target populated.

As a test, I created a Directory Junction from C:\Downloads pointing at D:\Downloads. As you can see, Everything shows the correct DL Attributes and has a Reparse Tag, but the Reparse Target is empty. You can also see that the File Explorer Properties correctly identifies the Junction target and so does PowerShell.



Am I doing something wrong, or is this just a bug in the alpha version?

[void]

Thank you for the bug report kyle,

Please make sure you are not indexing these properties.
(Please make sure these properties are not listed under Tools -> Options -> Properties)

Could you please send some debug output:

• In Everything 1.5, from the Tools menu, under the Debug submenu, click Start debug logging.
• Search for:
  exact:c:\downloads
• Make sure the Reparse Target column is shown.
• Press F5.
• From the Tools menu, under the Debug submenu, click Stop debug logging.
  --This will open your Everything Debug Log.txt in Notepad--
• Could you please post the output here.

From my testing, the Reparse Target is shown correctly for symlink (0xA000000C) and mount point (0xA0000003) tags.
It might be a permission issue, are you able to open C:\downloads from Everything?

[kyle]

I removed those properties from Tools -> Options -> Properties and rebuilt the index. Then I followed your steps and the Reparse Target is still not showing up. I verified that I am able to open that path from Everything. Here you go:

Code: Select all

2022-09-13 19:33:22.936: COMMAND 40036
2022-09-13 19:33:22.936: fileinfo clear all
2022-09-13 19:33:22.936: set sort 0 ascending 1 is valid 1
2022-09-13 19:33:22.936: already sorted
2022-09-13 19:33:22.936: finished sort, time taken 0.000022 seconds
2022-09-13 19:33:22.936: update selection 0.000008 seconds
2022-09-13 19:33:22.936: ready
2022-09-13 19:33:22.936: DB_WAIT: _db_ready_proc waiting for _db_sort_thread_proc...
2022-09-13 19:33:22.936: DB_WAIT: _db_ready_proc waited 0.000008 seconds
2022-09-13 19:33:22.936: back buffer grow 2560 x 26 in 0.000020
2022-09-13 19:33:22.937: new results 1
2022-09-13 19:33:22.937: ui->listview_was_focus_in_view 1
2022-09-13 19:33:28.822: MENUBAR ENTER OPEN
2022-09-13 19:33:28.823: MENUBAR POPUP 7
2022-09-13 19:33:28.824: WM_INITMENUPOPUP 00000000022300a5 0000000000000000
2022-09-13 19:33:29.983: WM_INITMENUPOPUP 0000000001aa016b 0000000000000004
2022-09-13 19:33:31.521: WM_UNINITMENUPOPUP 0000000001aa016b 0000000000000000
2022-09-13 19:33:31.522: WM_UNINITMENUPOPUP 00000000022300a5 0000000000000000
2022-09-13 19:33:31.522: MENUBAR LEAVE OPEN
2022-09-13 19:33:31.522: COMMAND 40210

[void]

Thank you for the debug logs.

There doesn't appear to be any errors thrown.

Everything is either not requesting the property correctly, or is seeing an empty reparse target.
I'll add some more debug output in the next alpha to try and catch the issue.

For now, could you please send some verbose debug output:

• In Everything 1.5, from the Tools menu, under the Debug submenu, click Start debug logging.
• From the Tools menu, under the Debug submenu, check Verbose.
• Search for:
  exact:c:\downloads
• Make sure the Reparse Target column is shown.
• Press F5.
• From the Tools menu, under the Debug submenu, click Stop debug logging.
  --This will open your Everything Debug Log.txt in Notepad--
• Could you please post the output here.

The verbose output should confirm the property is requested correctly.

[kyle]

Code: Select all

2022-09-13 21:33:18.687: COMMAND 40036
2022-09-13 21:33:18.687: fileinfo clear all
2022-09-13 21:33:18.687: event post: 00007ff7bfb267c0 0000000009eeb160
2022-09-13 21:33:18.687: create thread
2022-09-13 21:33:18.687: set sort 0 ascending 1 is valid 1
2022-09-13 21:33:18.687: already sorted
2022-09-13 21:33:18.687: finished sort, time taken 0.000018 seconds
2022-09-13 21:33:18.687: update selection 0.000011 seconds
2022-09-13 21:33:18.687: event post: 00007ff7bfb267e0 0000000009eeb160
2022-09-13 21:33:18.687: event post: 00007ff7bfb26770 0000000009eeb160
2022-09-13 21:33:18.687: event post: 00007ff7bfab2b40 00000000015770d0
2022-09-13 21:33:18.687: MSG OK
2022-09-13 21:33:18.687: MSG: 0000000000090bd0 1401 0000000000000000 0000000000000000
2022-09-13 21:33:18.687: TRAY 00001401 0000000000000000 0000000000000000
2022-09-13 21:33:18.687: update EVENTs
2022-09-13 21:33:18.687: EVENT: 00007ff7bfb267c0 0000000009eeb160
2022-09-13 21:33:18.687: EVENT: 00007ff7bfb267e0 0000000009eeb160
2022-09-13 21:33:18.687: EVENT: 00007ff7bfb26770 0000000009eeb160
2022-09-13 21:33:18.687: EVENT: 00007ff7bfab2b40 00000000015770d0
2022-09-13 21:33:18.687: ready
2022-09-13 21:33:18.687: DB_WAIT: _db_ready_proc waiting for _db_sort_thread_proc...
2022-09-13 21:33:18.687: DB_WAIT: _db_ready_proc waited 0.000006 seconds
2022-09-13 21:33:18.687: when ready 7 0000000000000000 0000000000000000
2022-09-13 21:33:18.688: back buffer grow 2560 x 26 in 0.000023
2022-09-13 21:33:18.688: new results 1
2022-09-13 21:33:18.688: listview message 0000004e 0000000000002719 000000000133e0f0
2022-09-13 21:33:18.688: listview message 0000004e 0000000000002719 000000000133e0f0
2022-09-13 21:33:18.689: ui->listview_was_focus_in_view 1
2022-09-13 21:33:18.689: DB MSG: 00007ff7bfb26770 0000000009eeb160
2022-09-13 21:33:18.689: _db_monitor_process_update_event_available_event_proc
2022-09-13 21:33:18.689: MSG OK
2022-09-13 21:33:18.689: MSG: 0000000000020c3c 0200 0000000000000000 00000000000a01b4
2022-09-13 21:33:18.689: UI message 00000020 0000000000020c3c 0000000002000001
2022-09-13 21:33:18.689: MSG OK
2022-09-13 21:33:18.689: MSG: 0000000000020c34 000f 0000000000000000 0000000000000000
2022-09-13 21:33:18.689: MSG OK
2022-09-13 21:33:18.689: MSG: 0000000000020c3c 000f 0000000000000000 0000000000000000
2022-09-13 21:33:18.689: MSG OK
2022-09-13 21:33:18.689: MSG: 0000000000020c2a 000f 0000000000000000 0000000000000000
2022-09-13 21:33:18.689: MSG OK
2022-09-13 21:33:18.689: MSG: 00000000000c0ae6 000f 0000000000000000 0000000000000000
2022-09-13 21:33:18.690: combobox 000f 0000000000000000 0000000000000000
2022-09-13 21:33:18.690: combobox 0085 0000000000000001 0000000000000000

more to come

[kyle]

Code: Select all

...
2022-09-13 21:33:18.690: request property 281 C:\Downloads
2022-09-13 21:33:18.690: get property 281 C:\Downloads
2022-09-13 21:33:18.690: get property system property 281 C:\Downloads
2022-09-13 21:33:18.690: get property 177 C:\Downloads
2022-09-13 21:33:18.690: get property system property 177 C:\Downloads
2022-09-13 21:33:18.690: get property 47 C:\Downloads
2022-09-13 21:33:18.690: get property system property 47 C:\Downloads
...

still more to come

[kyle]

the rest...

Code: Select all

...

[void]

Thank you for the logs.

Code: Select all

2022-09-13 21:33:18.690: get property 281 C:\Downloads
2022-09-13 21:33:18.690: get property system property 281 C:\Downloads

Everything is requesting the property fine and seeing an empty reparse target from the system.

I'll add more debug output in the next alpha update to work out the issue.
I'll make a post here once this is ready.

[kyle]

OK. Thanks!

Please make sure you are not indexing these properties.
(Please make sure these properties are not listed under Tools -> Options -> Properties)

Out of curiosity, why is it that Reparse Target should not be indexed? I had it indexed initially and checked the box for Fast Sort. Without it, the reparse-target: query is pretty slow, whereas it was instantaneous before.

I'm assuming you'd like for me to leave things as-is until the next alpha release?

[void]

You can re-index this property.

Disabling it was only to check for errors in gathering the reparse target property.

Indexing this property makes it tricky to see these errors as the property is requested in the background.
When not indexed, a simple F5 keypress will force Everything to regather this property so we can check for any errors.

[void]

Everything 1.5.0.1319a adds more verbose debug output for gathering reparse targets.

Could you please disable reparse target indexing and send some verbose debug output with this version:

• In Everything 1.5.0.1319a, from the Tools menu, click Options.
• Click the Properties tab on the left.
• Select Reparse Target.
• Click Remove.
• Click OK.
• From the Tools menu, under the Debug submenu, click Start debug logging.
• From the Tools menu, under the Debug submenu, check Verbose.
• Search for:
  exact:c:\downloads
• Make sure the Reparse Target column is shown.
• Press F5.
• From the Tools menu, under the Debug submenu, click Stop debug logging.
  --This will open your Everything Debug Log.txt in Notepad--
• Could you please post the output here.

The verbose output should contain the raw reparse target information.
Please feel free to reindex your reparse target property once the log is sent.

[kyle]

Here's the (likely) relevant part. The rest is at https://pastebin.com/bNPCNy7M.

2022-09-17 13:26:51.195: get property 281 C:\Downloads
2022-09-17 13:26:51.195: get property system property 281 C:\Downloads
2022-09-17 13:26:51.195: reparse target 52
2022-09-17 13:26:51.195: 0000 03 00 00 a0 2c 00 00 00 00 00 20 00 22 00 00 00 ....,..... ."...
2022-09-17 13:26:51.195: 0010 5c 00 3f 00 3f 00 5c 00 44 00 3a 00 5c 00 44 00 \.?.?.\.D.:.\.D.
2022-09-17 13:26:51.195: 0020 6f 00 77 00 6e 00 6c 00 6f 00 61 00 64 00 73 00 o.w.n.l.o.a.d.s.
2022-09-17 13:26:51.195: 0030 00 00 00 00 ....
2022-09-17 13:26:51.195: reparse tag a0000003
2022-09-17 13:26:51.195: PrintNameLength 0

[kyle]

This is probably all related, but there is another bug with this, and it is the same with v1.4 and the es command line tool.

Everything finds the junction itself:


But doesn't find anything "under" it.


Makes sense if Everything doesn't think it's a Directory Junction, I suppose.

[void]

Thank you very much for the logs kyle,

There's no "display target" for this reparse point.

I will have a fix soon.
I will make a post here once this is available.



Everything doesn't automatically follow junctions with NTFS indexing.

To manually add folder junctions to your Everything 1.5 index:

• (For example add the folder junction "C:\ProgramData\Application Data" which links to "C:\ProgramData")
• In Everything, from the Tools menu, click Options.
• Click the NTFS tab on the left.
• Right click in the NTFS volumes list and click Add....
• To the right of GUID, click Select....
• Select the drive where the folder junction target resides (eg: C: drive).
• Change the Path to mount location in Everything (eg: C:\ProgramData\Application Data)
• Change the Root to the target path without the root part (eg: ProgramData)
• Repeat for additional folder junctions.
• Click OK.

[kyle]

Thanks! I literally use Everything multiple times every day - indispensable. Thank you so much for making this tool.

I followed your directions to add the NTFS Volumes for the Directory Junctions that I have directly on my C:\ drive - some pointing to locations on the same volume and others pointing locations on the D:\ drive volume. It works - I can search using the Directory Junction path and it shows me the files in the target location.

I have a feature request based on my observations of how this currently works. Would it be possible to essentially treat Directory Junctions as first-class citizens instead of as Volumes? Without knowing your architecture, I can't say for certain, but it seems like each Volume's objects are treated as unique and independent, which is ultimately the problem.

Each file that is a descendent of the target of a Directory Junction (if added as an NTFS Volume) gets "counted" multiple times, based on how many Volumes end up pointing to the file. This has a couple undesirable side effects.
1. Search results look as though there are more files than actually exist.
2. Status bar totals are inflated; both the objects count in the bottom left and the size in the bottom right.

The Full Path column shows the path using the Directory Junction, as opposed to the actual target directory. I think this makes sense when explicitly specifying the Directory Junction in the path. Where it gets a little tricky is cases like when the search is a file or just some text, because that could match against a descendent of both the target and the Directory Junction(s). In cases like that, it probably makes sense to use the actual target in the Full Path column (i.e., exclude any path that has a Directory Junction somewhere in it) because there could be multiple Directory Junctions that target an ancestor of that same file. If another column could show all of those excluded paths (due to Directory Junctions) that would be helpful in several ways, especially by making it easy to find all OS-respected paths to a file.

Additionally, perhaps an option allowing the user to designate an override preference for search results that have Directory Junctions that lead to them. For example, I could specify that I want search result paths to show C:\Sysinternals instead of D:\PortableApps\NirLauncher\Sysinternals, even though it's not the actual path because that's how I most often want to work with files in that location.

[void]

Thanks for the feedback kyle,

Everything must treat each reparse target as a separate volume.
Otherwise, changes made under your reparse targets would be missed.



Everything 1.5.0.1320a should now gather the reparse target correctly.

[kyle]

Given that constraint, do you have a suggestion regarding all of the "duplicate" files?

Would it help if I provided more examples to better illustrate the problem?

Hopefully there is something we can figure out so any given file will only show up once in the result set (if configured to do so). This would not apply to hard links.

If you're willing, I'd be glad discuss more offline. Thanks!

[void]

I recommend excluding D:\Downloads (the folder junction target) and accessing this folder from C:\Downloads only:

• Setup Everything to index your D:\Downloads folder as C:\Downloads as mentioned above.
• Add D:\Downloads to your Exclude list under Tools -> Options -> Exclude.

Would it help if I provided more examples to better illustrate the problem?

Yes, please.



There is also a distinct:name search function.

[kyle]

I think I was able to get this mostly working by using the Exclude folders (as you suggested). It still has some shortcomings, but it's certainly better than nothing.

Here's a relatively small example of a folder

C:\kk

and two related Directory Junctions, both of which are added as Local NTFS volumes. One thing I wish it did differently (and maybe it can be made to) is that I wish the Exclude folders would show up (or have the option for them to show up) in the result set. This would be really beneficial because it would at least show that the folder exists and show its target. The contents of the Exclude folders would still not show up - only the excluded folders themselves.

Directory Junctions:

Code: Select all

C:\kl -> C:\kk\kyle-lib
C:\ps -> C:\kl\src\PowerShell

Exclude folders:

Code: Select all

C:\kk\kyle-lib
C:\kk\kyle-lib\src\PowerShell
C:\kl\src\PowerShell

Directory structure:

Code: Select all

C:\kk
|-- other-folders
|-- kyle-lib
|-- even-more-folders

C:\kk\kyle-lib
|-- some-files
|-- some-folders
|-- src

C:\kk\kyle-lib\src
|-- project-folders
|-- PowerShell

C:\kk\kyle-lib\src\PowerShell
|-- .vscode
|-- PSScriptAnalyzerSettings.psd1
|-- history_home.txt
|-- history_work.txt
|-- modules
|-- powershell.config.json
|-- profile.ps1
|-- scripts

C:\kl
|-- some-files
|-- some-folders
|-- src

C:\kl\src
|-- project-folders
|-- PowerShell

C:\kl\src\PowerShell
|-- .vscode
|-- PSScriptAnalyzerSettings.psd1
|-- history_home.txt
|-- history_work.txt
|-- modules
|-- powershell.config.json
|-- profile.ps1
|-- scripts

C:\ps
|-- .vscode
|-- PSScriptAnalyzerSettings.psd1
|-- history_home.txt
|-- history_work.txt
|-- modules
|-- powershell.config.json
|-- profile.ps1
|-- scripts

[void]

thing I wish it did differently (and maybe it can be made to) is that I wish the Exclude folders would show up (or have the option for them to show up) in the result set. This would be really beneficial because it would at least show that the folder exists and show its target. The contents of the Exclude folders would still not show up - only the excluded folders themselves.

Instead of excluding results please try Omit Results:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Exclude tab on the left.
• For each folder:
  • Click Remove.
• Click OK.
  ---Everything will perform a reindex---
• From the Index menu, check Enable Result omissions.
• From the Index menu, click Organize Result omissions.
• Click Add folder... and add your reparse targets:
  C:\kk\kyle-lib
  C:\kk\kyle-lib\src\PowerShell
  C:\kl\src\PowerShell
• Click OK.
• You can now quickly toggle your result omissions.

[kyle]

OK, I made the change as you suggested. Omit Results appears to work and is likely a better solution (at least for my use case) than Exclude. I will definitely make use of that feature. Thanks for pointing it out!

Now that I have all of my (previously) Exclusions as Omit Results instead, am I correct in my understanding that I will still not see that an omitted Directory Junction exists, unless I already know that it exists and toggle it off from my Omitted Results? I feel like this still doesn't really solve the problem, which is I want to still see the omitted folder (Directory Junction in my case), I just don't want to see the contents (unless I toggle the Omitted Result off). Otherwise it's a case of you don't know what you don't know, ya know?

Is there any way that you could make this possible? I really feel that it is a valuable use case.

[void]

Instead of omitting the folders directly, try omitting a file and folder filter:

Remove the folder result omissions:
C:\kk\kyle-lib
C:\kk\kyle-lib\src\PowerShell
C:\kl\src\PowerShell

Add the file and folder filter result omissions:
C:\kk\kyle-lib\**
C:\kk\kyle-lib\src\PowerShell\**
C:\kl\src\PowerShell\**


This will keep the reparse target folder in your results, but hide any subfolders and files.

[kyle]

Hmn... OK, so that definitely did what I asked, in that the result set now includes that folder but does not include its contents. But I didn't think it all through well enough, because even though I can see the folder, I don't know that its contents are omitted. I was thinking that it would have the Reparse Target and I could identify it by that column, but in this use case, the folder is the target (it doesn't have a target).

Is there (or can there be) a column (or something else) that identifies the result as having its contents omitted?

Along that same vein, how about a column that shows all of the files/folders that use a result as their Reparse Target? That would be really helpful as well. It would be very similar to the Hard Link Filenames column.

Thanks for being so responsive and helping me out with this. Excellent product!

[void]

Thanks for the feedback Kyle,

I will consider a property to identify omitted content.

I will consider a property to identify files/folders belonging to a folder junction.

I will add a "Volume Path" property.
This would report the owning volume of the file/folder.
For example:
C:\abc.txt would report C:
C:\mnt\E would report E:
C:\mnt\E\abc.txt would report E:
However, this will probably not be useful for your case.
It would be useful if the reparse target was on a different volume.
(You could do this now with the Volume Label property)

I will consider a property to show the full path with reparse points resolved.
For example:
C:\kl\abc.txt => C:\kk\kyle-lib\abc.txt

There is a "Index Number" property which you could use now to identify your folder junctions.
For example:
C:\abc.txt => 1
C:\kk\kyle-lib => 1
C:\kl\abc.txt => 2

[kyle]

I will consider a property to show the full path with reparse points resolved.
For example:
C:\kl\abc.txt => C:\kk\kyle-lib\abc.txt

That would be cool!

I will consider a property to identify omitted content.

Thank you!

I will consider a property to identify files/folders belonging to a folder junction.

Could you elaborate?


I have one more scenario and I'm not sure what is the best approach. Are you familiar with Scoop? The way it works is that every application you install gets its own subdirectory in the main apps directory (e.g., scoop\apps\7zip). Within each application directory, there is a version-specific directory (e.g., scoop\apps\7zip\22.01) and a current directory, which is a Directory Junction to the version-specific directory (e.g., scoop\apps\7zip\current --> scoop\apps\7zip\22.01). They implemented it this way so we could always use the current path without having to know (and keep up-to-date with) the "current" version - it's actually pretty smart and works really well. However... I imagine you can already see where I'm going with this. Everything doesn't handle this well because it doesn't follow Directory Junctions. I could follow the same pattern you've outlined above (NTFS Indexes and Omitted Results), but I currently have 141 apps installed via Scoop (and it fluctuates). I think you would agree that is really not tenable. In this specific case (Scoop apps), it is WAY more important that the current directory is used instead of the version-specific directory. Wholistically, I'm not sure what is the best approach for Directory Junctions, but if you could provide specific logic to support Scoop (since it is such a well-known Windows installer) that would be excellent! What do you think?

[void]

Along that same vein, how about a column that shows all of the files/folders that use a result as their Reparse Target?

I will consider a property to identify files/folders belonging to a folder junction.

Could you elaborate?

I am not sure what this would be yet, it could be a simple "Is In Junction" (Yes/No) property.
-or-
The "last reparse target".
eg: C:\kl\abc.txt => C:\kk\kyle-lib
eg: C:\kl\subfolder\abc.txt => C:\kk\kyle-lib
eg: C:\windows\notepad.exe =>


What about indexing "c:\scoop" as a folder index under Tools -> Options -> Folders.
This would follow all junctions.
However, you will end up with duplicates from the NTFS index.

You could exclude the results from your NTFS index with the following search:
!<index-type:ntfs c:\scoop\>

Add this to your Everything filter or a new filter.

[kyle]

What about indexing "c:\scoop" as a folder index under Tools -> Options -> Folders.
This would follow all junctions.
However, you will end up with duplicates from the NTFS index.

You could exclude the results from your NTFS index with the following search:
index-type:ntfs !c:\scoop\

That did not work for me, but this does !<index-type:ntfs C:\scoop\>.

Add this to your Everything filter

Same here. Using this works !<index-type:ntfs C:\scoop\>.

Did you forget to add the !<> (or assumed I knew that)? Or am I missing something (perhaps something misconfigured elsewhere)?

Regardless, it appears to work well enough. Thank you again!

[void]

Sorry, you are correct, the search should have been:

!<index-type:ntfs c:\scoop\>