.metadata.efu Specification

[void]

.metadata.efu are sidecar files used to describe property values for files/folders.

File Format
Lookup Behavior
Filename Rules
CSV Rules
Encoding
Property Value Syntax
Why efu?
Why .metadata.efu?
.metadata.efu in Everything



File Format

.metadata.efu files are CSV files.
The CSV header describes the property type for each column.
There must be a Filename column.

.metadata.efu example:

Code: Select all

Filename,Rating,Tags
"file.txt",75,"My Tags"

file.txt in the same location as this .metadata.efu will have a rating of 75 (4/5 stars)
file.txt in the same location as this .metadata.efu will have the tag: My Tags



Lookup Behavior

When retrieving property values, .metadata.efu files are checked starting from the location of the target file or folder and moving up the directory hierarchy toward the root.

Lookup stops once a non-NULL property value is found.

For example, the file

C:\folder\subfolder\file.txt

will check for properties in the following .metadata.efu files:

c:\folder\subfolder\.metadata.efu
c:\folder\.metadata.efu
c:\.metadata.efu

Filename Rules

Filenames on Windows are case insensitive.
Case sensitive on unix.



Filenames are relative to the .metadata.efu
/ or \ can be used as a path separator on Windows.
/ is the path separator on unix.
Relative filenames can not go up directories (no ..)



CSV Rules

In the case of duplicated columns, the last column is used.

In the case of duplicated items, the last item is used.



A CSV cell with no text is null.
Use "" for an empty value.
null values are treated as no value.

Use "" inside quotes to escape a literal double quote (")



Encoding

.metadata.efu files should be encoded with UTF-8.
A UTF-8 BOM is recommended, but not required.



Property Value Syntax

Date syntax is Windows FILETIME in decimal or ISO-8601.

Rating syntax is: 1 (1/5 star) to 99 (5/5 stars)

Length/Duration syntax is: A 100 nanosecond interval (Windows FILETIME) or [[[d:]hh:]mm:]ss[.SSSSSSS]

Multi-string syntax: A semicolon (;) delimited list of strings.
Use ; inside literal double quotes (") to escape a literal ;
Use "" inside double quotes to escape a literal double quote in CSV.

Common properties:

Width	Integer
Height	Integer
Bit Depth	Integer
Length	Length/Duration
Audio Sample Rate	Integer
Audio Channels	Integer
Audio Bits Per Sample	Integer
Audio Format	String
Title	String
Artist	String
Album	String
Year	Integer
Comment	String
Track	Integer
Genre	String
Frame Rate	Double
Video Format	String
Rating	Integer 1-99
Tags	Multi-String
CRC32	Hex-String
CRC64	Hex-String
MD5	Hex-String
SHA1	Hex-String
SHA256	Hex-String
SHA384	Hex-String
SHA512	Hex-String
Description	String
Version	String
Subject	String
Authors	Multi-String
Date Taken	Date
Software	String
Copyright	String

Why efu?
EFU/CSV is well defined and will not change.
EFU files can be opened in Everything.
Human readable/editable.

EFU File Lists



Why .metadata.efu?
Following other standards where files starting with . are usually hidden. (.htaccess .gitignore)
There's already .metadata on unix which is a binary file.



.metadata.efu in Everything

Everything 1.5 will support .metadata.efu sidecar files for most Everything properties and Property System properties.
This does not include the size, date modified, date created, date access, attributes and other indexed values (like name length).

Everything will lookup .metadata.efu files from your index.
Lookup is instant and free.



.metadata.efu files are cached.
Everything will not automatically update property values for files/folders if you modify the .metadata.efu externally.
Support for live updates might be added in a future version.

To refresh property values after modifying your .metadata.efu externally:

• In Everything, press F5 to refresh properties.

To refresh indexed property values after modifying your .metadata.efu externally:

• In Everything, Select your indexed property values.
• press Ctrl + F5 to refresh indexed properties.

Everything gives priority to .metadata.efu properties.
If no property value is specified, Everything will continue checking the Windows Property System / built-in property handler for property values.

Property Gathering Priority



To disable .metadata.efu files:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  metadata
• Select: metadata_efu
• Set the value to: false
• Click OK.

.metadata.efu files are enabled by default.



To only search the target file/folder location for a .metadata.efu file

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  metadata
• Select: metadata_efu_max_search_depth
• Set the value to: 0
• Click OK.

Use 255 to search all the way to the root.
Set to 0 to only search the same location as the target file/folder.
Set to 1 to only search the same location and parent folder as the target file/folder.
Everything will search all the way to the root by default.



To search for files with a defined .metadata.efu property value, include the following in your search:

is-metadata-efu-property:

To search for files with a defined .metadata.efu property value for a specific property, include the following in your search:

is-metadata-efu-property:<property-canonical-name>

where <property-canonical-name> is the specific property.

For example, to find files/folders with a tag from a .metadata.efu file, include the following in your search:

is-metadata-efu-property:tag

[brandySnake]

I'm trying to use this feature with 1399a, but I still can't get it to work.
Below is a step-by-step description of the actual testing procedure I performed. Please let me know if you notice anything.

Test Procedure for This Feature
0) Check the settings (metadata_efu: true, metadata_efu_max_search_depth: 255)
1) Export(Ctrl+S) the file list of a folder to .efu
2) Rename the exported .efu file to .metadata.efu
3) Place .metadata.efu in the folder from Step 1
4) In the file list, select all files directly under the folder from Step 1 and press Ctrl+F5 (to rebuild the property index).
5) Enter is-metadata-efu-property: in the search box.
6) The file list (query results) is empty...


The following are my questions. (The numbers are unrelated to the steps above)
Please forgive me for asking without fully understanding the topic.
1) I understand that this feature provides the same functionality as the property definition feature using property_user_values. Is my understanding correct?.
2) Which has higher priority, this feature or property_user_values?
3) Is this feature affected by the settings of property_builtin_handler, property_system, and summary_information?
4) In the File Format section of the description of this feature in the first post on this topic, the Filename does not include the path to the parent folder (it just says "file.txt"). Is this format actually allowed? If so, will it be interpreted as a file directly under that folder?

This is a machine translation, so there may be some strange English expressions. I apologize in advance.

[void]

Thanks for testing .metadata.efu brandySnake,

6) The file list (query results) is empty...

Please try a refresh (Just F5 before hitting Ctrl + F5)
Everything will refresh the .metadata.efu cache on Ctrl + F5 in the next alpha update

1) I understand that this feature provides the same functionality as the property definition feature using property_user_values. Is my understanding correct?.

Yes.

2) Which has higher priority, this feature or property_user_values?

property_user_values.
Property gathering priority

3) Is this feature affected by the settings of property_builtin_handler, property_system, and summary_information?

No.

If you disable property_system, any Windows Property System property values in the .metadata.efu will not map correctly and will be ignored.

4) In the File Format section of the description of this feature in the first post on this topic, the Filename does not include the path to the parent folder (it just says "file.txt"). Is this format actually allowed? If so, will it be interpreted as a file directly under that folder?

Correct, full path and names will not work.
Filenames MUST be relative to the .metadata.efu file.

[brandySnake]

Thank you for your reply.
The fog is clearing a bit more than before.

Filenames MUST be relative to the .metadata.efu file.

The efu files exported in step 1 above contain the full path in the Filename field, so I added the following step.
- 1.5) Edit all filename fields in the exported .efu files to contain only the filename.
I also pressed F5 just before pressing Ctrl+F5 in step 5.
However, the result of the query is-metadata-efu-property: is empty...

So, I tried including .efu files in the index, and after rebuilding the index, the files included in the .efu file created in step 1 appeared in the result of is-metadata-efu-property: !! Yes!!
In conclusion, even if metadata_efu: true is set, it doesn't actually work if .efu files are excluded from indexing.
Looking back at the explanation on this topic, I realize that you already made it clear...

Everything will lookup .metadata.efu files from your index.

The fog has almost cleared. Thanks for bearing with me.


I think the property definition override feature, including this one, will be a highlight feature of Everything 1.5.
Currently, manual operation (Ctrl+F5) is required to update the index after updating csv/efu files, but I'm really looking forward to the arrival of the automatic update feature.

[void]

Thank you for your reply.

- 1.5) Edit all filename fields in the exported .efu files to contain only the filename.

Everything has an advanced "file_list_relative_paths" setting to export to efu with relative filenames.

I don't really recommend enabling the option as paths will be relative to where you save the EFU file.

To enable relative EFU filenames on export:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  relative
• Select: file_list_relative_paths
• Set the value to: true
• Click OK.

Everything 1.4 and 1.5 has full support for relative filenames in EFU files when opening file lists.

Can I use relative paths in a file list?
file_list_relative_paths

So, I tried including .efu files in the index, and after rebuilding the index, the files included in the .efu file created in step 1 appeared in the result of is-metadata-efu-property: !! Yes!!

I initially supported .metadata.efu files without it being indexed.
I may add an option to do this a future version.
I changed to only indexed .metadata.efu files to make the lookup free.

Currently, manual operation (Ctrl+F5) is required to update the index after updating csv/efu files, but I'm really looking forward to the arrival of the automatic update feature.

Tracking changes to .metadata.efu will most likely require the indexing of efu filenames (when implemented).

[brandySnake]

Thanks for the additional information.
The steps I showed you above were purely for testing purposes.
That said, I'm glad to have the opportunity to learn about the path format for EFU files.

Also, I created a small DB instance for testing (which only indexed a limited range and limited extensions) and used that for testing.
The instance I actually use regularly indexes everything, including .efu.

I've now come to realize once again how useful this feature is.

[nullpinter]

If my understanding is correct, does this mean that we can rename the previous version of aaa.efu to aaa.metadata.efu to seamlessly upgrade to the new metadata-enabled efu?

I tried this, and it works fine in 1.5a. I want to confirm if this is the expected behavior? I am using efu as a very important offline index.

[win32]

I'm wondering if a looser format would be more appropriate for this than CSV.
Folders contain different filetypes with each filetype having different properties.
CSV files define a schema-like top row, which might be constraining. Another readable approach might be an ini/toml file with filenames as labels and property=value pairs.

[void]

If my understanding is correct, does this mean that we can rename the previous version of aaa.efu to aaa.metadata.efu to seamlessly upgrade to the new metadata-enabled efu?

No.
There's no such thing as a metadata enabled efu.
.metadata.efu is just a regular EFU file with relative filenames.

I assume your aaa.efu is an exported EFU file list.
This will contain absolute filenames and already contains properties.

I'm wondering if a looser format would be more appropriate for this than CSV.

I don't mind INI, even JSON, I will consider other formats for a future release.
Thanks for the suggestion.
For now, Everything will stick to CSV for simplicity and wide support.

[nullpinter]

If my understanding is correct, does this mean that we can rename the previous version of aaa.efu to aaa.metadata.efu to seamlessly upgrade to the new metadata-enabled efu?

No.
There's no such thing as a metadata enabled efu.
.metadata.efu is just a regular EFU file with relative filenames.

I assume your aaa.efu is an exported EFU file list.
This will contain absolute filenames and already contains properties.

Yes, I do have the old efu with absolute paths. However, if the filename isn't changed to aaa.metadata.efu, the metadata stored inside (such as the description) won't be recognized by Everything.

I'm not sure if this is expected behavior or a UB, but I'm just looking for a way to attach metadata to offline(path not available) efu files.

[void]

Yes, I do have the old efu with absolute paths. However, if the filename isn't changed to aaa.metadata.efu, the metadata stored inside (such as the description) won't be recognized by Everything.

Everything will still recognize the metadata when you open aaa.efu from File -> Open File List.

[nullpinter]

Yes, I do have the old efu with absolute paths. However, if the filename isn't changed to aaa.metadata.efu, the metadata stored inside (such as the description) won't be recognized by Everything.

Everything will still recognize the metadata when you open aaa.efu from File -> Open File List.

Thank you for your reply. I just realized that during testing, I accidentally typed an extra letter in one file, which is why it wasn't correctly recognized as a metadata name. This was entirely my mistake.

[jimspoon]

It seems that Everything can use property values defined in the metadata.efu files only for properties that are already defined or registered as Everything or Windows System properties. It would be most helpful if Everything would provide a way to add their own custom properties to the property scheme, and then provide a way to populate the metadata.efu files with the values for those properties.

Also it would be good if there were a way for the properties and values to travel automatically with their file. Since the metadata.efu files specify values for files located at a specific relative path from the location of the .efu file, if you move the file, the specified file will no longer exist, and Everything won't be able to find the properties and values for the file at its new location. I imagine the line in the metadata.efu file pointing to a non-existent file won't do any harm, though, and will probably not be incorporated into the Everything index which will index only property values for files that still exist at the locations referenced in the metadata.efu files.

[void]

It seems that Everything can use property values defined in the metadata.efu files only for properties that are already defined or registered as Everything or Windows System properties.

Correct.
I would like to keep metadata.efu simple and efficient for Everything 1.5.

It would be most helpful if Everything would provide a way to add their own custom properties to the property scheme, and then provide a way to populate the metadata.efu files with the values for those properties.

Tricky to do with EFU as there's no property value type definition.
Is it a string? number? date etc..

Maybe property value definition could be done in the EFU header..
Maybe it could be done in Everything.

For now, there's "custom property 0" .. "custom property 9"
However, these are always treated as strings.
I have on my TODO list to specify the value type (eg: string / number / date)
You can set the property name under Tools -> Options -> Advanced -> custom_property_*

Also it would be good if there were a way for the properties and values to travel automatically with their file.

I will consider an option to do this.
I will look into search functions that can show missing or broken files from .metadata.efu files.

Thank you for the suggestions.

[nullpinter]

Would you consider adding default support for Blake3 hex strings in the metadata field? Blake3 is being adopted more and more widely.

[void]

I will consider a blake3 property in Everything.
Thank you for the suggestion.

For now, there's crc32, crc64, md5, sha1, sha256, sha384 and sha512

[void]

Everything 1.5.0.1401a fixes an issue with parsing ISO8601 dates and adds [[[d:]hh:]mm:]ss[.SSSSSSS] duration syntax.