voidhash

Plug-in and third party software discussion.
Post Reply
void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

voidhash

Post by void » Mon Jan 31, 2022 6:56 am

voidhash creates a .sfv, .md5, .sha1 and .sha256 file containing a hash and filename for all files in the specified path (recurses subfolders).
.sfv, .md5, .sha1 and .sha256 files are created in each directory using the parent folder name.
Does nothing if the .sfv, .md5, .sha1 and .sha256 file already exists and has a file size. (doesn't overwrite existing hashes)
Does not change the date modified timestamp on folders.
voidhash is designed for static / archive folders.



voidhash-1.0.1.3.zip

Source code included.



Usage:

voidhash.exe [options] <path>

Example: voidhash.exe -sha256 "D:\Backup"

Options:
-sfv
-md5
-sha1
-sha256
-sha512
Specify hash algorithms.
Default is: -sfv -md5 -sha1 -sha256

-bufsize <size>
Specify buffer size in bytes
Default is: -bufsize 8388608

-nosubfolders
Ignore subfolders
Default is to recurse subfolders.



Example usage:

DIR C:\Everything

Everything.exe

voidhash.exe C:\Everything

DIR C:\Everything

Everything.exe
Everything.sfv
Everything.md5
Everything.sha1
Everything.sha256



Everything.sha256 example:

bbd01c54a6fc88301ec4f219b7cf1092d329897f837a74ad0dd0f8a357c7e1d7 *Everything.exe



To check the integrity of a file I recommend 7zip.
You can also use the Everything sfv pass, md5sum pass, sha1sum pass, sha256sum pass properties.



sha256sum
sha1sum
md5sum
sfv

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Jan 31, 2022 4:06 pm

Works.

Salamander has a Plugins | Checksum -> Verify Checksums...
& the .md5, .sfv, .sha, .sha256 all check out just fine.


If you
voidhash .
(CWD), you end up with filenames ..md5, ..sha1, ..sha256 ...
I suppose that's fine.

If you
voidhash PATH
, you end up with BASENAME.md5, BASENAME.sha ...
& the hashes are placed in PATH.


xxHash (for fast* non-crypto hash) ;-).
*"fast" may depend on the device. Like if you have an abysmally slow external HDD (30MB/s), limiting factor is going to be "bus" throughput rather then hash speed.

xxHash has a number of algorithms (with somewhat confusing terminology), with XXH3 being the fastest (over XXH128, by a hair), XXH128 just about equal, XXH64 (default) slower. XXH3 & XXH64 have the same length hash. XXH128 is 2x the length. (So when visually reading, one may prefer XXH3 over XXH128 for that reason.)

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Mon Jan 31, 2022 4:14 pm

voidhash.exe "c:\folder\" does not create sidecar files
voidhash.exe c:\folder\ does create sidecar files without basename (".md5", etc)

(note the trailing backslash)

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Jan 31, 2022 4:15 pm

(Optional) Stats (like from FcHash)
Total: 248files, 109.5MiB, 0.7sec, 163.1MiB/s
?
(That would make it easier to compare performance. FcHash doesn't output to .md5..., but still)

Recursion (subdirectories)?

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Jan 31, 2022 4:18 pm

If you use forward slashes - as should be ;-), voidhash appears to go through the motions, but nothing is actually done.

Code: Select all

C:\TMP\1099>vhash c:/tmp/1099
voidhash 1.0 (c) voidtools 2022
parse c:/tmp/1099
parse c:/tmp/1099...
parse c:/tmp/1099\X...

C:\TMP\1099>vhash c:\tmp\1099
voidhash 1.0 (c) voidtools 2022
parse c:\tmp\1099
parse c:\tmp\1099...
parse c:\tmp\1099\X...
(Well, I had to see what would happen if I tried ;-).)

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Mon Jan 31, 2022 4:27 pm

therube wrote:
Mon Jan 31, 2022 4:18 pm
If you use forward slashes - as should be ;-),
Forward slashes? That is way beyond my tunnelvision! :) Next level stuff ..

(btw: I'm a little surprised you renamed it to vhash, instead of v# ...)

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Jan 31, 2022 5:01 pm

That's cause I've already (temporarily) named FcHash as hash.exe, so I guess vhash.exe will have to do for now.


Fails on LFN.

Code: Select all

C:\TMP\1099>vhash "C:\Local\SANDBOX\CLEAN\drive\E\Program Files\Mozilla Firefox
91.0\PROFILE\storage\default\moz-extension+++334f1b5b-4f41-4a3f-b743-0da5d57ee10
d^userContextId=4294967295\idb\3647222921wleabcEoxlt-eengsairo.files"

voidhash 1.0 (c) voidtools 2022
parse C:\Local\SANDBOX\CLEAN\drive\E\Program Files\Mozilla Firefox 91.0\PROFILE\
storage\default\moz-extension+++334f1b5b-4f41-4a3f-b743-0da5d57ee10d^userContext
Id=4294967295\idb\3647222921wleabcEoxlt-eengsairo.files
parse C:\Local\SANDBOX\CLEAN\drive\E\Program Files\Mozilla Firefox 91.0\PROFILE\
storage\default\moz-extension+++334f1b5b-4f41-4a3f-b743-0da5d57ee10d^userContext
Id=4294967295\idb\3647222921wleabcEoxlt-eengsairo.files...

CreateFileW C:\Local\SANDBOX\CLEAN\drive\E\Program Files\Mozilla Firefox 91.0\PR
OFILE\storage\default\moz-extension+++334f1b5b-4f41-4a3f-b743-0da5d57ee10d^userC
ontextId=4294967295\idb\3647222921wleabcEoxlt-eengsairo.files\xxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxx012345678999-255.char.name.part.incl.extension.txt
The 3647222921wleabcEoxlt-eengsairo.files.md5 ... files are created, but are 0-bytes.


(LFN support is one reason FcHash is appealing to me.)

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Jan 31, 2022 6:22 pm

A switch is needed to select (particular) hash methods (in addition to defaulting to all available).

raccoon
Posts: 719
Joined: Thu Oct 18, 2018 1:24 am

Re: voidhash

Post by raccoon » Mon Jan 31, 2022 8:53 pm

Also switches for sha512 and inclusion of bytesize values :)

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Tue Feb 01, 2022 7:17 am

Thanks for the feedback guys,

Hashing speed is likely going to come down to disk IO.

I have put on my TODO list:
canonicalize path argument. (convert . to an absolute path)
remove trailing \ in path argument
stat output
disable recursion option
convert forward slashes (/) to backslashes (\).
LFN support
hash algorithm switches. (if none specified use all, eg: /sha1 /md5 /sfv)
sha512 support
option to save file size in .md5, .sha1 and .sha256 output.

Thank you for the suggestions.

I wasn't expecting any interest in voidhash.
This is just a tool I wrote for personal use.
It's only a few lines of code.

This might be a good place to recommend other hashing tools.

raccoon
Posts: 719
Joined: Thu Oct 18, 2018 1:24 am

Re: voidhash

Post by raccoon » Tue Feb 01, 2022 7:58 am

For 20 years I've loved the idea of competing hash functions, and watched as new ones supplanted old ones in HashTab / HashCheck. If you've written any of your own functions, I'd live to see how they stack up against those in common circulation.

The thing that all hashers seem to lack is parallel hashing of multiple algorithms and recycling work so the same operations aren't performed twice. Eg, if you know you're going to be hashing SHA256 and SHA512, why not do them together. I don't think anybody does, yet.

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Tue Feb 01, 2022 8:02 am

if you know you're going to be hashing SHA256 and SHA512, why not do them together
voidhash does this!
7zip does too.
voidhash will only read content once.

I can look into multiple threads for voidhash (one thread per algorthim).
Thanks for the suggestions.

EricB
Posts: 40
Joined: Wed Jun 26, 2013 8:56 am

Re: voidhash

Post by EricB » Tue Feb 01, 2022 8:33 am

Just curious... it is stated that existing hashfiles are left alone, but this is only valuable when the folder content is utter static. How about updates? And how about having it team up with Everything, using the DM of the hashfile to check if files have been altered after hash file generation?

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Tue Feb 01, 2022 8:38 am

voidhash is designed for static / archive folders.

I've put on my TODO list:
to add an option to regenerate sidecar files.
add missing files to the sidecar file.

I will look into to an option to update only changed files by tracking file timestamps.
Thank you for the suggestions.

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Sat Feb 05, 2022 8:51 am

voidhash-1.0.1.1.zip
  • canonicalize path argument. (convert . to an absolute path)
  • remove trailing \ in path argument.
  • disable recursion option -nosubfolders
  • convert forward slashes (/) to backslashes (\).
  • added hash algorithm switches. (if none specified use all, eg: -sha256 -sha1 -md5 -sfv)
  • added multiple thread support. (one thread per algorithm)
  • added timing information.
  • added long filename support.
  • added sha512 support.

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Sat Feb 05, 2022 9:05 am

voidhash-1.0.1.2.zip
  • improved long filename support.

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Sat Feb 05, 2022 9:36 am

voidhash-1.0.1.3.zip
  • improved short filename support and relative filename support.

hamid56
Posts: 8
Joined: Sun Jul 20, 2014 9:38 am

Re: voidhash

Post by hamid56 » Sat Feb 05, 2022 10:45 am

Dear David
is there a solution for making generated hash by voidhash compatible with 7zip hash checker(via rightclick on hash-> 7zip->crc sha->test archive:checksum)?

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Sat Feb 05, 2022 10:56 am

I recommend using 7zips CRC SHA -> * context menu and manually checking the computed crc/sha against voidhash.
There might be tools out there that can check .sfv, .md5, .sha1 and .sha256 files with a context menu item.
Please recommend any here if you know of one.

Everything has the SFV Pass, md5sum Pass, sha1sum Pass, sha256sum Pass and sha512sum Pass properties for file verification (slow).

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Sat Feb 05, 2022 11:33 am

It looks like forward slashes - like in c:/folder/ - are still unsupported in v1.0.1.2 ?

No sidecar files are being generated with the following command in CMD:

Code: Select all

T:\>\voidhash-1.0.1.2\voidhash.exe t:/logs/
voidhash 1.0.1.2 (c) voidtools 2022

T:\>

void
Developer
Posts: 9622
Joined: Fri Oct 16, 2009 11:31 pm

Re: voidhash

Post by void » Sat Feb 05, 2022 11:37 am

The sidecar files might already exist.

voidhash will not overwrite existing sidecar files.
Updating sidecar files is on my TODO list.

Please try deleting the existing .sfv, .md5 .sha1 and .sha256 files and calling voidhash again.

Please try version 1.0.1.3.

Does the issue persist?

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Sat Feb 05, 2022 12:54 pm

With version 1.0.1.3 I get the expected results (there were no previous sidecar files in the testfolders, btw.)

"Problem" solved!

hamid56
Posts: 8
Joined: Sun Jul 20, 2014 9:38 am

Re: voidhash

Post by hamid56 » Sat Feb 05, 2022 7:26 pm

after 1h try and error finally i find a nice program that successfully support checking file integrity via all kind of hash generated by "voidhash".
multihasher is downloadable via this link
https://www.abelhadigital.com/multihasher/

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Feb 07, 2022 4:46 pm

In addition to "sidecars" (of a directory), option to output straight hashes (of file(s))?


And, vhash.exe -?|-h.
(Heh. Oh, it's there, simply by virtue of not entering any arguments.)

all files in the specified path (recurses subfolders)
I'm still not seeing that?
Oh, I bet it's cause I'm using (.). Nope.

Well color me confused. There they are, in each (non-empty) subdirectory.
(Wonder if I ever tested on a non-empty tree?)

Does not change the date modified timestamp on folders.
Nice.


Maybe an option to specify the output file name?

As it is if I hash DIRX & "DIRX - copy", I end up with DIRX.md5 & "DIRX - copy.md5".
Some file managers, with some operations (like a Compare) will (by default) automatically pick up the same named file in the other panel. So if I have both directories open, one in each panel, & I'm sitting on DIRX.md5 (in 1 panel), I can Ctrl+Shift+C (Compare Files), & it will automatically (attempt) to pick up a file named DIRX.md5 - from the other panel. And if the file names are the same, the compare proceeds, at that point.

As it is now, with differing file names, I'd need to highlight the wanted filename in the other panel, first.
So instead of an automatic FileCompare DIRX.md5 against DIRX.md5 (from two different directories), I need to hightlight the wanted file in the other panel, so that I end up with FileCompare DIRX.md5 against "DIRX - copy.md5".

If I could voidhash -o 0hash (or ..0hash, or...), specifying 0hash as the output filename(s), then when I go to compare hashes from different directories, I need on select that 1 file in my file manager.


(I'll just note that 7-zip [at lease 7z.exe] can additionally do CRC32, CRC64, BLAKE2sp [but not sfv or sidecars].)

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Mon Feb 07, 2022 5:24 pm

I thought that CRC32 = SFV?

I was already writh=ing a script to convert 7za output to Everything, but stopped when someone posted an already existing method.
I did notice that 7za can also hash named datastreams (ADS). That's a big bonus (to me at least).

therube
Posts: 3457
Joined: Thu Sep 03, 2009 6:48 pm

Re: voidhash

Post by therube » Mon Feb 07, 2022 5:30 pm

I thought that CRC32 = SFV?
Yes it is.
I did notice that 7za can also hash named datastreams (ADS).
Maybe even stdin (I didn't look)?

NotNull
Posts: 3730
Joined: Wed May 24, 2017 9:22 pm

Re: voidhash

Post by NotNull » Mon Feb 07, 2022 6:26 pm

therube wrote:
Mon Feb 07, 2022 5:30 pm
Maybe even stdin (I didn't look)?
Yep, that too.
Although if you do an "echo 123|7za -si h", it will also catch the \r\n at the end. (-si = read from standard input)

(I edited your post instead of replying to it. By accident, obviously. Hopefully all is undone now)

Post Reply