Request: all dicts in one unzipped file (as an option)

[FFAMax]

Hey Alex, to optimize how dict's data loaded to GPU, should be great to give more data in bulk to reduce moving data back and forth and reduce time lost for data loading and workload adjusting.

[FFAMax]

user@b250:~/hashcat$ time /usr/bin/hashcat -d 1,4,5,6,7 -w 1 -m22000 --advice-disable --logfile-disable --potfile-disable --nonce-error-corrections=8 --session help_crack   -ohelp_crack.key help_crack.hash wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz wp_es.txt.gz
.... skipped
real	0m42.953s
user	0m37.991s
sys	0m9.111s

user@b250:~/hashcat$ cat  wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz wp_es.txt.gz > combined.txt.gz

user@b250:~/hashcat$ wc -l combined.txt.gz
182373 combined.txt.gz
user@b250:~/hashcat$ wc -l  wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz wp_es.txt.gz
    8845 wpskey0.txt.gz
    9097 wpskey1.txt.gz
    8339 wpskey2.txt.gz
    8683 wpskey3.txt.gz
    9057 wpskey4.txt.gz
    8433 wpskey5.txt.gz
    8603 wpskey6.txt.gz
    7889 wpskey7.txt.gz
    8838 wpskey8.txt.gz
    8402 wpskey9.txt.gz
   17459 openwall.txt.gz
   19913 wp_fr.txt.gz
   14914 wpchit_bg.txt.gz
   22831 cow_ud_pc.txt.gz
   21070 wp_es.txt.gz
  182373 total

# Double checking that nothing missed
user@b250:~/hashcat$ zcat wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz wp_es.txt.gz | wc -l
16726271
user@b250:~/hashcat$ zcat combined.txt.gz | wc -l
16726271

user@b250:~/hashcat$ time /usr/bin/hashcat -d 1,4,5,6,7 -w 1 -m22000 --advice-disable --logfile-disable --potfile-disable --nonce-error-corrections=8 --session help_crack   -ohelp_crack.key help_crack.hash combined.txt.gz
... skipped
real	0m24.956s
user	0m17.326s
sys	0m5.531s

# unzipped
user@b250:~/hashcat$ time /usr/bin/hashcat -d 1,4,5,6,7 -w 1 -m22000 --advice-disable --logfile-disable --potfile-disable --nonce-error-corrections=8 --session help_crack   -ohelp_crack.key help_crack.hash combined.txt
...
real	0m23.821s
user	0m17.028s
sys	0m5.531s

# On a single device zipped
user@b250:~/hashcat$ time /usr/bin/hashcat -d 2 -w 1 -m22000 --advice-disable --logfile-disable --potfile-disable --nonce-error-corrections=8 --session help_crack   -ohelp_crack.key help_crack.hash combined.txt.gz
...
real	0m53.044s
user	0m15.490s
sys	0m3.067s

# On a single device unzipped
user@b250:~/hashcat$ time /usr/bin/hashcat -d 2 -w 1 -m22000 --advice-disable --logfile-disable --potfile-disable --nonce-error-corrections=8 --session help_crack   -ohelp_crack.key help_crack.hash combined.txt
...
real	0m47.768s
user	0m8.666s
sys	0m3.290s

[RealEnder]

Well, the issue here is you're using cat to concatenate compressed dictionary files.
In help_crack.py, if we detect hashcat, we're using directly gz-compressed dictionaries to reduce IO and disk usage.
Try the same, but like this:
zcat wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz ... | gzip > combined.txt.gz
Also, wc will not work directly with compressed files.

[FFAMax]

$ cat  wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz  > combined-cat.txt.gz
$ zcat  wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey5.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz openwall.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz cow_ud_pc.txt.gz | gzip >  combined-zcat.txt.gz
$ gzip -d combined-cat.txt.gz; gzip -d combined-zcat.txt.gz
$ md5sum combined-zcat.txt combined-cat.txt
cb5e9ac14ebb60366f8571b19da73dc0  combined-zcat.txt
cb5e9ac14ebb60366f8571b19da73dc0  combined-cat.txt

[FFAMax]

to reduce IO and disk usage.

So increasing CPU usage :)

tmpfs, load more hashes to re-use loaded to GPU dicts - are options

[FFAMax]

Spoiler: combined, compressed is better option when interface is limited.

Before move forward, just double check data are the same

$cat cracked.txt.gz openwall.txt.gz prdict.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz > combined.txt.gz && gzip -d combined.txt.gz
$ cat combined.txt | md5sum
5b3b8b05cee061d97e6480c68c820692  -
$ zcat cracked.txt.gz openwall.txt.gz prdict.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz | md5sum
5b3b8b05cee061d97e6480c68c820692  -

So we have proof, combined.txt is the same.

1. Let's proof one file is better than many (3x GPU via PCIe GEN 1@ 1x)

user@b250:~/hashcat$ time hashcat -m 22000 -d 5,6,7 help_crack.hash cracked.txt.gz openwall.txt.gz prdict.txt.gz wp_fr.txt.gz wpchit_bg.txt.gz wpskey0.txt.gz wpskey1.txt.gz wpskey2.txt.gz wpskey3.txt.gz wpskey4.txt.gz wpskey6.txt.gz wpskey7.txt.gz wpskey8.txt.gz wpskey9.txt.gz

real	0m42.251s
user	0m27.575s
sys	0m2.822s

2. With one big compressed (3x GPU via PCIe GEN 1@ 1x)

user@b250:~/hashcat$ time hashcat -m 22000 -d 5,6,7 help_crack.hash combined.txt.gz

real	0m21.758s
user	0m7.207s
sys	0m2.524s

3. One big, decompressed (3x GPU via PCIe GEN 1@ 1x)

user@b250:~/hashcat$ time hashcat -m 22000 -d 5,6,7 help_crack.hash combined.txt

real	0m33.783s
user	0m24.593s
sys	0m2.601s

4. Put file to ramdisk (3x GPU via PCIe GEN 1@ 1x)

mount -t tmpfs -o size=200M tmpfs ramdisk
cp combined.txt ramdisk/
user@b250:~/hashcat$ time hashcat -m 22000 -d 5,6,7 help_crack.hash ramdisk/combined.txt

real	0m33.906s
user	0m23.746s
sys	0m2.660s