Fix progress with patterns and backup discovery

[acammack]

Hi all, this resolves the issues where patterns were not counted towards possible attempts, file extension searching was not applied to patterns in GobusterDir, and backups were being brute-forced for the entire wordlist. These all had pretty tangled fixes so I lumped them into this PR.

$  ./gobuster dir -p patterns.txt -w testwords.txt -x .php,.html -db -u https://testing.test
===============================================================
Gobuster v3.7
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://testing.test
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                testwords.txt
[+] Patterns:                patterns.txt (2 entries)
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.7
[+] Extensions:              php,html
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/index.html           (Status: 200) [Size: 179]
/index.html.bak       (Status: 200) [Size: 179]
Progress: 33 / 33 (100.00%)
===============================================================
Finished
===============================================================
$ wc -l testwords.txt patterns.txt 
 3 testwords.txt
 3 patterns.txt
 5 total

Note the 33 total attempts for:

3 words * 3 patterns * (2 file extensions + no file extension) = 27 original attempts
plus 6 backup styles for the original find of "index.html"

Since it's in this wheelhouse, I'm open to also changing the pattern generation and wordlist behavior to either have a flag for the exclusion of the bare words from the wordlist that's either on or off by default or even make patterns always supersede the wordlist and make users add a plain {GOBUSTER} pattern if they want the original word in addition to the patterns. I'm not sure which would be the most useful, though.

Looking through the old issues, it seems that (1) filtering the whole wordlist through the patterns and (2) applying the discovery patterns exactly once would be the most consistent behavior.

Fixes #298, #405, #480, #533, #319, #364

There are still a couple of things to do before this is ready, but I wanted to get this open for some eyes:

• I need to track successes from success pattern generation to keep it from infinitely generating requests against malicious services/with misconfiguration, possibly with a configurable chain generation limit
• I want to add a global flag for post-find pattern expansion, so that if the user desired everything could be checked with patterns after a successful find

[firefart]

Nice thanks already! One thing I noticed is that we should close all channels when we are done sending on them

[acammack]

One thing I noticed is that we should close all channels when we are done sending on them

Hmm, is that necessary here? By the time we would close the wordChan and moreWordsChan channels, all the other goroutines referring to them will have finished and close won't have anyone to notify. The resources for the channels still won't be freed up until after Run() returns either way. I guess it wouldn't hurt anything either, but I feel it would muddy the intent of the code by implying that there might be consumers that need to be notified that the channels are done.

[firefart]

ok, so as long as no one is waiting for a close, it should be ok then

[acammack]

Yes, all the producers and consumers are terminated by contexts now and verified with wait groups and request counts. I think this also fixes a few minor issues that have already been closed, like #541

[firefart]

Awesome, so just put the PR from draft to normal once you are ready so I can have a test. Thanks so far!

[acammack]

This should be good to test. I think the changes in here will help with adding machine readable results and a pause/state file, though I'll leave those to the future. I think we could also make our string manipulation more efficient, but that's also another PR.