cisagov#532, provide configuration options for pulling from threat in…

…tel feeds

docs/kubernetes.md

@@ -428,6 +428,8 @@ Lookup extracted file hashes with VirusTotal? (y / N): n
 
 Download updated file scanner signatures periodically? (y / N): y
 
+Configure pulling from threat intelligence feeds for Zeek intelligence framework? (y / N): n
+
 Should Malcolm run and maintain an instance of NetBox, an infrastructure resource modeling tool? (y / N): y
 
 Should Malcolm enrich network traffic using NetBox? (Y / n): y

docs/malcolm-config.md

@@ -128,9 +128,9 @@ Although the configuration script automates many of the following configuration
     - `ZEEK_DISABLE_ICS_ALL` and `ZEEK_DISABLE_ICS_…` - if set to `true`, these variables can be used to disable Zeek's protocol analyzers for Operational Technology/Industrial Control Systems (OT/ICS) protocols
     - `ZEEK_DISABLE_BEST_GUESS_ICS` - see ["Best Guess" Fingerprinting for ICS Protocols](ics-best-guess.md#ICSBestGuess)
     - `ZEEK_EXTRACTOR_MODE` – determines the file extraction behavior for file transfers detected by Zeek; see [Automatic file extraction and scanning](file-scanning.md#ZeekFileExtraction) for more details
-    - `ZEEK_INTEL_FEED_SINCE` - when querying a [TAXII](zeek-intel.md#ZeekIntelSTIX), [MISP](zeek-intel.md#ZeekIntelMISP), or [Mandiant](zeek-intel.md#ZeekIntelMandiant) threat intelligence feed, only process threat indicators created or modified since the time represented by this value; it may be either a fixed date/time (`01/01/2021`) or relative interval (`30 days ago`)
+    - `ZEEK_INTEL_FEED_SINCE` - when querying a [TAXII](zeek-intel.md#ZeekIntelSTIX), [MISP](zeek-intel.md#ZeekIntelMISP), or [Mandiant](zeek-intel.md#ZeekIntelMandiant) threat intelligence feed, only process threat indicators created or modified since the time represented by this value; it may be either a fixed date/time (`01/01/2025`) or relative interval (`7 days ago`)
     - `ZEEK_INTEL_ITEM_EXPIRATION` - specifies the value for Zeek's [`Intel::item_expiration`](https://docs.zeek.org/en/current/scripts/base/frameworks/intel/main.zeek.html#id-Intel::item_expiration) timeout as used by the [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) (default `-1min`, which disables item expiration)
-    - `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` - specifies a [cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) indicating the refresh interval for generating the [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) files (defaults to empty, which disables automatic refresh)
+    - `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` - Specifies a [cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) (using [`cronexpr`](https://github.com/aptible/supercronic/tree/master/cronexpr#implementation)-compatible syntax) indicating the refresh interval for generating the [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) files (defaults to empty, which disables automatic refresh)
     - `ZEEK_JA4SSH_PACKET_COUNT` - the Zeek [JA4+ plugin](https://github.com/FoxIO-LLC/ja4) calculates the JA4SSH value once for every *x* SSH packets; *x* is set here (default `200`)
     - `ZEEK_LIVE_CAPTURE` - if set to `true`, Zeek will monitor live traffic on the local interface(s) defined by `PCAP_FILTER`
         + See [**Tuning Zeek**](live-analysis.md#LiveAnalysisTuningZeek) for other variables related to managing Zeek's performance and resource utilization.

docs/malcolm-hedgehog-e2e-iso-install.md

@@ -239,39 +239,49 @@ The [configuration and tuning](malcolm-config.md#ConfigAndTuning) wizard's quest
         + Users should answer **N** unless they plan to use SFTP/SCP to [upload](upload.md#Upload) PCAP files to Malcolm; answering **Y** will expose TCP port 8022 in Malcolm's firewall for SFTP/SCP connections
 * **Enable file extraction with Zeek?**
     - Answer **Y** to indicate that Zeek should [extract files](file-scanning.md#ZeekFileExtraction) transfered in observed network traffic.
-* **Select file extraction behavior**
-    - This determines which files Zeek should extract for scanning:
-        + `none`: no file extraction
-        + `interesting`: extraction of files with mime types of common attack vectors
-        + `mapped`: extraction of files with recognized mime types
-        + `known`: extraction of files for which any mime type can be determined
-        + `all`: extract all files
-        + `notcommtxt`: extract all files except common plain text files
-* **Select file preservation behavior**
-    - This determines the behavior for preservation of Zeek-extracted files:
-        +  `quarantined`: preserve only flagged files in `./zeek-logs/extract_files/quarantine`
-        + `all`: preserve flagged files in `./zeek-logs/extract_files/quarantine` and all other extracted files in `./zeek-logs/extract_files/preserved`
-        + `none`: preserve no extracted files
-* **Enter maximum allowed space for Zeek-extracted files (e.g., 250GB) or file system fill threshold (e.g., 90%)**
-    - Files [extracted by Zeek](file-scanning.md#ZeekFileExtraction) can be periodically pruned to ensure the disk storage they consume does not exceed a user-specified threshold. See the documentation on [managing Malcolm's disk usage](malcolm-config.md#DiskUsage) for more information.
-* **Expose web interface for downloading preserved files?**
-    - Answering **Y** enables access to the Zeek-extracted files path through the means of a simple HTTPS directory server at **https://<Malcolm host or IP address>/extracted-files/**. Beware that Zeek-extracted files may contain malware.
-* **ZIP downloaded preserved files?**
-    - Answering **Y** will cause that Zeek-extracted files downloaded as described under the previous question will be archived using the ZIP file format.
-* **Enter ZIP archive password for downloaded preserved files (or leave blank for unprotected)** and **Enter AES-256-CBC encryption password for downloaded preserved files (or leave blank for unencrypted)**
-    - A non-blank value will be used as either the ZIP archive file password (if the previous question was answered **Y**) or as the encryption key for the file to be AES-256-CBC-encrypted in an `openssl enc`-compatible format (e.g., `openssl enc -aes-256-cbc -d -in example.exe.encrypted -out example.exe`).
-* **Scan extracted files with ClamAV?**
-    - Answer **Y** to scan extracted files with [ClamAV](https://www.clamav.net/), an antivirus engine.
-* **Scan extracted files with Yara?**
-    - Answer **Y** to scan extracted files with [Yara](https://github.com/VirusTotal/yara), a tool used to identify and classify malware samples.
-* **Scan extracted PE files with Capa?**
-    - Answer **Y** to scan extracted executable files with [Capa](https://github.com/fireeye/capa), a tool for detecting capabilities in executable files.
-* **Lookup extracted file hashes with VirusTotal?**
-    - Answer **Y** to be prompted for a [**VirusTotal**](https://www.virustotal.com/en/#search) API key, which will be used for submitting the hashes of extracted files. Only specify this option if the Malcolm instance has Internet connectivity.
-* **Enter VirusTotal API key**
-    - Specify the [**VirusTotal**](https://www.virustotal.com/en/#search) [API key](https://support.virustotal.com/hc/en-us/articles/115002100149-API) as indicated under the previous question.
-* **Download updated file scanner signatures periodically?**
-    - If the Malcolm instance has Internet connectivity, answer **Y** to enable periodic downloads of signatures used by ClamAV and YARA.
+    - **Select file extraction behavior**
+        + This determines which files Zeek should extract for scanning:
+            * `none`: no file extraction
+            * `interesting`: extraction of files with mime types of common attack vectors
+            * `mapped`: extraction of files with recognized mime types
+            * `known`: extraction of files for which any mime type can be determined
+            * `all`: extract all files
+            * `notcommtxt`: extract all files except common plain text files
+    - **Select file preservation behavior**
+        + This determines the behavior for preservation of Zeek-extracted files:
+            *  `quarantined`: preserve only flagged files in `./zeek-logs/extract_files/quarantine`
+            * `all`: preserve flagged files in `./zeek-logs/extract_files/quarantine` and all other extracted files in `./zeek-logs/extract_files/preserved`
+            * `none`: preserve no extracted files
+    - **Enter maximum allowed space for Zeek-extracted files (e.g., 250GB) or file system fill threshold (e.g., 90%)**
+        + Files [extracted by Zeek](file-scanning.md#ZeekFileExtraction) can be periodically pruned to ensure the disk storage they consume does not exceed a user-specified threshold. See the documentation on [managing Malcolm's disk usage](malcolm-config.md#DiskUsage) for more information.
+    - **Expose web interface for downloading preserved files?**
+        + Answering **Y** enables access to the Zeek-extracted files path through the means of a simple HTTPS directory server at **https://<Malcolm host or IP address>/extracted-files/**. Beware that Zeek-extracted files may contain malware.
+    - **ZIP downloaded preserved files?**
+        + Answering **Y** will cause that Zeek-extracted files downloaded as described under the previous question will be archived using the ZIP file format.
+    - **Enter ZIP archive password for downloaded preserved files (or leave blank for unprotected)** and **Enter AES-256-CBC encryption password for downloaded preserved files (or leave blank for unencrypted)**
+        + A non-blank value will be used as either the ZIP archive file password (if the previous question was answered **Y**) or as the encryption key for the file to be AES-256-CBC-encrypted in an `openssl enc`-compatible format (e.g., `openssl enc -aes-256-cbc -d -in example.exe.encrypted -out example.exe`).
+    - **Scan extracted files with ClamAV?**
+        + Answer **Y** to scan extracted files with [ClamAV](https://www.clamav.net/), an antivirus engine.
+    - **Scan extracted files with Yara?**
+        + Answer **Y** to scan extracted files with [Yara](https://github.com/VirusTotal/yara), a tool used to identify and classify malware samples.
+    - **Scan extracted PE files with Capa?**
+        + Answer **Y** to scan extracted executable files with [Capa](https://github.com/fireeye/capa), a tool for detecting capabilities in executable files.
+    - **Lookup extracted file hashes with VirusTotal?**
+        + Answer **Y** to be prompted for a [**VirusTotal**](https://www.virustotal.com/en/#search) API key, which will be used for submitting the hashes of extracted files. Only specify this option if the Malcolm instance has Internet connectivity.
+    - **Enter VirusTotal API key**
+        + Specify the [**VirusTotal**](https://www.virustotal.com/en/#search) [API key](https://support.virustotal.com/hc/en-us/articles/115002100149-API) as indicated under the previous question.
+    - **Download updated file scanner signatures periodically?**
+        + If the Malcolm instance has Internet connectivity, answer **Y** to enable periodic downloads of signatures used by ClamAV and YARA.
+* **Configure pulling from threat intelligence feeds for Zeek intelligence framework?**
+    - Answer **Y** to configure pulling from threat intelligence feeds to populate the [Zeek intelligence framework](zeek-intel.md#ZeekIntel). Answer **N** to leave settings for pulling from threat intelligence feeds unmodified.
+    - **Pull from threat intelligence feeds on startup?**
+        + Answer **Y** for Malcolm to pull from threat intelligence feeds when the `zeek-offline` container starts up.
+    - **Cron expression for scheduled pulls from threat intelligence feeds**
+        + Specifies a [cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) (using [`cronexpr`](https://github.com/aptible/supercronic/tree/master/cronexpr#implementation)-compatible syntax) indicating the refresh interval for generating the [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) files.
+    - **Threat indicator "since" period**
+        + When querying a [TAXII](zeek-intel.md#ZeekIntelSTIX), [MISP](zeek-intel.md#ZeekIntelMISP), or [Mandiant](zeek-intel.md#ZeekIntelMandiant) threat intelligence feed, only process threat indicators created or modified since the time represented by this value; it may be either a fixed date/time (`01/01/2025`) or relative interval (`7 days ago`).
+    - **`Intel::item_expiration` timeout for intelligence items (`-1min` to disable)**
+        + Specifies the value for Zeek's [`Intel::item_expiration`](https://docs.zeek.org/en/current/scripts/base/frameworks/intel/main.zeek.html#id-Intel::item_expiration) timeout as used by the [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) (default `-1min`, which disables item expiration).
 * **Should Malcolm run and maintain an instance of NetBox, an infrastructure resource modeling tool?**
     - Answer **Y** to enable [NetBox](https://netbox.dev/), a tool for modeling networks and documenting network assets.
 * **Should Malcolm enrich network traffic using NetBox?**

docs/ubuntu-install-example.md

@@ -190,6 +190,8 @@ Lookup extracted file hashes with VirusTotal? (y / N): n
 
 Download updated file scanner signatures periodically? (Y / n): n
 
+Configure pulling from threat intelligence feeds for Zeek intelligence framework? (y / N): n
+
 Should Malcolm run and maintain an instance of NetBox, an infrastructure resource modeling tool? (y / N): n
 
 1: no

docs/zeek-intel.md

@@ -20,6 +20,8 @@ docker compose exec --user $(id -u) zeek /usr/local/bin/docker_entrypoint.sh tru
 
 As multiple instances of this container may be running in a Malcolm deployment (i.e., a `zeek-live` container for [monitoring local network interfaces](live-analysis.md#LocalPCAP) and a `zeek` container for scanning [uploaded PCAPs](upload.md#Upload)), only the non-live container is responsible for creating and managing the Zeek intel files, which are then shared and used by both types of container instances.
 
+Additional settings governing Malcolm's behavior when pulling from threat intelligence feeds may be specified during Malcolm configuration (see the [**end-to-end Malcolm installation example**](malcolm-hedgehog-e2e-iso-install.md#MalcolmConfig)).
+
 For a public example of Zeek intelligence files, see Critical Path Security's [repository](https://github.com/CriticalPathSecurity/Zeek-Intelligence-Feeds), which aggregates data from various other threat feeds into Zeek's format.
 
 ## <a name="ZeekIntelSTIX"></a>STIX™ and TAXII™