+Plugin Development
+In this guide we are going to learn how to develop a new plugin for Faraday.
+
+To add custom plugins in faraday, you first need to run this command:
+faraday-manage settings -a update reports
+
+Create Plugin structure
+Create the folder new_plugin inside the custom_plugins folder and add the following files inside the new_plugin folder:
+Plugin Structure:
+new_plugin/
+ __init__.py # leave this file empty
+ plugin.py
+
+
Warning
+
Please respect both file names:
+__init__.py and plugin.py
+
+Plugin code
+There are two types of plugins:
+
+- File plugins (plugins that process the contents of a file)
+- Command plugins (plugins that process the execution of a command)
+
+A faraday plugin can be either or both of these types.
+For example the ping plugin only process the execution of the command, but the nmap command supports both the xml generated by nmap or process the execution of nmap.
+In your plugin.py file you must provide a function called createPlugin for the plugin manager to instantiate the plugin
+def createPlugin(ignore_info=False, hostname_resolution=True):
+ return YourPluginClass(ignore_info=ignore_info)
+
Plugin classes
+PluginBase is the base class.
+For File plugins we provide all this classes.
+But you can add other by creating a subclass of PluginByExtension
+
+- PluginBase
+- PluginByExtension
+- PluginXMLFormat
+- PluginJsonFormat
+- PluginMultiLineJsonFormat
+- PluginCSVFormat
+- PluginZipFormat
+
+
+
+For Command plugins only you must inherit from PluginBase
+Your plugin class __init__ method must define the id, that will be the identifier of your plugin.
+Here's an example of the Ping tool plugin:
+
class CmdPingPlugin(PluginBase):
+ def __init__(self, *arg, **kwargs):
+ super().__init__(*arg, **kwargs)
+ self.id = "ping"
+ self.name = "Ping"
+ self.plugin_version = "0.0.1"
+ self.version = "1.0.0"
+ self._command_regex = re.compile(r'^(sudo ping|ping|sudo ping6|ping6)\s+.*?')
+
Beware that it must be unique, you can list all the plugins identifiers with this command: faraday-plugins list-plugins
+File Plugins
+Each of the subclasses for File plugins provide a different way for the plugin to detect if it can process the file
+PluginByExtension:
+This class adds the functionality for identify a file by its extension by setting the attribute extension.
+It can be one extension (".xml") or a list (['.xml', '.xxx']) if your file can have multiple extensions.
+You never inherit directly from this class in a plugin but from some of its subclasses that are more specific.
+class ExampleXMLTool(PluginByExtension):
+
+ def __init__(self):
+ super().__init__()
+ self.extension = ".xml"
+
Use this class if the plugin generates vulnerabilities from a xml file.
+If your report has xml format but a different extension (like nessus), remember to define the extension attribute
+To identify the file set in the identifier_tag attribute the main tag of the xml (ScanGroup in the example), it also can be one tag or a list of tags.
+You can also set the identifier_tag_attributes attribute, it is a set of attributes that the main tag must have.
+This is used to be more specific for example if the main tag is too generic and the xml of other plugin has the same one.
+Example XML
+
<?xml version="1.0"?>
+<ScanGroup ExportedOn="2021-11-05T11:18:57.673843">
+ <Scan>
+ </Scan>
+</ScanGroup>
+
from urllib.parse import urlparse
+from faraday_plugins.plugins.plugin import PluginXMLFormat
+import xml.etree.ElementTree as ET
+
+class ExampleToolXmlParser:
+
+ def __init__(self, xml_output):
+ self.vulns = self.parse_xml(xml_output)
+
+ def parse_xml(self, xml_output):
+ vulns = []
+ tree = ET.fromstring(xml_output)
+ items = tree.iterfind('details/item')
+ for item in items:
+ ip = item.get('ip')
+ os = item.get('os')
+ uri = item.find('uri').text
+ url = urlparse(uri)
+ hostname = [url.netloc]
+ path = url.path
+ if url.scheme == 'https':
+ port = 443
+ else:
+ port = 80
+ issue = item.find('issue')
+ severity = issue.get('severity')
+ issue_text = issue.text
+ vuln = {'ip': ip, 'uri': uri, 'os': os,
+ 'hostname': hostname, 'port': port, 'path': path,
+ 'issue_text': issue_text, 'severity': severity}
+ vulns.append(vuln)
+ return vulns
+
+
+class ExampleToolPlugin(PluginXMLFormat):
+ def __init__(self, *arg, **kwargs):
+ super().__init__(*arg, **kwargs)
+ self.identifier_tag = "example_tool"
+ self.id = "example_tool"
+ self.name = "Name of the tool"
+ self.plugin_version = "0.0.1"
+
+ def parseOutputString(self, output, debug=False):
+ parser = ExampleToolXmlParser(output)
+ for vuln in parser.vulns:
+ h_id = self.createAndAddHost(vuln['ip'], vuln['os'], hostnames=vuln['hostname'])
+ s_id = self.createAndAddServiceToHost(h_id, 'webserver', protocol='tcp', ports=vuln['port'])
+ v_id = self.createAndAddVulnWebToService(h_id, s_id, vuln['issue_text'], severity=vuln['severity'],
+ path=vuln['path'])
+
+def createPlugin(*args, **kwargs):
+ return ExampleToolPlugin(*args, **kwargs)
+
Use this class if the plugin generates vulnerabilities from a json file.
+If yor report has json format but a different extension (not .json), remember to define the extension attribute
+To identify the file set in the json_keys attribute a set with some identifiers keys of the json object
+Example JSON
+
{
+ "name": "some name",
+ "hosts": ["host1", "host2"],
+ "other_field": "some value"
+}
+
class ExampleJsonTool(PluginJsonFormat):
+ def __init__(self, ignore_info):
+ super().__init__(ignore_info)
+ self.json_keys = {'name', 'hosts'}
+
Is the same as PluginJsonFormat but use it when the file has multiple lines and every line is a json.
+Example JSON
+
{ "name": "some name", "hosts": ["host1", "host2"],"other_field": "some value"}
+{ "name": "other name", "hosts": ["host3", "host4"],"other_field": "other value"}
+
class ExampleMultipleJsonTool(PluginMultiLineJsonFormat):
+ def __init__(self, ignore_info):
+ super().__init__(ignore_info)
+ self.json_keys = {'name', 'hosts'}
+
Use this class if the plugin generates vulnerabilities from a csv file.
+If yor report has csv format but a different extension (not .csv), remember to define the extension attribute
+To identify the file set in the csv_headers attribute a set with some identifiers headers of the csv object
+Example CSV
+
name,ip,os
+host1,10.10.10.10,windows
+host2,10.10.10.11,linux
+
class ExampleCSVTool(PluginCSVFormat):
+ def __init__(self, ignore_info):
+ super().__init__(ignore_info)
+ self.json_keys = {'name', 'ip'}
+
Use this class if the plugin generates vulnerabilities from a zip file.
+If yor report has csv format but a different extension (not .zip), remember to define the extension attribute
+To identify the file set in the files_list attribute a set with some file names inside the zip file.
+class ExampleJsonTool(PluginZipFormat):
+ def __init__(self, ignore_info):
+ super().__init__(ignore_info)
+ self.files_list = {'file1.txt', 'file2.txt'}
+
Generating data
+The PluginBase class give you all the main methods to create hosts, services and vulnerabilities
+After you parse the data you will need to create the objects with the information to send to faraday.
+This are the main methods
+def createAndAddHost(self, name, os="unknown", hostnames=None, mac=None, scan_template="", site_name="",
+ site_importance="", risk_score="", fingerprints="", fingerprints_software=""):
+
+def createAndAddServiceToHost(self, host_id, name, protocol="tcp?", ports=None, status="open", version="unknown",
+ description=""):
+
+def createAndAddVulnToHost(self, host_id, name, desc="", ref=None, severity="", resolution="", vulnerable_since="",
+ scan_id="", pci="", data="", external_id=None, run_date=None):
+
+def createAndAddVulnToService(self, host_id, service_id, name, desc="", ref=None, severity="", resolution="",
+ risk="", data="", external_id=None, run_date=None):
+
+def createAndAddVulnWebToService(self, host_id, service_id, name, desc="", ref=None, severity="", resolution="",
+ website="", path="", request="", response="", method="", pname="",
+ params="", query="", category="", data="", external_id=None, run_date=None):
+
createAndAddHost and createAndAddServiceToHost will give you a host ID and a service ID.
+You need to send those to the create vulnerabilities methods
+The main method to parse data and create the vulnerabilities objects is parseOutputString, no mather if is a File or a Command plugin.
+Command Plugins
+For command plugins the detection is done via regex.
+If the plugin is only for command you inherit from PluginBase if it also will support file detection inherit from the appropriate class for that type of file.
+In both cases to detect a command you will need to set the _command_regex attribute with the regex to match the command
+Here's an example of the Ping tool plugin:
+
class CmdPingPlugin(PluginBase):
+ def __init__(self, *arg, **kwargs):
+ super().__init__(*arg, **kwargs)
+ self.id = "ping"
+ self.name = "Ping"
+ self.plugin_version = "0.0.1"
+ self.version = "1.0.0"
+ self._command_regex = re.compile(r'^(sudo ping|ping|sudo ping6|ping6)\s+.*?')
+
+ def parseOutputString(self, output):
+ reg = re.search(r"PING ([\w\.-:]+)( |)\(([\w\.:]+)\)", output)
+ if re.search("0 received|unknown host", output) is None and reg is not None:
+ ip_address = reg.group(3)
+ hostname = reg.group(1)
+ self.createAndAddHost(ip_address, hostnames=[hostname])
+ return True
+
+ def _isIPV4(self, ip):
+ if len(ip.split(".")) == 4:
+ return True
+ else:
+ return False
+
In this case the plugin manager will run the command and send the output to the parseOutputString method.
+But if the tool generate an ouput file instead of send the data to stdout you will need to check that the command has the required parameters to do that.
+To do that you will need to implement the processCommandString method, by default it will return the command without modifications.
+To use output files you will need to set 2 attributes in you plugin: _use_temp_file and _temp_file_extension
+If _use_temp_file is set the plugin will create a temp file and assign its path to the attribute self._output_file_path of the plugin.
+If you need that the temp file has and specific extension use the _temp_file_extension attribute like in the example.
+If the _use_temp_file attribute is set the plugin will send te content of that file to parseOutputString()instead of the output of the command.
+Here is and extract of the nmap plugin to see how processCommandString will modify the original command and return a new command with the required parameters.
+class NmapPlugin(PluginXMLFormat):
+ """
+ Example plugin to parse nmap output.
+ """
+
+ def __init__(self, *arg, **kwargs):
+ super().__init__(*arg, **kwargs)
+ self.identifier_tag = "nmaprun"
+ self.id = "Nmap"
+ self.name = "Nmap XML Output Plugin"
+ self.plugin_version = "0.0.3"
+ self.version = "6.40"
+ self.framework_version = "1.0.0"
+ self.options = None
+ self._current_output = None
+ self._command_regex = re.compile(r'^(sudo nmap|nmap|\.\/nmap)\s+.*?')
+ self._use_temp_file = True
+ self._temp_file_extension = "xml"
+ self.xml_arg_re = re.compile(r"^.*(-oX\s*[^\s]+).*$")
+ self.addSetting("Scan Technique", str, "-sS")
+
+ def parseOutputString(self, output):
+ ...
+
+ def processCommandString(self, username, current_path, command_string):
+ """
+ Adds the -oX parameter to get xml output to the command string that the
+ user has set.
+ """
+ super().processCommandString(username, current_path, command_string)
+ arg_match = self.xml_arg_re.match(command_string)
+ if arg_match is None:
+ return re.sub(r"(^.*?nmap)",
+ r"\1 -oX %s" % self._output_file_path,
+ command_string)
+ else:
+ return re.sub(arg_match.group(1),
+ r"-oX %s" % self._output_file_path,
+ command_string)
+
Full Example
+
+This is an example of a Faraday Plugin that process a xml report.
+In this example we will create a plugin to analyze this XML provided by the output of a tool.
+example.xml
+
<?xml version="1.0" ?>
+<!DOCTYPE example_tool>
+<example_tool scanstart="Thu Nov 9 15:59:13 2017">
+ <details>
+ <item id="999979" ip="10.23.49.232" os="linux">
+ <uri>http://test.com/example.php</uri>
+ <issue severity="low">Some vuln text</issue>
+ </item>
+ <item id="39023023" ip="10.232.62.20" os="linux">
+ <uri>http://test.com/login.php</uri>
+ <issue severity="low">Some other text</issue>
+ </item>
+ <item id="8348343" ip="10.12.37.24" os="linux">
+ <uri>http://test.com/example.php</uri>
+ <issue severity="low">Yet another vuln text</issue>
+ </item>
+ <statistics elapsed="402" itemsfound="3" itemstested="10" />
+ </details>
+</example_tool>
+
plugin.py
+from urllib.parse import urlparse
+from faraday_plugins.plugins.plugin import PluginXMLFormat
+import xml.etree.ElementTree as ET
+
+class ExampleToolXmlParser:
+
+ def __init__(self, xml_output):
+ self.vulns = self.parse_xml(xml_output)
+
+ def parse_xml(self, xml_output):
+ vulns = []
+ tree = ET.fromstring(xml_output)
+ items = tree.iterfind('details/item')
+ for item in items:
+ ip = item.get('ip')
+ os = item.get('os')
+ uri = item.find('uri').text
+ url = urlparse(uri)
+ hostname = [url.netloc]
+ path = url.path
+ if url.scheme == 'https':
+ port = 443
+ else:
+ port = 80
+ issue = item.find('issue')
+ severity = issue.get('severity')
+ issue_text = issue.text
+ vuln = {'ip': ip, 'uri': uri, 'os': os,
+ 'hostname': hostname, 'port': port, 'path': path,
+ 'issue_text': issue_text, 'severity': severity}
+ vulns.append(vuln)
+ return vulns
+
+
+class ExampleToolPlugin(PluginXMLFormat):
+ def __init__(self):
+ super().__init__()
+ self.identifier_tag = "example_tool"
+ self.id = "example_tool"
+ self.name = "Name of the tool"
+ self.plugin_version = "0.0.1"
+
+ def parseOutputString(self, output, debug=False):
+ parser = ExampleToolXmlParser(output)
+ for vuln in parser.vulns:
+ h_id = self.createAndAddHost(vuln['ip'], vuln['os'], hostnames=vuln['hostname'])
+ s_id = self.createAndAddServiceToHost(h_id, 'webserver', protocol='tcp', ports=vuln['port'])
+ v_id = self.createAndAddVulnWebToService(h_id, s_id, vuln['issue_text'], severity=vuln['severity'],
+ path=vuln['path'])
+
+def createPlugin(ignore_info=False, hostname_resolution=True):
+ return ExampleToolXmlParser(ignore_info=ignore_info)
+
+Test and Debug
+You can test your plugin by enabling the custom_plugins_folder setting, and try it with faraday.
+You can also test your plugin from the command line.
+Test Plugins
+List all available plugins
+Verify that the plugins is loaded by the plugin manager
+faraday-plugins list-plugins
+
+Available Plugins:
+...
+...
+...
+example_tool - Name of the tool
+Loaded Plugins: 84
+
Test plugin report detection
+Verify that your file is detected by your plugin
+faraday-plugins detect-report /path/to/report.xml
+
+Plugin: example_tool
+
Test plugin process report
+Verify that your plugin parses the file ok and generate the json structure that will be loaded into faraday
+faraday-plugins process-report /path/to/report.xml
+
+
+{"hosts": [{"ip": "10.23.49.232", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.232.62.20", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some other text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/login.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.12.37.24", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Yet another vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}], "command": {"tool": "example_tool", "command": "example_tool", "params": "/path/to/report.xml", "user": "faraday", "hostname": "", "start_date": "2020-04-01T18:43:34.552623", "duration": 2650, "import_source": "report"}}
+
You can optionally specify the plugin id to not do the detection step and force to process it with the specific plugin
+faraday-plugins process-report --plugin_id YourPluginId /path/to/report.xml
+
+
+{"hosts": [{"ip": "10.23.49.232", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.232.62.20", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some other text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/login.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.12.37.24", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Yet another vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}], "command": {"tool": "example_tool", "command": "example_tool", "params": "/path/to/report.xml", "user": "faraday", "hostname": "", "start_date": "2020-04-01T18:43:34.552623", "duration": 2650, "import_source": "report"}}
+
If you do not have faraday-server installed or don't have the custom_plugins_folder setting, you can use the --custom_plugins_folder parameter with any if the commands (list, detect and process)
+Example:
+faraday-plugins list-plugins --custom-plugins-folder /home/user/.faraday/plugins/
+
Logging
+In the PluginBase there is a logger defined in self.logger that you can use.
+If you need to debug for plugins with the command line set this variable:
+export PLUGIN_DEBUG=1
+faraday-plugins process-report appscan /path/to/report.xml
+2019-11-15 20:37:03,355 - faraday.faraday_plugins.plugins.manager - INFO [manager.py:113 - _load_plugins()] Loading Native Plugins...
+2019-11-15 20:37:03,465 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [acunetix]
+2019-11-15 20:37:03,495 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [amap]
+2019-11-15 20:37:03,549 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [appscan]
+2019-11-15 20:37:03,580 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [arachni]
+2019-11-15 20:37:03,613 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [arp_scan]
+2019-11-15 20:37:03,684 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [beef]
+2019-11-15 20:37:03,714 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [brutexss]
+2019-11-15 20:37:03,917 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [burp]
+2019-11-15 20:37:03,940 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()] Load Plugin [dig]
+...
+
+* Once you enable 2FA, Faraday will show you a QR code and thus you can scan it with your chosen app.
and fill in the server URL, for example 
+ If PostgreSQL is running in a remote machine please follow 
Important: In case of being in a SELinux environment run prior to the checks: 
.
on the left menu.
in the top right corner.
in the top right corner.
located next to the filters button in the top right corner.
+
