Refactored database handling (now SQLiteDB instead of separate JSON)

README.md

@@ -1,52 +1,69 @@
-<h1 align="center">SearchPass</h1>
-<h3 align="center">Get default passwords for network devices by vendor.</h3>
-
-[![PyPi](https://img.shields.io/pypi/v/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
-[![Python Versions](https://img.shields.io/pypi/pyversions/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
-[![Build Status](https://github.com/dhondta/searchpass/actions/workflows/python-package.yml/badge.svg)](https://github.com/dhondta/searchpass/actions/workflows/python-package.yml)
-[![Known Vulnerabilities](https://snyk.io/test/github/dhondta/searchpass/badge.svg?targetFile=requirements.txt)](https://snyk.io/test/github/dhondta/searchpass?targetFile=requirements.txt)
-[![License](https://img.shields.io/pypi/l/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
-
-This tool is similar to the Ruby implementation [SearchPass](https://github.com/michenriksen/searchpass) *for offline searching of default credentials for network devices, web applications and more*. The present tool expands its capabilities to **more databases of credentials** and allows to **update the local database**, a bit like [SearchSploit](https://www.exploit-db.com/searchsploit) allows to update references to exploits on your local machine.
-
-It relies on [`pybots`](https://github.com/dhondta/pybots) for abstracting robots that download from the sources of default credentials and on [`dictquery`](https://github.com/cyberlis/dictquery) for querying the underlying data using the `--query` option.
-
-```session
-$ pip install searchpass
-[...]
-
-$ searchpass --help
-[...]
-This tool aims to search for default passwords of common devices based on criteria like the vendor or the model.
-It works by caching the whole lists of known default passwords downloaded from various sources (relying on pybots ;
- including CIRTnet, DataRecovery, PasswordDB, RouterPasswd or even SaynamWeb) to perform searches locally.
-
-usage: searchpass [-e] [--passwords] [-q QUERY] [--usernames] [--update] [-h] [--help] [-v]
-
-optional arguments:
-  -e, --empty           include empty username or password (default: False)
-  --passwords           get passwords only (default: False)
-  -q QUERY, --query QUERY
-                        search query (default: None)
-  --usernames           get usernames only (default: False)
-
-extra arguments:
-  --update       update passwords databases
-  -h             show usage message and exit
-  --help         show this help message and exit
-  -v, --verbose  verbose mode (default: False)
-
-Usage examples:
-  searchpass --update
-  searchpass --passwords
-  searchpass --query "username=='user'" --passwords
-```
-
-
-## :clap:  Supporters
-
-[![Stargazers repo roster for @dhondta/searchpass](https://reporoster.com/stars/dark/dhondta/searchpass)](https://github.com/dhondta/searchpass/stargazers)
-
-[![Forkers repo roster for @dhondta/searchpass](https://reporoster.com/forks/dark/dhondta/searchpass)](https://github.com/dhondta/searchpass/network/members)
-
-<p align="center"><a href="#"><img src="https://img.shields.io/badge/Back%20to%20top--lightgrey?style=social" alt="Back to top" height="20"/></a></p>
+<h1 align="center">SearchPass</h1>
+<h3 align="center">Get default passwords for network devices by vendor.</h3>
+
+[![PyPi](https://img.shields.io/pypi/v/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
+[![Build Status](https://github.com/dhondta/searchpass/actions/workflows/python-package.yml/badge.svg)](https://github.com/dhondta/searchpass/actions/workflows/python-package.yml)
+[![Known Vulnerabilities](https://snyk.io/test/github/dhondta/searchpass/badge.svg?targetFile=requirements.txt)](https://snyk.io/test/github/dhondta/searchpass?targetFile=requirements.txt)
+[![License](https://img.shields.io/pypi/l/searchpass.svg)](https://pypi.python.org/pypi/searchpass/)
+
+This tool is similar to the Ruby implementation [SearchPass](https://github.com/michenriksen/searchpass) *for offline searching of default credentials for network devices, web applications and more*. The present tool expands its capabilities to **more databases of credentials** and allows to **update the local database**, a bit like [SearchSploit](https://www.exploit-db.com/searchsploit) allows to update references to exploits on your local machine.
+
+It relies on :
+- [`tinyscript`](https://github.com/dhondta/python-tinyscript), for the CLI tool mechanics
+- [`pybots`](https://github.com/dhondta/python-pybots) for abstracting robots that download from the sources of default credentials
+- [`sqlite3`](https://docs.python.org/3/library/sqlite3.html) for querying the underlying data using the `--query` option
+
+Data from the different sources gets normalized into a SQLite DB when updating the tool. [`searchpass´](https://github.com/dhondta/searchpass) package embeds a database updated end 2024.
+
+```session
+$ pip install searchpass
+[...]
+
+$ searchpass --help
+searchpass 2.0.0
+Author   : Alexandre D'Hondt ([email protected])
+Copyright: © 2021-2024 A. D'Hondt
+License  : GPLv3 (https://www.gnu.org/licenses/gpl-3.0.fr.html)
+Source   : https://github.com/dhondta/searchpass
+
+This tool aims to search for default passwords of common devices based on criteria like the vendor or the model.
+It works by caching the whole lists of known default passwords downloaded from various sources (relying on pybots ;
+ including CIRTnet, DataRecovery, PasswordDB, RouterPasswd or even SaynamWeb) to perform searches locally.
+
+usage: searchpass [-e] [--passwords] [-q QUERY] [--usernames] [--reset] [--show] [--stats] [--update] [-h] [--help] [-v]
+
+search options:
+  -e, --empty           include empty username or password (default: False)
+  --passwords           get passwords only (default: False)
+  -q QUERY, --query QUERY
+                        search query (default: None)
+  --usernames           get usernames only (default: False)
+
+action arguments:
+  --reset     remove cached credentials databases
+  --show      show records of credentials databases
+  --stats     get statistics on credentials databases
+  --update    update credentials databases
+
+extra arguments:
+  -h               show usage message and exit
+  --help           show this help message and exit
+  -v, --verbose    verbose mode (default: False)
+
+Usage examples:
+  searchpass --update
+  searchpass --passwords
+  searchpass --stats
+  searchpass --query "username='user'
+  searchpass --query "username LIKE \"Admin%%\"" --passwords
+```
+
+
+## :clap:  Supporters
+
+[![Stargazers repo roster for @dhondta/searchpass](https://reporoster.com/stars/dark/dhondta/searchpass)](https://github.com/dhondta/searchpass/stargazers)
+
+[![Forkers repo roster for @dhondta/searchpass](https://reporoster.com/forks/dark/dhondta/searchpass)](https://github.com/dhondta/searchpass/network/members)
+
+<p align="center"><a href="#"><img src="https://img.shields.io/badge/Back%20to%20top--lightgrey?style=social" alt="Back to top" height="20"/></a></p>

pyproject.toml

@@ -1,21 +1,25 @@
 [build-system]
-requires = ["setuptools>=66.0", "setuptools-scm"]
+requires = ["setuptools>=70.0", "setuptools-scm"]
 build-backend = "setuptools.build_meta"
 
 [tool.setuptools.dynamic]
-version = {attr = "searchpass.__main__.__version__"}
+version = {attr = "searchpass.__info__.__version__"}
 
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.setuptools.package-data]
+"*" = ["*.json"]
+
 [project]
 name = "searchpass"
 authors = [
   {name="Alexandre D'Hondt", email="[email protected]"},
 ]
 description = "Default passwords search tool supporting many open source databases"
 license = {file = "LICENSE"}
-keywords = ["default-passwords", "search-tool"]
+keywords = ["default-passwords", "default-credentials", "network-devices", "search-tool", "pentest-tool",
+            "cybersecurity-tool"]
 requires-python = ">=3.8,<4"
 classifiers = [
   "Development Status :: 5 - Production/Stable",
@@ -29,13 +33,16 @@ classifiers = [
   "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
   "Topic :: Software Development :: Libraries :: Python Modules",
   "Topic :: Security",
 ]
 dependencies = [
-  "dictquery",
   "pybots>=2.1.3",
   "tinyscript>=1.30.16",
+  "requests>=2.32.2",  # SNYK-PYTHON-REQUESTS-6928867
+  "urllib3>=1.26.19",  # SNYK-PYTHON-URLLIB3-7267250
+  "zipp>=3.19.1",      # SNYK-PYTHON-ZIPP-7430899
 ]
 dynamic = ["version"]
 

requirements.txt

@@ -1,3 +1,4 @@
-dictquery
 pybots>=2.1.3
 tinyscript>=1.30.16
+requests>=2.32.2  # SNYK-PYTHON-REQUESTS-6928867
+urllib3>=1.26.19  # SNYK-PYTHON-URLLIB3-7267250

src/searchpass/VERSION.txt

@@ -1 +1 @@
-1.0.6
+2.0.0

src/searchpass/__info__.py

@@ -0,0 +1,17 @@
+# -*- coding: UTF-8 -*-
+"""Searchpass package information.
+
+"""
+import os
+from datetime import datetime
+
+
+__author__    = "Alexandre D'Hondt"
+__copyright__ = f"© 2021-{datetime.now().year} A. D'Hondt"
+__email__     = "[email protected]"
+__license__   = "GPLv3 (https://www.gnu.org/licenses/gpl-3.0.fr.html)"
+__source__    = "https://github.com/dhondta/searchpass"
+
+with open(os.path.join(os.path.dirname(__file__), "VERSION.txt")) as f:
+    __version__ = f.read().strip()
+

src/searchpass/__init__.py

@@ -0,0 +1,6 @@
+# -*- coding: UTF-8 -*-
+from .db import CredentialsDB
+
+
+__all__ = ["CredentialsDB"]
+

src/searchpass/__main__.py

@@ -1,17 +1,11 @@
 # -*- coding: UTF-8 -*-
-from argparse import Action, SUPPRESS
-from dictquery import match
-from pybots.apis.security.default_credentials import *
-from pybots.apis.security.default_credentials import __all__
-from pybots.core.utils.api import APIError
+from argparse import Action, SUPPRESS as _SUP
 from tinyscript import *
 
+from .__info__ import __author__, __copyright__, __email__, __license__, __version__
+from .db import CredentialsDB
+
 
-__author__    = "Alexandre D'Hondt"
-__email__     = "[email protected]"
-__version__   = "1.0.5"
-__copyright__ = ("A. D'Hondt", 2021)
-__license__   = "gpl-3.0"
 __source__    = "https://github.com/dhondta/searchpass"
 __doc__       = """
 This tool aims to search for default passwords of common devices based on criteria like the vendor or the model.
@@ -21,106 +15,69 @@
 __examples__  = [
     "--update",
     "--passwords",
-    "--query \"username=='user'\" --passwords",
+    "--stats",
+    "--query \"username='user'",
+    "--query \"username LIKE \\\"Admin%%\\\"\" --passwords",
 ]
 
-
-APIS = __all__[:]
-KEYS = {
-    'username': ["user", "username", "user-id"],
-    'password': ["password", "pass", "pswd", "passwd"],
+DB_ACTIONS = {
+    'reset':  "remove cached credentials databases",
+    'show':   "show records of credentials databases",
+    'stats':  "get statistics on credentials databases",
+    'update': "update credentials databases",
 }
-PATH = ts.Path("~/.searchpass", create=True, expand=True)
-
-
-class _UpdateAction(Action):
-    """ Custom action for updating the passwords databases. """
-    def __init__(self, option_strings, dest=SUPPRESS, help=None):
-        super(_UpdateAction, self).__init__(option_strings=option_strings, dest=SUPPRESS, default=SUPPRESS, nargs=0,
-                                           help=help)
-
-    def __call__(self, parser, namespace, values, option_string=None):
-        update()
-        parser.exit()
-
-
-def load(warn=False):
-    """ This loads and aggregates the records for every source of default passwords from the cached JSON files. """
-    data = []
-    for api_name in APIS:
-        name = api_name[:-3]
-        path = PATH.joinpath("%s.json" % name.lower())
-        if path.exists():
-            with path.open() as f:
-                d = json.load(f)
-            for vendor, lst in d.items():
-                for item in (lst.get('data') if isinstance(lst, dict) else lst):
-                    item['vendor'] = vendor
-                    for k in ["username", "password"]:
-                        for k2 in KEYS[k]:
-                            try:
-                                item[k] = item.pop(k2)
-                            except KeyError:
-                                pass
-                    data.append(item)
-        elif warn:
-            logger.warning("'%s' does not exist" % path)
-    return data
 
 
-def update(warn=False):
-    """ This downloads the records for each source of default passwords separately and caches them to JSON files. """
-    data = {}
-    for api_name in APIS:
-        name = api_name[:-3]
-        path = PATH.joinpath("%s.json" % name.lower())
-        path.touch()
-        logger.info("Downloading default credentials from %s..." % name)
-        data = {}
-        with globals()[api_name]() as api:
-            for vendor in api.vendors:
-                try:
-                    data[vendor] = api.credentials(vendor)['data']
-                except APIError as e:
-                    if warn:
-                        logger.warning(e)
-        with path.open('w') as f:
-            json.dump(data, f, indent=2)
+def custom_action(method):
+    class _Action(Action):
+        """ Custom action for applying a method of searchpass.db.CredentialsDB. """
+        def __init__(self, option_strings, dest=_SUP, help=None):
+            super(_Action, self).__init__(option_strings=option_strings, dest=_SUP, default=_SUP, nargs=0, help=help)
+
+        def __call__(self, parser, namespace, values, option_string=None):
+            namespace._action = method
+    return _Action
 
 
 def main():
     """ Tool's main function """
-    parser.register('action', 'update', _UpdateAction)
-    fmt = parser.add_mutually_exclusive_group()
+    from shutil import which
+    if not which("visidata"):
+        del DB_ACTIONS['show']
+    for a in DB_ACTIONS.keys():
+        cls = globals()[f'_{a.capitalize()}Action'] = custom_action(a)
+        parser.register('action', a, cls)
+    so = parser.add_argument_group("search options")
+    fmt = so.add_mutually_exclusive_group()
     fmt.add_argument("-e", "--empty", action="store_true", help="include empty username or password")
     fmt.add_argument("--passwords", action="store_true", help="get passwords only")
-    parser.add_argument("-q", "--query", help="search query")
+    so.add_argument("-q", "--query", help="search query")
     fmt.add_argument("--usernames", action="store_true", help="get usernames only")
-    extra = parser.add_argument_group("extra arguments")
-    extra.add_argument("--update", action="update", help="update passwords databases")
+    dba = parser.add_argument_group("action arguments")
+    for a, h in DB_ACTIONS.items():
+        dba.add_argument(f"--{a}", action=a, help=h)
     initialize()
-    data, creds = load(), {}
-    for item in data:
-        if args.query is None or match(item, args.query):
-            user, pswd = item.get('username'), item.get('password')
-            if user is None or pswd is None:
-                continue
-            if not args.empty and (user == "" or pswd == ""):
-                continue
-            creds.setdefault(user, [])
-            creds[user].append(pswd)
-    if args.usernames:
-        for u in sorted(set(creds.keys())):
-            print(u)
-    elif args.passwords:
-        pswds = []
-        for plst in creds.values():
-            pswds.extend(plst)
-        for p in sorted(set(pswds)):
-            print(p)
+    args.logger = logger
+    credsdb = CredentialsDB(**vars(args))
+    m = getattr(args, "_action", None)
+    if m is not None:
+        getattr(credsdb, m)(**vars(args))
     else:
-        for user, pswds in sorted(creds.items(), key=lambda x: x[0].lower()):
-            for pswd in set(pswds):
-                print("%s:%s" % (user, pswd))
+        creds = credsdb.search(**vars(args))
+        if len(creds) == 0:
+            logger.warning("No credential found")
+        elif args.usernames:
+            for u in sorted(set(creds.keys())):
+                print(u)
+        elif args.passwords:
+            pswds = []
+            for plst in creds.values():
+                pswds.extend(plst)
+            for p in sorted(set(pswds)):
+                print(p)
+        else:
+            for user, pswds in sorted(creds.items(), key=lambda x: x[0].lower()):
+                for pswd in set(pswds):
+                    print("%s:%s" % (user, pswd))
     return 0