Add ROMPatcher

- Add initial code for patching, works for PS2 for now
- Add patch options for xdelta
- Updated script-level docs
- Update GUI
- Update GUI docs
- Fix bug with parsing getting languages wrong
- Fix bug with ROMParser occasionally thinking multiple patch files exist
- Fix crash if file isn't downloaded on the move
- Various bugfixes encountered moving to fresh install
- Group patches by platform
- Blackify
- Included some missing regex
- ROMParser now matches demos/preproduction for RA hashes
- ROMParser will now just give up if multiple patch files are found. These seem to generally be translations
- Fix bug where retool priority could be overwritten by dat parsing
- Overhaul ROMMover to check file exists in final location
- Overhaul ROMCleaner to account for edge cases

.gitignore

@@ -158,3 +158,6 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
+
+# ROMSearch user-generated files
+saved_config.json

CHANGES.rst

@@ -4,19 +4,43 @@
 Features
 --------
 
+- Added ROMPatcher, which patches ROMs if necessary for RetroAchievements
 - Added ROMCleaner, which will clean out deleted ROMs within the ROM directory
 
 Fixes
 -----
 
+DupeParser
+~~~~~~~~~~
+
+- Don't overwrite retool priority from parsing dat
+
+ROMCleaner
+~~~~~~~~~~
+
+- Significantly overhauled to account for various edge cases
+
 ROMDownloader
 ~~~~~~~~~~~~~
 
 - Tidy logging for removed files
 
+ROMMover
+~~~~~~~~
+
+- Check final final exists before moving
+
+ROMParser
+~~~~~~~~~
+
+- Fixed bug where languages could be parsed wrongly
+- Tidied up parsing RA hashes, and will now give up when multiple patch files are found
+
 General
 ~~~~~~~
 
+- Ensure things are kept as strings throughout
+- Included more regex
 - Include explicit package versions
 - Enable dependabot
 

README.md

@@ -18,6 +18,7 @@ ROMSearch offers the ability to:
 * Parse DAT files as well as filenames for ROM information
 * Remove dupes from ROM lists using DAT files as well as the excellent ``retool`` clonelists
 * Match ROM hashes to RetroAchievements to get compatible ROMs 
+* Patch ROM files so RetroAchievements can be enabled
 * Moving files to a structured location, including potentially additional files that may be needed
 * Discord integration so users can see the results of runs in a simple, clean way
 

docs/configs/config.rst

@@ -12,6 +12,7 @@ Syntax: ::
     dirs:
       raw_dir: [raw_dir]                # Raw directory to download ROM files using ROMDownloader
       rom_dir: [rom_dir]                # Final output directory for ROMs. Will automatically subdivide by platform
+      patch_dir: [patch_dir]            # Directory for patch files. Will divide up by game
       ra_hash_dir: [ra_hash_dir]        # Directory for parsed RA platform hashes
       dat_dir: [dat_dir]                # Directory to place raw .dat files
       parsed_dat_dir: [parsed_dat_dir]  # Directory to place parsed .dat files, as well as clonelists
@@ -42,6 +43,7 @@ Syntax: ::
       run_datparser: true               # OPTIONAL. Whether to run DATParser. Defaults to true
       run_dupeparser: true              # OPTIONAL. Whether to run DupeParsed. Defaults to true
       run_romchooser: true              # OPTIONAL. Whether to run ROMChooser. Defaults to true
+      run_rompatcher: false             # OPTIONAL. Whether to run ROMPatcher. Defaults to false
       run_rommover: true                # OPTIONAL. Whether to run ROMMover. Defaults to true
       dry_run: false                    # OPTIONAL. Set to true to not make any changes to filesystem. Defaults to false
 
@@ -83,6 +85,9 @@ Syntax: ::
                                         #           ['games', 'applications']. Defaults to 'all_but_games', which will
                                         #           remove everything except games
 
+    rompatcher:                         # ROMPatcher specific options
+      xdelta_path: [path_to_xdelta]     # OPTIONAL. This is where xdelta is located on your filesystem
+
     discord:                            # OPTIONAL. If defined, supply a webhook URL so that ROMSearch can post Discord
       webhook_url: [webhook_url]        #           notifications
 

docs/configs/platforms.rst

@@ -6,14 +6,20 @@ ROMSearch can work for a number of platforms, and the config for each is stored
 
 Syntax: ::
 
-    group: ["Redump", "No-Intro"]    # The group for these files. Currently either "Redump" or "No-Intro
-    dir: [dir]                       # The remote dir for rclone. This is usually something like [group]/[platform]
-    unzip: true                      # OPTIONAL. Whether to unzip files when moving. Defaults to false
-    additional_dirs:                 # OPTIONAL. Occasionally, files may need to be pulled from other remote directories.
-      [additional_dir]: [dir]        #           These can be specified here
-
-    ra_id: [id]                      # OPTIONAL. The RetroAchievements console ID, from their API_GetConsoleIDs
-    ra_hash_method: ["md5", "custom] # OPTIONAL. The RetroAchievements hash method. Supports "md5" for now
+    group: ["Redump", "No-Intro"]     # The group for these files. Currently either "Redump" or "No-Intro
+    dir: [dir]                        # The remote dir for rclone. This is usually something like [group]/[platform]
+    unzip: true                       # OPTIONAL. Whether to unzip files when moving. Defaults to false
+    additional_dirs:                  # OPTIONAL. Occasionally, files may need to be pulled from other remote directories.
+      [additional_dir]: [dir]         #           These can be specified here
+
+    ra_id: [id]                       # OPTIONAL. The RetroAchievements console ID, from their API_GetConsoleIDs
+    ra_hash_method: ["md5", "custom"] # OPTIONAL. The RetroAchievements hash method. Supports "md5" and "custom"
+
+    patch_method: ["xdelta"]          # OPTIONAL: Method for patching ROMs. Supports "xdelta", "rompatcher.js"
+    file_exts:                        # OPTIONAL: Potential file extensions. ROMPatcher uses this to figure
+       - [ext]                        #           out the file to patch
+    patch_file_exts:                  # OPTIONAL: Potential file extensions. ROMPatcher uses this to figure
+       - [ext]                        #           out the patch file
 
 Nintendo - Game Boy
 ===================

docs/index.rst

@@ -19,7 +19,6 @@
    utils
    known_issues
    changelog
-   reference_api
 
 Indices and tables
 ==================

docs/intro.rst

@@ -22,6 +22,7 @@ ROMSearch offers the ability to:
 * Parse DAT files as well as filenames for ROM information
 * Remove dupes from ROM lists using DAT files as well as the excellent ``retool`` clonelists
 * Match ROM hashes to RetroAchievements to get compatible ROMs
+* Patch ROM files so RetroAchievements can be enabled
 * Moving files to a structured location, including potentially additional files that may be needed
 * Discord integration so users can see the results of runs in a simple, clean way
 

docs/modules.rst

@@ -19,4 +19,5 @@ the available modules.
   modules/romparser
   modules/romchooser
   modules/rommover
+  modules/rompatcher
   modules/romcleaner

docs/modules/romcleaner.rst

@@ -2,7 +2,7 @@
 ROMCleaner
 ##########
 
-ROMCleaner will remove files on disk (and from the cache file) that are no longer included in the user config
+ROMCleaner will remove files on disk (and from the cache file) that are no longer included in the user config.
 
 ROMCleaner has no config arguments.
 

docs/modules/rommover.rst

@@ -4,6 +4,8 @@ ROMMover
 
 ROMMover will move the final chosen ROM files to a clean directory structure, optionally unzipping files along the way.
 
+If a patch file is found and ROMPatcher is turned on, then the file will also be patched at this stage.
+
 For more details on the ROMMover arguments, see the :doc:`config file documentation <../configs/config>`.
 
 API

docs/modules/rompatcher.rst

@@ -0,0 +1,25 @@
+##########
+ROMPatcher
+##########
+
+If a patch file is found and the ROMPatcher module is selected, then this will download patch files and apply to ROMs.
+To highlight if a ROM has been patched, the final file will have a (ROMPatched) in the file name.
+
+Currently, ROMSearch only uses ``xdelta`` to patch ROMs.
+
+For more details on the ROMPatcher arguments, see the :doc:`config file documentation <../configs/config>`.
+
+xdelta
+======
+
+To use ``xdelta``, download the latest xdelta3 release from
+`here <https://github.com/jmacd/xdelta-gpl/releases/latest>`_. After unzipping the .exe file, add the path to this file
+into your config, under ``xdelta_path`` in the ``rompatcher`` config section.
+
+API
+===
+
+.. autoclass:: romsearch.ROMPatcher
+    :no-index:
+    :members:
+    :undoc-members:

pyproject.toml

@@ -38,6 +38,7 @@ dependencies = [
     "pathvalidate>=3.2.0",
     "PySide6>=6.7.0",
     "pyyaml>=6.0.1",
+    "wget>=3.2",
     "xmltodict>=0.13.0",
 ]
 

romsearch/__init__.py

@@ -13,6 +13,7 @@
     ROMCleaner,
     ROMMover,
     ROMParser,
+    ROMPatcher,
     ROMSearch,
 )
 
@@ -26,5 +27,6 @@
     "ROMCleaner",
     "ROMMover",
     "ROMParser",
+    "ROMPatcher",
     "ROMSearch",
 ]

romsearch/configs/defaults.yml

@@ -232,4 +232,5 @@ ra_patch_checks:
   - "version_no"
   - "revision"
   - "multi_disc"
-  - "demo"
+  - "demos"
+  - "preproduction"

romsearch/configs/platforms/Sony - PlayStation 2.yml

@@ -4,3 +4,10 @@ unzip: true
 
 ra_id: 21
 ra_hash_method: "custom"
+
+# For the ROM patcher
+patch_method: "xdelta"
+file_exts:
+  - ".iso"
+patch_file_exts:
+  - ".xdelta"

romsearch/configs/regex.yml

@@ -474,6 +474,10 @@ cozmic_collection:
   pattern: "\\(Darius Cozmic Collection\\)"
   group: "modern_version"
 
+disney_afternoon_collection:
+  pattern: "\\(The Disney Afternoon Collection\\)"
+  group: "modern_version"
+
 disney_classic_games:
   pattern: "\\(Disney Classic Games\\)"
   group: "modern_version"
@@ -494,6 +498,10 @@ konami_collector_series:
   pattern: "\\(Konami Collector's Series\\)"
   group: "modern_version"
 
+mega_man_legacy_collection:
+  pattern: "\\(Mega Man Legacy Collection\\)"
+  group: "modern_version"
+
 mega_man_x_legacy:
   pattern: "\\(Mega Man X Legacy Collection\\)"
   group: "modern_version"
@@ -507,7 +515,7 @@ namco_anthology_2:
   group: "modern_version"
 
 namcot_collection:
-  pattern: "\\(Namcot Collection.*?\\)"
+  pattern: "\\s?\\(Namcot Collection.*?\\)"
   group: "modern_version"
 
 namco_museum_archives:
@@ -538,6 +546,10 @@ seiken_denetsu_collection:
   pattern: "\\(Seiken Densetsu Collection\\)"
   group: "modern_version"
 
+snk_40th_anniversary:
+  pattern: "\\(SNK 40th Anniversary Collection\\)"
+  group: "modern_version"
+
 steam:
   pattern: "\\(Steam\\)"
   group: "modern_version"
@@ -603,6 +615,10 @@ debug_version:
   pattern: "\\(Debug Version\\)"
   group: "demoted_version"
 
+digital:
+  pattern: "\\(Digital\\)"
+  group: "demoted_version"
+
 earlier:
   pattern: "\\(Earlier\\)"
   group: "demoted_version"
@@ -611,6 +627,10 @@ early:
   pattern: "\\(Early\\)"
   group: "demoted_version"
 
+evercade:
+  pattern: "\\(Evercade\\)"
+  group: "demoted_version"
+
 famicombox:
   pattern: "\\(FamicomBox\\)"
   group: "demoted_version"
@@ -627,6 +647,26 @@ jncota:
   pattern: "\\(Jncota\\)"
   group: "demoted_version"
 
+jy_company:
+  pattern: "\\(J\\.Y\\. Company\\)"
+  group: "demoted_version"
+
+kaiser:
+  pattern: "\\(Kaiser\\)"
+  group: "demoted_version"
+
+kickstarter:
+  pattern: "\\(Kickstarter\\)"
+  group: "demoted_version"
+
+kickstarter_evercade:
+  pattern: "\\(Kickstarter\\, Evercade\\)"
+  group: "demoted_version"
+
+kickstarter_version:
+  pattern: "\\(Kickstarter Version\\)"
+  group: "demoted_version"
+
 limited_run:
   pattern: "\\(Limited Run Games\\)"
   group: "demoted_version"
@@ -663,6 +703,10 @@ piko_interactive:
   pattern: "\\(Piko Interactive\\)"
   group: "demote_version"
 
+pixel_heart:
+  pattern: "\\(Pixel Heart\\)"
+  group: "demoted_version"
+
 rerelease:
   pattern: "\\(Rerelease\\)"
   group: "demoted_version"
@@ -671,10 +715,18 @@ reprint:
   pattern: "\\(Reprint\\)"
   group: "demoted_version"
 
+retrozone:
+  pattern: "\\(RetroZone\\)"
+  group: "demoted_version"
+
 shokai_seisanban:
   pattern: "\\(Shokai Seisanban\\)"
   group: "demoted_version"
 
+whirlwind_manu:
+  pattern: "\\(Whirlwind Manu\\)"
+  group: "demoted_version"
+
 # GENERALLY IGNORE
 
 60_pin_cart:
@@ -700,6 +752,9 @@ pal:
   pattern: "([?:-][\\s])?[(]?PAL(?: [a-zA-Z]+| 50[Hh]z)?(?:\\)?| (?=\\())"
   flags: "NOFLAG"
 
+supervision:
+  pattern: "\\(SuperVision\\)"
+
 43_406_pcb:
   pattern: "\\(43-406 PCB\\)"