py3 prep

[ThePotatoGuy]

primary checklist for things we can do now to make py3 transition easier

NOTE: any changes we make should be 100% compatible with current submods. This includes renames of anything. If a property is moved/renamed, alias the old one.

• adjust usage of global statements
  • global declarations must always be before the var is used. these issues are only in stock ddlc code.
• handling py errors in DDLC code
  • might need to include new override rpys
• remove calls to iteritems/itervalues/etc...
  • I've been purposefully using built-in dict iteration (so always iterating over keys) to avoid having to change this later. I think there is rarely a situation where the performance benefit of iteritems/itervalues is significant over basic key iteration.
• Functions like map or filter return iterators in Python 3.9+, we should check the places where we use those functions. If it needs a list instead of iterator, we should convert it explicitly (e.g. list(map(..., ...))).
• http.client -> httplib check
  • attempt imports from http.client if httplib not found - also we should get Add API Key settings screen + framework #8399 in first because that adds a precedent for import checks
• Fix updater in general. This broke in renpy 7.4 and causes the current.json to be downloaded prior to checking it, making MAS think it is always up to date
• StringIO -> from io import StringIO
  • StringIO is now in the io lib
• Some uses of StringIO need to be converted to io.BytesIO
  • (base64 is bytes only, so the StringIO file-like will not accept)
• Reduce dependency on base DDLC scripts.rpa to ensure all errors are our own/all API is accessible/defined on MAS-level
• Tweak our custom interpolation formatter to always eval not worth the performance impact
• Update our python-packages to python3 versions
  • Look into win10toast as the new notif engine for MAS as it supports more advanced features (buttons in notifs + images, etc.) Additionally update win32api/gui to maintain our active window checks
• Update our python tools to use py3
• make sure menutil3 changes make it into r8
• Modify our code persistent reads to use codecs.decode to test them
• viewkeys -> keys
  • views haven't existed for a long time
• Fix spritepack loading.
  • They aren't loaded at all (spj log indicates dict iteration issues (dictionary changed size during iteration)
  • just need to wrap .keys() calls with a list - bad for efficiency, but should be ok in spritepack load code.
• Fix background changing
  • When changing backgrounds, we get RunTimeErrors in deco_iter_adv regarding dict size changes during iteration
  • same thing as spritepacks - should be find in bg code
• Fix everything lint reports
• fix CI + determining if prebuilt is needed or not
  • fixed in CI fixes for R8 #10203

=======================================
notes for myself:

• bring back rpa checks
• docking station needs fix (see issues channel on d)
• sounds like other I/O needs fixes based on boop comment
• double check the unchecked stuff above (not sure about httplib if it matters?)
• updater updates should be last

[multimokia]

I'm also noting something.

Renpy 8 bundles certifi, however this may be an issue for long-term support due to expiry. #8399 makes a move to handle certs, however we should look into updating the internal certifi cert bundle as supplying our own vers of certifi fails to load if it's used by a python-package, which hampers the submod compatibility point.

[Booplicate]

• Chess needs to be updated
  
• • Check for other places where we're working with files, we might need to convert I/O into/from bytes
• We also need to disable mirroring stdout/stderr to console for non-devs
• Need to fix this splash crash I'm sorry, but an uncaught exception occurred. Compiling ATL code at game/splash.rpy:87 File "game/splash.rpy", line 87, in <module> NameError: name 'Particle Burst' is not defined
• Pong is broken
• Poems/notes are broken

BRAEKING CHANGE:

• Can click on the calendar right after selecting the birthdate interrupting the flow

[ThePotatoGuy]

more breaking changes

• MAS_O31_COSTUME_CG_MAP changed format from obj, str to str, str

other things:

• need at least fonts.rpa, so maybe we should just require everything except scripts.rpa ?
• delete all old update scripts ? users wont be able to in-game update to 0.13 anyway
• calendar has a todo

[Booplicate]

delete all old update scripts ? users wont be able to in-game update to 0.13 anyway

I thought we've decided to do in 2 steps: change the urls, then let update to r8 if the system allows. Has something changed?

[ThePotatoGuy]

didnt we have to use the installer to update users from r6 to r8?

[Booplicate]

I'd prefer to support both the installer and in-game update if possible. Why couldn't we use the updater again, I forgot lol? We should probably write the updating process down.

[ThePotatoGuy]

the build stuff for r8 is using the market thing instead of Mod, since its a different package or platform or whatever it wont just work from lets say a v11 to a v13. I believe we talked about having the cutover be handled through the installer, then post installer cutover we can use the in-game updater again.

ok but also - I was thinking about in-game/installer updating when I wrote that. if someone manually pastes a v13 over a pre-last-v12 build then the older update scripts would be necessary probably. We should test what happens when someone does that.

[multimokia]

Looking through my files because I know I had a solution to Ren'Py's persistent signature system that effectively disables it and returns it to the original system we know from r6/7

[ThePotatoGuy]

ok lets make that the last thing in the current pr, seems important enough as its related to saves

[multimokia]

Ah, found it, ironically in some emails as I discussed with another user who had a similar issue:

python early:
    def verify_data_override(data, signatures, check_verifying=True):
        return True
    renpy.savetoken.verify_data = verify_data_override

Should be the only override needed. This basically cuts out any of the signature verification/checking

[Booplicate]

the build stuff for r8 is using the market thing instead of Mod

I'm not sure why we used Mod, it's pointless. We also can't use market since it doesn't work for mac, I believe multi tested it on a VM or smth? I put market there because it's the way to build the game that worked for me on windows and linux, but perhaps we need 3 builds for each OS (well linux and windows can done in 1 build, technically).

We've switched from the installation into game/ to installation into the basedir last year for this reason too - this is how the game should be installed/updated because we'll need to update the libs and executables. Our updating process should cover that, correct me if I'm wrong on that.

it wont just work from lets say a v11 to a v13.

As far as I know it should still work, we can make an alpha release for us to play around and see how it works.

[Booplicate]

Also iirc renpy has done some changes to zsync and updater, we will need to make sure that our updater works for post py3 updates.