3.0.0b1

[ewjoachim]

What's Changed

Breaking changes

A long, complex, necessary and highly appreciated work from @onlyann who has been quite active this year: rewriting most of the main job processing loop of the Procrastinate Worker (#1114) as discussed in #933. It was followed by quite a few improvements we wanted to make for a long time and took the opportunity of a breaking release to do, as described below.

Changes include:

• The migration process has changed, especially if you apply migrations while the application is running. The way it was before was that you had to apply the migrations that came with version X.Y.Z before upgrading the code to X.Y.Z with complex procedures around minor/major versions. The new way is that migrations are noted as pre or post, you apply the pre migrations before deploying the accompagnying new version of Procrastinate, and you apply the post migrations after deploying the said code. If you are several versions late, you'll have to repeat these for every intermediate release that comes with migrations. See the Migration documentation.
• Aborted asynchronous jobs are now cancelled through asyncio's cancel method, which can be handled gracefully by users using shielding, as explained in the Cancel documentation. Synchronous jobs will not be cancelled, but job_context.should_abort() will return True so that you can handle the abortion request in your code.
• A better graceful stopping mechanism with a timeout controlled by shutdown_graceful_timeout. During shutdown, jobs (both synchronous and asynchronous) get an abort request as described in the previous bullet point. See all details in the Shutdown documentation.
• The way a job's abortion request is stored used to be part of the job's status (ABORTING). It's now an independent field (Job.abort_requested). This has implications if you introspect job states, such as with the Job objects or via the Django integration.
• The timeout worker option was renamed to fetch_job_polling_interval to better reflect its purpose. It's accompanied by a new setting abort_job_polling_interval. Both regulate how often the worker contacts the database to request new jobs or cancellation requests, in case they don't receive notifications from the LISTEN/NOTIFY mechanism. This is explained in the Polling documentation. Note that the work from @onlyann led to workers making significantly less SQL queries, especially for workers with high concurrency settings.
• There have been a few changes on the JobContext object that your tasks receive if the task decorator was passed with pass_context=True:
  • job_context.job_result was removed. The only relevant information it contained was the start timestamp, which is now accessible via job_context.start_timestamp
  • The attributes app and job had static typing, mentioning that those attributes could be None while they actually were always set. They are now properly typed.
  • job_context.should_abort_async() was removed. Knowing whether the job should abort or not, is not doing I/Os anymore, so both sync and async code can now use job_context.should_abort(). job_context.abort_reason tells you whether a job is being aborted because the user requested an abortion, or because the worker is stopping.
  • The job_context.task attribute is now a read-only property. Not that it wasn't a good idea to modify it before, but in the very surprising case that one would do, that's now not possible anymore.
• In the arguments of builtin_tasks.remove_old_jobs(), remove_error was renamed to remove_failed for consistency. Similarly, in JobManager.delete_old_jobs(), include_error was renamed include_failed.
• Support for Python 3.8 was removed following its deprecation.

Bug Fixes

• Respect the priority of a periodic task by @medihack in Respect the priority of a periodic task #1217
• Fix a performance regression when fetching a job by @onlyann in fix fetch job performance regression #1225

Related PRs:

• Sync tasks will get an abort request on shutdown (by @onlyann in Shutdown timeout - abort sync tasks #1215).
• Abort job polling interval by @onlyann in Abort job polling interval #1192
• Refactor worker by @onlyann in refactor worker #1114
• Add support for abort notification by @onlyann in Add support for abort notification #1172
• Abort async tasks using asyncio by @onlyann in Abort async tasks using asyncio #1190
• Use additional abort field on jobs table for abortion requests by @medihack in Use additional abort field on jobs table for abortion requests #1139
• Simplify job context by @onlyann in Simplify job context #1213
• Rename remove_error and include_error to remove_failed and include_failed by @medihack in Rename remove_error and include_error to remove_failed and include_failed #1168
• Upgrade migration process by @medihack & @ewjoachim in Use nox to test different environments #1245
• Drop Python 3.8 support by @medihack in Drop Python 3.8 support #1224

Miscellaneous

• Fill missing generic type parameters for JobContext.task by @fau-st in Fill missing generic type parameters for JobContext.task #1218
• Async InMemoryConnector (testing) by @onlyann in Async InMemoryConnector (testing) #1159
• Allow running on windows by @spapas in Allow running on windows #1236
• Constraint to prevent jobs in todo state with abort_requested by @medihack in Constraint to prevent jobs in todo state with abort_requested #1256
• Add devcontainer setup by @medihack in Add devcontainer setup #1264
• Don't use dropped python version for static type checking by @medihack in Don't use dropped python version for static type checking #1269
• Fix to lowest Python version for static type checking in CI by @medihack in Fix to lowest Python version for static type checking in CI #1271
• Update v3 from main branch by @medihack in Update v3 from main branch #1278
• Merge branch 'main' into v3 by @medihack in Merge branch 'main' into v3 #1166
• V3 update by @ewjoachim in V3 update #1181
• Merge main into v3 by @medihack in Merge main into v3 #1203

Documentation

• Update CONTRIBUTING.md by @ewjoachim in Update CONTRIBUTING.md #1204
• Add state graph (using mermaid) to the discussion page by @medihack in Add state graph (using mermaid) to the discussion page #1208
• Fix two small typos in Django docs by @tobami in Fix two small typos in Django docs #1238

Dependencies

• Make docs dependencies non optional again by @medihack in Make docs dependencies non optional again #1202

Kudos

MVPs

• Special thanks to @onlyann who has brought significant changes and helped us bring a lot of the good stuff you can see in this release.
• Never-ending stream of kudos to @medihack who stepped up as a maintainer and has been an excellent steward & force for good to the project.

New contributors for this release

• @tobami made their first contribution in Fix two small typos in Django docs #1238
• @spapas made their first contribution in Allow running on windows #1236

Feedback

We'd love for you to share your thoughts about this release to us:

• Was the upgrade simple enough? Were these release notes detailed enough? Was the timing good? What do you think of the project and its updates? In the future, would you prefer new features or more stability? Please feel free to chime in in the discussion accompanying this release to make your voice heard.

Full Changelog: 2.14.0...3.0.0b1

---

This discussion was created from the release 3.0.0b1.