Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework Update JSON encoders and decoders #3261

Merged
merged 3 commits into from
Jan 15, 2025
Merged

Conversation

fthomas
Copy link
Member

@fthomas fthomas commented Jan 7, 2024

This is an attempt to clarify the various Encoder and Decoder instances that make up Encoder[Update] and Decoder[Update] that are used by the PullRequestRepository for writing/reading updates to/from disk.

I had a hard time understanding what was going on because

  • nearly all of the instances are implicit (for derivation) but only the instances for Update are used by other parts of the program
  • semi-derivation was mixed with manual modifications
  • decoders for compatibility with old caches depend on derivation and are defined ad-hoc

In this change I tried to bring more clarity to these instances by

  • giving compatibility decoders their own name
  • forgoing semi-derivation and define all instances manually. IMHO this makes it easier to evolve them if there are compatibility constraints because of old caches
  • making every instance except Encoder[Update] and Decoder[Update] private and non-implicit
  • only testing them via the public Encoder[Update] and Decoder[Update] instances

This might be a controversial change and I ackknowledge that one gains more understanding by doing this rework. I'm therefore biased that these changes actually clarify anything. I'm fine with letting this age for a while before merging to see if I still feel this is an improvement then.

Copy link

codecov bot commented Jan 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.81%. Comparing base (757f9bb) to head (61e4780).
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3261      +/-   ##
==========================================
+ Coverage   89.77%   89.81%   +0.03%     
==========================================
  Files         171      171              
  Lines        4970     4989      +19     
  Branches      494      495       +1     
==========================================
+ Hits         4462     4481      +19     
  Misses        508      508              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@alejandrohdezma
Copy link
Member

@fthomas do we still need to keep compatibility for old caches? wouldn't have those already expire and one with the new encoder be created?

@fthomas
Copy link
Member Author

fthomas commented Jan 7, 2024

wouldn't have those already expire and one with the new encoder be created?

This is true for caches that have been updated since #2898 (which was the last change to the Update encoding). But we can't be sure there are no older caches lingering around that will be used again in the future. If we break compatibility with those, the people who run the Scala Steward instance need to remove these old caches for it to work again. I know that this situation is very unlikely, but is unsatisfactory that it could exist.

On the other hand it is very tempting to just semi-derive all our instances again and remove the compatibility decoders. It would simplify that part of the code base as much as possible.

@alejandrohdezma
Copy link
Member

wouldn't have those already expire and one with the new encoder be created?

This is true for caches that have been updated since #2898 (which was the last change to the Update encoding). But we can't be sure there are no older caches lingering around that will be used again in the future. If we break compatibility with those, the people who run the Scala Steward instance need to remove these old caches for it to work again. I know that this situation is very unlikely, but is unsatisfactory that it could exist.

On the other hand it is very tempting to just semi-derive all our instances again and remove the compatibility decoders. It would simplify that part of the code base as much as possible.

That makes sense. Thanks for the clarification 😊

Copy link
Member

@mzuehlke mzuehlke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This brings consistency to all the different version 👍

The other alternative would be to have matching case classes for all different versions, then use semiauto and convert afterwards into the current version.
But I have the feeling that your version is easier to understand.

@fthomas
Copy link
Member Author

fthomas commented Jan 8, 2024

The other alternative would be to have matching case classes for all different versions, then use semiauto and convert afterwards into the current version.

That sounds interesting! I'll give it a try.

@fthomas
Copy link
Member Author

fthomas commented Jan 15, 2025

The other alternative would be to have matching case classes for all different versions, then use semiauto and convert afterwards into the current version.

I tried this now here: https://github.com/scala-steward-org/scala-steward/compare/topic/rework-update-codecs-2
I felt optimistic while writing the compat classes but semiauto just doesn't produce the same encoders and decoders as we have now and I wasn't able to just write them as implicit val updateEncoder: Encoder[Update] = deriveEncoder etc. I'm also a lot less confident that the generated decoders are correct and it is hard to reason about them if they are not explicit.

This is an attempt to clarify the various `Encoder` and `Decoder` instances
that make up `Encoder[Update]` and `Decoder[Update]` that are used by the
`PullRequestRepository` for writing/reading updates to/from disk.

I had a hard time understanding what was going on because
* nearly all of the instances are implicit (for derivation) but only the
  instances for `Update` are used by other parts of the program
* semi-derivation was mixed with manual modifications
* decoders for compatibility with old caches depend on derivation and
  are defined ad-hoc

In this change I tried to bring more clarity to these instances by
* giving compatibility decoders their own name
* forgoing semi-derivation and define all instances manually. IMHO this
  makes it easier to evolve them if there are compatibility constraints
  because of old caches
* making every instance except `Encoder[Update]` and `Decoder[Update]`
  private and non-implicit
* only testing them via the public `Encoder[Update]` and `Decoder[Update]`
  instances

This might be a controversial change and I ackknowledge that one gains
more understanding by doing this rework. I'm therefore biased that these
changes actually clarify anything. I'm fine with letting this age for a
while before merging to see if I still feel this is an improvement then.
@fthomas fthomas force-pushed the topic/rework-update-codecs branch from 026eb23 to f362631 Compare January 15, 2025 07:39
Comment on lines 208 to 217
Encoder.instance[ForArtifactId] { forArtifactId =>
Json.obj(
"ForArtifactId" -> Json.obj(
"crossDependency" -> forArtifactId.crossDependency.asJson,
"newerVersions" -> forArtifactId.newerVersions.asJson,
"newerGroupId" -> forArtifactId.newerGroupId.asJson,
"newerArtifactId" -> forArtifactId.newerArtifactId.asJson
)
)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): You can also use Encoder/Decoder forProduct to reduce boilerplate from this file

This comment follows the conventionalcomments.org standard

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this?

Copy link
Member

@mzuehlke mzuehlke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your manual version is easier to reason about. I agree 👍

@fthomas fthomas merged commit 63a0476 into main Jan 15, 2025
18 checks passed
@fthomas fthomas added this to the 0.32.2 milestone Jan 15, 2025
@fthomas fthomas deleted the topic/rework-update-codecs branch January 15, 2025 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants