Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix registering uploads in repeating groups with Objects API (v2) #5059

Merged
merged 4 commits into from
Jan 31, 2025

Conversation

sergei-maertens
Copy link
Member

Closes #4689

🤠 🔫 #2713 is so relevant...

Changes

  • Added regression test for file uploads in repeating groups
  • Added case for nested repeating groups (because this is possible)
  • Moved the file upload processing entirely to the mapping handler
  • Added cases for editgrid handling so that files within that are also properly dealt with

Backporting this will be... fun.

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Problem detection in the admin email digest is handled
  • Release management

    • I have labelled the PR as "needs-backport" accordingly
  • I have updated the translations assets (you do NOT need to provide translations)

    • Ran ./bin/makemessages_js.sh
    • Ran ./bin/compilemessages_js.sh
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

@sergei-maertens sergei-maertens added the needs-backport Fix must be backported to stable release branch label Jan 29, 2025
Copy link

codecov bot commented Jan 29, 2025

Codecov Report

Attention: Patch coverage is 92.42424% with 5 lines in your changes missing coverage. Please review.

Project coverage is 96.72%. Comparing base (9f92893) to head (c1f3989).
Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
...ons/contrib/objects_api/submission_registration.py 62.50% 3 Missing ⚠️
...s/registrations/contrib/objects_api/handlers/v2.py 95.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5059      +/-   ##
==========================================
- Coverage   96.72%   96.72%   -0.01%     
==========================================
  Files         770      770              
  Lines       26491    26542      +51     
  Branches     3454     3460       +6     
==========================================
+ Hits        25623    25672      +49     
- Misses        606      607       +1     
- Partials      262      263       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sergei-maertens
Copy link
Member Author

We decided to backport this only to 3.0.x - the code on 2.8.x is too different and this bugfix is already unreasonably large.

@sergei-maertens sergei-maertens changed the title Issue/4689 upload in repeating groups Fix registering uploads in repeating groups with Objects API (v2) Jan 30, 2025
@sergei-maertens sergei-maertens force-pushed the issue/4689-upload-in-repeating-groups branch from e42436b to 2ae1a36 Compare January 30, 2025 07:51
Copy link
Contributor

@robinmolen robinmolen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks good.. Maybe that Vasileios (or Viktor) see something

Copy link
Contributor

@vaszig vaszig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know about the test coverage, though I see you added test for a nested nested repeating group so I guess we are ok.

Moved the file component key/value processing to the mapped variable
processing instead of special casing this in the registration handler,
since we will need access to the component types to handle editgrids
which have file uploads inside, as these have different data paths
*and* will require recursion as well since there can be editgrids
inside editgrids that have this problem.

The alternative is special casing repeating groups too, which breaks
the mechanism to do component-specific post-processing in the
dedicated function.

This also updates the query for the submission variables so that if we
have a more exact data path for an upload (inside a repeating group)
that we use that instead of messing with the container editgrid which
incorrectly gets replaced now. For uploads *not* in repeating groups,
this data path is empty because it's identical to the variable key,
so we can coalesce there at the DB level to calculate this in a
unified way.

See also #2713 that highlights the difficulties with how file uploads
are now processed, which requires some proper re-structuring.
Fixed/added some type definitions to make it easier to reason about
the data being operated on.
We must look up the uploads in the url map and replace only the
file component nodes in the item values rather than the whole
repeating group. This also needs to recurse, since in theory
the repeating group can have another repeating group inside
it.
@sergei-maertens sergei-maertens force-pushed the issue/4689-upload-in-repeating-groups branch from 2ae1a36 to c1f3989 Compare January 31, 2025 07:59
@sergei-maertens sergei-maertens merged commit 9631c60 into master Jan 31, 2025
29 of 31 checks passed
@sergei-maertens sergei-maertens deleted the issue/4689-upload-in-repeating-groups branch January 31, 2025 08:11
sergei-maertens added a commit that referenced this pull request Jan 31, 2025
Moved the file component key/value processing to the mapped variable
processing instead of special casing this in the registration handler,
since we will need access to the component types to handle editgrids
which have file uploads inside, as these have different data paths
*and* will require recursion as well since there can be editgrids
inside editgrids that have this problem.

The alternative is special casing repeating groups too, which breaks
the mechanism to do component-specific post-processing in the
dedicated function.

This also updates the query for the submission variables so that if we
have a more exact data path for an upload (inside a repeating group)
that we use that instead of messing with the container editgrid which
incorrectly gets replaced now. For uploads *not* in repeating groups,
this data path is empty because it's identical to the variable key,
so we can coalesce there at the DB level to calculate this in a
unified way.

See also #2713 that highlights the difficulties with how file uploads
are now processed, which requires some proper re-structuring.

We must look up the uploads in the url map and replace only the
file component nodes in the item values rather than the whole
repeating group. This also needs to recurse, since in theory
the repeating group can have another repeating group inside
it.

Backport-of: #5059
@sergei-maertens
Copy link
Member Author

Backported to 3.0.x in 3f96489

The coverage failure can be ignored:

  • part of it is code that wasn't covered to begin with (missing variable key in values)
  • the case _: assert_never(component) branch, which by design should be unreachable. I wish coverage/codecov was smarter about this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-backport Fix must be backported to stable release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Objecttype v2 mapping: Upload in repeating group "replaces" all other data in repeating group on submit
3 participants