Update WF1 to use Collections for referencing mapping logic instead of embedded mapping JSON in job code

[AishaHassen]

Background, context, and business value

The current implementation of WF1 is using the mappings.json that is manually inserted and hardcoded in the job code. To make the updating of the metadata mapping more dynamic and remove the manual copying and pasting of the mapping logic into the job code, we want to use the "mappings" object in the collections named metadata-mappings in the project to reference metadata mappings.

The specific request, in as few words as possible

Request Type: Change Request on WF1

1. Update Fetch Metadata step:

• Change the Adaptor for this job to Common since we are no longer fetching anything using an API
• Add code to fetch metadata from collections named metadata-mappings

collections.get('metadata-mappings', "mappings")

• Remove the lines that are hardcoding the data(line: 30)) and replace them with what we get from metadata-mappings collection
• Ensure that the data object is being saved in state and being referenced in the following steps.
• **Error Handling: ** If the mappings object does not exist in collections, ensure that the error is handled properly. The workflow should't crash but it should fail and subsequent steps should not execute.

Collections

• The collection that we are consuming in this step has already been created and populated on this issue in msf-lime-mosul-staging project
• Collection name: metadata-mappings
• Object of interest in the collection: mappings

Project and Branch

Project: msf-lime-mosul-staging
Branch: staging

Testing Guidance

• Run WF1 with mappings populated and ensure that the mappings that are saved in Collections are being used.
• Run WF1 with mappings set to empty and ensure that the error is handled properly.

Toggl

MSF 2024

[aleksa-krolls]

hey @AishaHassen So looking at this issue, I think we should have a "staging" project for each existing project.

What I think we should consider "prod" projects:

1. msf-lime-mosul --> current staging project that I would like to start treating as the "prod" project that we can use to generate the project.yaml from
2. msf-lime-metadata --> current staging project that we will start treating as the "prod" project

New "staging" projects:

1. msf-lime-mosul-staging --> I think we need to spec the request to create this project and be sure to update this issue to make sure the collection is updated in this project
2. msf-lime-metadata-staging --> creation of this project is already spec'd

Our workflow will then be to make any changes and test in both "staging" projects... once we're happy, we can push to the "prod" projects. MSF can then export the project.yaml directly from the prod msf-lime-mosul project and know that it's tested and has the tested collection of metadata-mappings in it.

If you're aligned with me on this approach, can you please (1) draft a quick lucidchart diagram showing these environments and testing workflow, and (2) make sure all of these collections-related issues are referencing the right project? Otherwise lmk if we need to have a quick chat about it.

[AishaHassen]

hey @aleksa-krolls , I think we are aligned on our thinking with the project config.

1. New staging project for both the metadata and mosul projects has been created and we have an issue for both(Configure new Staging branch and Github Sync #85 and #9) to set up a staging branch and sync to both project from/to github
2. For collections:
   We are going to instantiate the creation of metadata-mapping collection on msf-lime-mosul-staging but it will get created on msf-lime-metadata-staging (I will need to confirm that we have this capability on App)
3. I have drafted a lucid diagram on our project spaces are configured. Will draft something for the collections staff as well