NIH AIM:4 YR:2 TASK:1B | 2.4.1B | (started yr1) Resolve OAI-PMH harvesting issues

[mreekie]

Continuation of: #10

[cmbz]

Updated title from NIH AIM:4 YR:2 TASK:1A | 2.4.1A | (started yr1) Resolve OAI-PMH harvesting issues to NIH AIM:4 YR:2 TASK:1B | 2.4.1B | (started yr1) Resolve OAI-PMH harvesting issues

[cmbz]

The following issues identified in 4 | 1.4.1 | Resolve OAI-PMH harvesting issues | 5 #10 remain open at this time. Issues in bold are in NIH Backlog Queue:

• Refactor the OAI code, Step 2 #8842
• Revisit/reimplement the concept of a "Harvested file" dataverse#8629
• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. #3437
• Harvest: Do not list a set as available in ListSets until it has been successfully exported at least once. #3322
• Feature Request/Idea: Sanitize languages controlled vocabulary values #8243
• Harvesting: OAI sets are not updated when datasets are deleted. #8005
• Remaining mapping problems when harvesting from a repository using ISO 639-3 language codes #8578

[pdurbin]

Perhaps we could fix this one too:

• Malformed OAI-PMH requests generate an internal server error dataverse#9275

[cmbz]

@pdurbin Happy to add to bottom of prioritized list, here, but wary of taking on too much given that this task was slated for completion last year and has carried over. See original curated list here, for background.

[jggautier]

Hey @cmbz, here are those GitHub issues that I was referring to and you asked me to list here when we were Slacking with @landreev today. And below each issue I added the related email conversations with admins of other repositories.

1. Spike: Review if inability to harvest all records from certain repositories will be or has been resolved when other harvesting-related issues are addressed dataverse.harvard.edu#92

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=324230
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=329544
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=349681

2. Unable to remove datasets that Harvard Dataverse Repository harvested from Swedish National Data Service  dataverse.harvard.edu#142

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=314258

3. Purge all the stale Nesstar harvested dataverses in production dataverse.harvard.edu#153

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=314258

4. Re-harvest from Borealis Repository dataverse.harvard.edu#172

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=322153

5. Unable to create harvesting client for SRDA repository dataverse#7624

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=287243

6. Re-harvesting datasets from Roper Center dataverse.harvard.edu#204

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=330637

I think the second GitHub issue, IQSS/dataverse.harvard.edu#142, is a duplicate and more specific version of the third GitHub issue, IQSS/dataverse.harvard.edu#153, so maybe the second one can be closed?

[DS-INRAE]

This would also be important for harvesting (blocks the most datasets for us)

• Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values dataverse#9992

[cmbz]

2024/01/29

• I have moved all the remaining, open harvesting issues mentioned here to the following epic for tracking: GREI 3: HDV Task - Improve OAI-PMH Harvesting #171
• Please follow the new epic for updates about the individual issues.