feat(website): Column remapping when submitting metadata files

[fhennig]

resolves #3432

preview URL: https://column-remapping.loculus.org/

Summary

• When a metadata file is uploaded, it is now possible to add a column mapping
• The mapping is only applied when the file is uploaded
• If a new file is uploaded, the mapping is preserved as best as possible.
• If no mapping is configured, the file isn't read
• The mapping cannot map the same input field twice
• The mapping needs to contain all required input fields

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[theosanderson]

I think the logic needs to be reversed, with a row for each "Column in your file", for which you can choose the inputField to map to. This is because often the uploaded file may only have like 4 columns from the selection.

[fhennig]

ok!

[chaoran-chen]

What happens if you have a column that is not mentioned in the inputFields? The API/backend would accept it and leave it to the pipeline to decide what to do with it. With this PR, would the website now reject files that would still be possible to submit directly through the API?

[fhennig]

Files are generally not rejected.

The column remapping however would force you to either map a column to a column that is specified in the input fields, or to just not include the column.

You can also just ignore column remapping altogether, and still submit files with whatever structure you like to the backend.

[chaoran-chen]

Ah, so it's possible to skip the remapping? (Sorry, I haven't tried out the feature yet!)

[fhennig]

yes. Remapping is only done if you click the button to add a column mapping. I also took care to not decompress TSV files if no column mapping is to be applied! (I still need to test if this actually works the way I implemented it)

[rneher]

I just tried this on the preview.

Overall, the flow seems sensible to me. But there were a few things I stumbled on.

• we effectively do not have a required metadata format anymore. So the text on the left needs adjusting
• it would be good if the button "Column mapping" had help tool-tip or similar that explains what this does (e.g. "If you are not using our metadata template, this allows you to map columns in your file to the fields expected by the database")
• It was quite hard to find the entries in the long list. the list seems vaguely alphabetical, but important fields like "collection date" were all the way down in a seemingly random place

[rneher]

Ideally, I think we should have required columns at the top of the list. And if easy to do, we could highly columns that are already selected with a different background color or similar?

[rneher]

beyond that, I think it would be good to merge this in asap after some testing.

[fhennig]

I checked about "And if easy to do, we could highly columns that are already selected with a different background color or similar?" -> Unfortunately the native select element options cannot be styled. I experimented with putting UTF-8 symbols with the text but wasn't happy. I think it's quite helpful though to see which ones are already mapped. A proper solution would, however, involve getting rid of the native select element. Probably not for this PR.

EDIT: I ended up actually doing this; replacing the native select element. It is a bit usability improvement I think!

[fhennig]

Ok, I decided to use ListBox by headlessUI since we already use that for the Autocomplete Field as well, and now I have this:

[rneher]

I like the new UI. I would probably prefer if the required and desired fields would not be there twice. But not blocking.

Thanks! All requested changes have been addressed.

[fhennig]

@corneliusroemer In the monday meeting you said you wanted to have a look, and this is arguably a non-trivial PR; Your input is appreciated!

Chaoran already said he's fine with how things are, I've adress all of Richards points and I think all the compression related stuff that Anya mentioned is also sorted out.

[anna-parker]

LGTM!

I tested with metadata in all zip combinations except for .xz and this works, I also tried out some column renaming and read through the code. I'm not a typescript expert but everything looks very good and the feature is working really nicely!