NarrativeRecommender: Integration into the Narrative Service (Recommender functionality)

[HermannKroll]

We want to integrate our Narrative Recommender (graph-based recommendation) into our Narrative Service. However, the recommender relies on BM25 scores. Before we can start the integration, we need to discuss how we can compare the texts of documents. Maintaining another index for textual document content will come at high costs - so what are our alternatives?

• Compare the texts of documents in-memory (I don't know how this will work if we do not have IDF information)
• There is a Postgres Extension for BM25 computation. Is it useable?

If there is a way to do it in-memory, this could be the best way to go. The recommender has the following structure at the moment.

1. Take an input document
2. Identify relevant concepts in that document
3. First Stage Retrieval: Find documents that share the same relevant concepts
4. Second Stage Rescoring: Rescore documents based on their graph overlap and their textual similarity

We could reduce the number of documents retrieved by the first stage to a few hundreds. Comparison of texts should then be very fast.

[HermannKroll]

First prototype has been implemented. Textual similiarity is diabled for now.

[HermannKroll]

The similar articles function for COVID 19 pre-prints does not work. I assume that the "/" within the DOI causes problems, e.g.,
localhost:8123/?query=10.1101%2F2022.04.12.22273802&data_source=ZBMed&search_method=recommender&query_col=ZBMed

[HermannKroll]

Seems to work quite good. However, we cannot take it live right now. We should hide the recommender tab / and similar articles link in the document representation. Code can stay.

[HermannKroll]

We need to rework the loading animation. At the moment, there is little feedback that a search is conducted. Maybe we should clear the result list before starting the recommendation so that the user receives and immediate feedback.

We decided for a complete web page locking.

[HermannKroll]

We should also add logging (so write requested recommendation to a special log directory).

• timestamp
• time taken
• Document id (input)
• Document collection (input)
• Collections that have been used for search (query collections)

[HermannKroll]

Implemented and ready. UI elements are hidden at the moment.