reduce frequency of sending email notifications for ontology pull failures.

[alexskr]

Ontology pull mechanism sends a notification to ontology owner when pull fails. This notification can be useful to alert ontology owners about problems; however, it can be too noisy.

I would like to propose the following changes:

1. Send email only after n times of failures. Sometimes pull can fail due to a temporary/transient issue so notifying end users in those cases can be annoying.
2. Stop sending notification about pull failures everyday if pull location is bad. Currently we send notification every day for ontologies with bad pull location which can be annoying even for responsible users. We should send it once and perhaps send a follow after n times of failures but sending it every day for ever is a bit of an overkill.

[syphax-bouazzouni]

The difficulty here is how/where to save the number of failing attempts?

[alexskr]

if we don't want to alter linked data model then perhaps it can be stored in a flat file or redis. Its far from being optimal but should be sufficient.

Also it would be beneficial if ontoportal can flag that pull location is broken for an ontology after many failed attempts and disable pull. Currently cron pull process needlessly wastes resources trying to retrieve ontology which will never work until someone updates pull location

[syphax-bouazzouni]

I upvote for adding it as an attribute in ontologies linked data model, just need to think about it more in-depth.

I think as the OntologySubmission model is overloaded with metadata attributes (in AgroPortal case at least), we should have a new object(model) to store this sort of thing as Error states, log paths, ... everything not directly related to the metadata. More related to submission process info.

But if you plan to do it, in a relatively small time. You can just add a new attributes to OntologySubmission model, called for example attribute :pullLocationRetiries, enforce: [:integer] that will be incremented each time a pullLocation did not load succefully (here: https://github.com/ncbo/ncbo_cron/blob/master/lib/ncbo_cron/ontology_pull.rb#L47C11-L47C11)

[jonquet]

Eventually yes, we would need to separate "ontology/submission admin" attributes from other related to the description/metadata of an ontology/submission.
Such a parameter (number of retries) is definitively an admin thing... so why not kick off the creation of a new object in the model.

Note that this is not a high priority to me, there are more risk for us to loos our users because of system failure or no innovation in terms of the service we provide... than in the number of emails they get... when they are already not working on ontologies anymore ...