Updating source can keep a removed source in an endless loop

[kam193]

Describe the bug
I've noticed in logs that immediately after restart, the YARA updater container attempts to fetch a source that has been removed months ago.

Logs:

{"@timestamp": "2025-01-02 12:31:31,900", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.yara" }, "host": { "ip": "172.16.48.20", "hostname": "f8caf8b6c78b" }, "log": { "level": "ERROR", "logger": "assemblyline.updater.yara" }, "process": { "pid": "1" }, "message": "An error occurred running the update. Will retry...\nTraceback (most recent call last):\n  File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_v4_service/updater/updater.py\", line 541, in _run_source_updates\n    self.do_source_update(service=service)\n  File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_v4_service/updater/updater.py\", line 378, in do_source_update\n    source_obj = sources[source_name]\n                 ~~~~~~~^^^^^^^^^^^^^\nKeyError: '100DaysOfYara_2024'\n"}

After investigating with @cccs-rs, it's turned out the source had an outstanding error:

$ docker exec assemblyline-redis-1 redis-cli HGET service-updates-YARA 100DaysOfYara_2024.status
{"state": "ERROR", "message": "Update interrupted by server shutdown", "ts": "2024-06-18T22:39:47.576939Z"}

To Reproduce
Steps to reproduce the behavior:

This is hard to say, but probably something has to prevent the updater from removing the status when removing the source.

Expected behavior
Removed sources are ignored.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.5.0.67
• Yara service: 4.5.0.27

Additional context