Better hooks for recon automation

[0xJeti]

Hi @honoki,

I would like to start a discussion on hooks and their applicability in automatic recon.
Current implementation based on monitoring changes to documents via CouchDB _changes has some issues/limitations:

• update event doesn't carry the information what was updated. So you cannot really trigger any actions on assigning/removing tags, adding/removing IPs from domains, adding/removing scope to programs, changes on titles/return codes on services etc.
• There is no delete event. Deleted documents trigger update event - again without info that an "update" was in fact deletion).
• Recreated documents (deleted and added again) trigger update event where new event should be triggered.

I'm aware that all of those problems are mainly related to limited functionality of _changes endpoint.

So I started to think if maybe we can extend the database with something like another type of document (e.g. queue) where all changes will be stored and can be picked up by bbrf listen?

Another idea would be to extend all applicable document types (program, domain, ip, service, url) with last_changes object that will carry the information what was changed. Based on the bbrf listen can have more granular event handling.