Launching programs on reception and composition of mail

[Mihara]

Here I am with another feature request.

After spending some time considering the use of Pat as a building block in more complex systems by taking advantage of its mailbox structure, I realized that one feature is missing. While it can be replaced by a perpetually running external process or a cron job, it would be a lot better if Pat could be configured to do it by itself:

Pat should have a configuration option to spawn a child process on a freshly composed mail, (whether it comes in from web ui or otherwise) and another one to spawn one for a freshly received mail. Passing them a filename and not caring further what else it does is sufficient. Piping the entire file through such a process before saving the message on disk is also an option, but more prone to user errors resulting in lost mail.

This would permit some hackish solutions for #413, allow a trivial way to solve #37, and otherwise simplify many cool things.

[martinhpedersen]

Have you considered using inotifywait? That's what Pat uses internally to refresh it's state once a new message is received or posted to the outbox.

[martinhpedersen]

https://github.com/la5nta/pat/blob/master/websocket_hub.go#L100-L131

[Mihara]

Have you considered using inotifywait?

Yes, when I said "a perpetually running process or a cron job". It can already be done. There are reasons why I don't consider that a good solution:

1. Yet another program needs to be kept running in parallel waiting for notify or cycling on timer, when, in addition to Pat we already have, at the very least, modems to consider.
2. Are file writes to mailbox actually atomic? On the whole file level? What if my filter takes a while to complete due to making a number of http requests to decide what to write?
3. Now, instead of being a simple filter, this hypothetical program needs to keep track of which files it has already modified.

Etc, etc. Pat deciding when to invoke the filter instead is the natural solution that doesn't feel like a hack.

[martinhpedersen]

1. Are you concerned about the complexity of managing this extra process (e.g. start/stop), or is it the system load you are concerned about?
2. No, those operations are not atomic. This can easily be fixed in Pat by first writing to a temp file, and then renaming it once the data has been fully written. I think we should do this anyway, because the race condition you're concerned about is already there in Pat (in theory).
3. I'm not sure I follow.. inotifywait will give you the filename of the created message. You would only receive one event per message.

Although I understand it would be easier just to define some hooks in the config, my main concern here is the added complexity to the project when all of this could be implemented using standard unix tools. I mean, where do we draw the line? Some questions that comes to mind:

10. For inbound messages, when should we trigger this hook? Immediately after the message is received, or do we wait for the session to complete?
11. Should we block until the process exits?
12. What should we do if the process exists with an error code?
13. What should we do with data written to stdout/stderr?
14. Should the process inherit the environment of the parent process (i.e. Pat)?
15. Should we spawn the process if a message is moved from one location to another?
16. Should we spawn the process if the message is updated/modified?

I believe the answer to some of these questions depends on the concrete use case, so this might need to be configurable as well.

Btw, we also provide the eventlog.json file which you could read (in a separate process) for use cases like this. Pat appends events to this file, each event is a JSON encoded message separated by \n. We could extend this with missing information/events to suit your needs.

[Mihara]

Are you concerned about the complexity of managing this extra process (e.g. start/stop), or is it the system load you are concerned about?

I am more concerned about managing this extra process than load. Me, I'll be able to do it, no problem. Then I write a program for someone else, through some mischance it gets popular, and people all over are wracking their brains how to properly set it up...

I'm not sure I follow.. inotifywait will give you the filename of the created message. You would only receive one event per message.

Is a message only written to once by Pat in its lifecycle, though? I'm afraid I haven't gotten that deep yet.

Although I understand it would be easier just to define some hooks in the config, my main concern here is the added complexity to the project when all of this could be implemented using standard unix tools. I mean, where do we draw the line?

If you ask me, the line should be drawn where Pat would be able to invoke the equivalent of procmail, but it should not be such a program itself. To that end:

For inbound messages, when should we trigger this hook? Immediately after the message is received, or do we wait for the session to complete?

If the session protocol permits swapping between mail receive and send multiple times per session, and Pat can do that, and pick up newly appearing outbound mail while inbound mail is still transferring, then the hook should trigger immediately -- it's possible that a mail robot will answer a message before the session completes and save everyone time. Otherwise it can wait until the session is complete.

Should we block until the process exits?

The thread (or goroutine) that saves the message to disk should block until the process exits. Everything else shouldn't.

What should we do if the process exists with an error code?

Keep unfiltered message and save that instead, write an error in the log.

What should we do with data written to stdout/stderr?

Assuming a filter, save stdout to the file, print stderr in the log.

Should the process inherit the environment of the parent process (i.e. Pat)?

That would be a handy way to pass configuration variables to some of them, so yes.

Should we spawn the process if a message is moved from one location to another?

No, only when it enters the system and leaves.

Should we spawn the process if the message is updated/modified?

Also no.

I believe the answer to some of these questions depends on the concrete use case, so this might need to be configurable as well.

The most common - but far from only - use case I imagine is what, back in FidoNET era, was called a "mail robot" and is still occasionally seen in the form of email-based services. You send a message to a specially configured address, which a program then acts upon and sends you a reply. A filter is simply a more generic and powerful mechanism for implementing these than reading a mailbox, and beyond simple question-response will also allow general transformations on messages. Like silently converting jpeg images into webp to save transmission time, or transliterating Cyrillic into Latin to make a Russian email readable despite ISO-8859-1.