Performance question

[UEAB-BL]

Hello!
I get bursts of a couple hundred message, and this can take a while to process. The way Ive understood it, processing should be alot faster?
I only have one worker, I tried enabling another one but it doesnt seem to work, and I cant really find any documentation on it, just some clues here and there in the forum, some of which seem to be outdated.
Is there a way for me to improve performance? CPU and memory is not even visibly affected in htop during heavy processing.

This is from my log, with anonymized adresses and IPs. Does this pacing seem normal?
Its not really a problem for me currently. Ive had questions about performance on postal from other parties that sends ALOT more than me and this pacing would definetly not work for them. So I want to try and increase my performance as a proof of concept. What can/should I do?

2024-10-08 06:05:18 +0000 INFO running task component=worker thread=tasks task=SendNotificationsScheduledTask
2024-10-08 06:05:18 +0000 INFO scheduling task to next run at 2024-10-08 06:06:18 UTC component=worker thread=tasks task=SendNotificationsScheduledTask
2024-10-08 06:05:20 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140553
2024-10-08 06:05:20 +0000 INFO message processing complete component=worker thread=work0 original_queued_message=140549 queued_message=140553
2024-10-08 06:05:20 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work0 original_queued_message=140549 queued_message=140554
2024-10-08 06:05:20 +0000 INFO Sending message 1::140399 to [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140554
2024-10-08 06:05:20 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140561
2024-10-08 06:05:20 +0000 INFO message processing complete component=worker thread=work1 original_queued_message=140559 queued_message=140561
2024-10-08 06:05:20 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work1 original_queued_message=140559 queued_message=140562
2024-10-08 06:05:20 +0000 INFO Sending message 1::140407 to [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140562
2024-10-08 06:05:21 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140554
2024-10-08 06:05:21 +0000 INFO message processing complete component=worker thread=work0 original_queued_message=140549 queued_message=140554
2024-10-08 06:05:22 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work0 original_queued_message=140549 queued_message=140555
2024-10-08 06:05:22 +0000 INFO Sending message 1::140400 to [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140555
2024-10-08 06:05:29 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140562
2024-10-08 06:05:29 +0000 INFO message processing complete component=worker thread=work1 original_queued_message=140559 queued_message=140562
2024-10-08 06:05:29 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work1 original_queued_message=140559 queued_message=140563
2024-10-08 06:05:29 +0000 INFO Sending message 1::140408 to [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140563
2024-10-08 06:05:31 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140555
2024-10-08 06:05:31 +0000 INFO message processing complete component=worker thread=work0 original_queued_message=140549 queued_message=140555
2024-10-08 06:05:31 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work0 original_queued_message=140549 queued_message=140556
2024-10-08 06:05:31 +0000 INFO Sending message 1::140401 to [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140556
2024-10-08 06:05:35 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140563
2024-10-08 06:05:35 +0000 INFO message processing complete component=worker thread=work1 original_queued_message=140559 queued_message=140563
2024-10-08 06:05:35 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work1 original_queued_message=140559 queued_message=140564
2024-10-08 06:05:35 +0000 INFO Sending message 1::140409 to [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140564
2024-10-08 06:05:37 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140556
2024-10-08 06:05:37 +0000 INFO message processing complete component=worker thread=work0 original_queued_message=140549 queued_message=140556
2024-10-08 06:05:38 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work0 original_queued_message=140549 queued_message=140557
2024-10-08 06:05:38 +0000 INFO Sending message 1::140402 to [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140557
2024-10-08 06:05:45 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140564
2024-10-08 06:05:45 +0000 INFO message processing complete component=worker thread=work1 original_queued_message=140559 queued_message=140564
2024-10-08 06:05:45 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work1 original_queued_message=140559 queued_message=140565
2024-10-08 06:05:45 +0000 INFO Sending message 1::140410 to [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140565
2024-10-08 06:05:47 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140557
2024-10-08 06:05:47 +0000 INFO message processing complete component=worker thread=work0 original_queued_message=140549 queued_message=140557
2024-10-08 06:05:47 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work0 original_queued_message=140549 queued_message=140558
2024-10-08 06:05:47 +0000 INFO Sending message 1::140403 to [email protected] log_id=I2CVSW1K component=worker thread=work0 original_queued_message=140549 queued_message=140558
2024-10-08 06:05:56 +0000 INFO Accepted by 8.8.8.8:25 (mx3.secretlab.lab) for [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140565
2024-10-08 06:05:56 +0000 INFO message processing complete component=worker thread=work1 original_queued_message=140559 queued_message=140565
2024-10-08 06:05:56 +0000 INFO parsing message content as it hasn't been parsed before component=worker thread=work1 original_queued_message=140559 queued_message=140566
2024-10-08 06:05:56 +0000 INFO Sending message 1::140411 to [email protected] log_id=AGVFQIVW component=worker thread=work1 original_queued_message=140559 queued_message=140566

FYI, these are all going to the same recipient, but they should be used to getting ALOT of mail. But yeah, the bottleneck could perhaps be on their end.

[willpower232]

I only have one worker, I tried enabling another one but it doesnt seem to work, and I cant really find any documentation on it, just some clues here and there in the forum, some of which seem to be outdated.

Presuming you're using the default installation, you can use the --scale argument to add more workers https://github.com/orgs/postalserver/discussions/1907#discussion-3939219

[UEAB-BL]

Thanks! Stopping postal with "postal stop" and starting it with "postal start --scale worker=4" did increase the worker count! =)
But now I have 26 queued emails that seem to be stuck since the restart of postal. Cant redeliver, it just says "Message is currently being processed." Its been like that for 25 minutes now. Is there a way to force retry these or something? Or is it something that will solve itself somehow?

[willpower232]

Unfortunately I think you would have to get into the database and clear a few fields for those messages, there might be some code somewhere in a discussion here about it but basically the messages are attached to the old instance of the workers and not the current instances. I don't think you needed to postal stop in order to apply the scale but I might be wrong.

[UEAB-BL]

Thanks, yeah that helped! I used some info here:
#1021
I ran this while connected to the database, the issue being that the job was locked by a worker that no longer existed. But this command resets everything in queue, so probobaly dont use it if you have a current queue of messages that are not locked by error.
use postal; update queued_messages set locked_by = NULL, locked_at = NULL, retry_after = NULL, ip_address_id = NULL;
For anyones information, you can connect to the database by running the following commands on your server:
docker exec -it postal-mariadb sh
mariadb --user=user_name --password=your_password db_name