web server doesn't respond unless overriding Host header

[myster-t]

I followed the setup for postal using docker inside of an AWS Linux instance. Everything seems to be connected except the web server doesn't accept any web requests unless the header for Host is overwritten.

When I'm on the host server or within either the web or caddy dockers, either of these commands return the html page as expected:

wget -O - --header=Host:postal.mydomain.com http://localhost:5000/login?return_to=%2F
curl -vvv -H 'Host:postal.mydomain.com' http://localhost:5000/login?return_to=%2F

However, if I don't override the Host header, then the web server 403s. For instance, from within the web docker:

$ wget http://localhost:5000/login?return_to=%2F
--2024-09-09 01:51:05--  http://localhost:5000/login?return_to=%2F
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-09-09 01:51:05 ERROR 403: Forbidden.

What exactly is causing this to happen? Is there a postal.yml config I need to set?

Separately, if this is how the world works, then what should I set in my Caddyfile to reverse proxy this correctly? I tried this, caddy gets the response and logs the request, but the web server 403s:

postal.mydomain.com {
  reverse_proxy 172.0.0.1:5000 {
    header_up Host "postal.mydomain.com"
  }
}

[willpower232]

There must be something wrong in your config file already, it should just work normally unless you have other proxies or something else in the way

https://github.com/postalserver/install/blob/main/examples/Caddyfile

What did you set as web_hostname under postal in your config file?

[myster-t]

it's extremely vanilla haha

version: 2

postal:
  web_hostname: postal.mydomain.com
  web_protocol: https
  smtp_hostname: postal.mydomain.com

main_db:
  host: postal.cluster.us-west-2.rds.amazonaws.com
  username: postal
  password: <password>
  database: postal_mydomain_com

message_db:
  host: postal.cluster.us-west-2.rds.amazonaws.com
  username: postal
  password: <password>
  prefix: postal_mydomain_com_msg

smtp_server:
  default_bind_address: "::"

dns:
  # Specify the DNS records that you have configured. Refer to the documentation at
  # https://github.com/atech/postal/wiki/Domains-&-DNS-Configuration for further
  # information about these.
  mx_records:
    - postal.mydomain.com
  spf_include: spf.postal.mydomain.com
  return_path_domain: rp.postal.mydomain.com
  route_domain: routes.postal.mydomain.com
  track_domain: track.postal.mydomain.com

smtp:
  # Specify an SMTP server that can be used to send messages from the Postal management
  # system to users. You can configure this to use a Postal mail server once the
  # your installation has been set up.
  host: postal.mydomain.com
  port: 25
  username: # Complete when Postal is running and you can
  password: # generate the credentials within the interface.
  from_name: Postal Management Services
  from_address: [email protected]

rails:
  # This is generated automatically by the config initialization. It should be a random
  # string unique to your installation.
  secret_key: <autogen secret>

i didn't really configure anything ¯_(ツ)_/¯. i followed the instructions step by step. in fact, i even wrote down the steps i took so i can build a second postal server later:

# start aws linux t3a.medium with 30gb, postal vpc and postal keypair

yum install git jq
yum install docker
sudo mkdir -p /usr/local/lib/docker/cli-plugins/
sudo curl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 -o /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose

git clone https://github.com/postalserver/install /opt/postal/install
sudo ln -s /opt/postal/install/bin/postal /usr/bin/postal

service docker start

# update /opt/postal/config/postal.yml

postal initialize
postal make-user

postal start

docker run -d \
   --name postal-caddy \
   --restart always \
   --network host \
   -v /opt/postal/config/Caddyfile:/etc/caddy/Caddyfile \
   -v /opt/postal/caddy-data:/data \
   caddy

that's all i did and then i started debugging why the web ui wasn't accessible from my computer by testing if it was accessible from within the host shell. that's when i started testing with the Host header and trying out curls and wgets from within the host shell and the docker shells. it does seem everything's up and running (db's connected, ports are open, dns is set up, etc), but the web server is simply not responding to any requests without the Host header set to postal.mydomain.com

[willpower232]

Interesting, all looks good. Are you able to try postal logs web and see if there are any errors in case something else is struggling to get through?

[myster-t]

haha, nothing insightful:

fresh restart:

[root@ip-172-31-15-126 install]# postal logs web
web-1  | Loading config from /config/postal.yml
web-1  | Puma starting in single mode...
web-1  | * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
web-1  | *  Min threads: 5
web-1  | *  Max threads: 5
web-1  | *  Environment: production
web-1  | *          PID: 10
web-1  | * Listening on http://0.0.0.0:5000
web-1  | Use Ctrl-C to stop

request with header (i cut out the output to save space in this response) and the web logs after that:

[root@ip-172-31-15-126 install]# wget -O - --header=Host:postal.mydomain.com http://localhost:5000/login?return_to=%2F
--2024-09-10 03:26:57--  http://localhost:5000/login?return_to=%2F
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1940 (1.9K) [text/html]
Saving to: ‘STDOUT’

-                                                                          0%[                                                                                                                                                                                 ]       0  --.-KB/s               <!DOCTYPE html>
<html class='subPage'>
...
</html>
-                                                                        100%[================================================================================================================================================================================>]   1.89K  --.-KB/s    in 0s

2024-09-10 03:26:57 (224 MB/s) - written to stdout [1940/1940]

[root@ip-172-31-15-126 install]# postal logs web
web-1  | Loading config from /config/postal.yml
web-1  | Puma starting in single mode...
web-1  | * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
web-1  | *  Min threads: 5
web-1  | *  Max threads: 5
web-1  | *  Environment: production
web-1  | *          PID: 10
web-1  | * Listening on http://0.0.0.0:5000
web-1  | Use Ctrl-C to stop
web-1  | 2024-09-10 03:26:57 +0000 INFO   GET /login?return_to=%2F (200) event=request transaction=1bd326a3b20519c27916 controller=SessionsController action=new format=*/* method=GET path=/login?return_to=%2F request_id=52bb94bf-e433-40ad-ac54-e65f9c81dbfb ip_address=172.22.0.1 status=200 view_runtime=33.94002598361112 db_runtime=94.09992599487305
web-1  | 172.22.0.1 - - [10/Sep/2024:03:26:57 +0000] "GET /login?return_to=%2F HTTP/1.1" 200 - 0.2009

request without header and the web logs after that:

[root@ip-172-31-15-126 install]# wget http://127.0.0.1:5000/login?return_to=%2F
--2024-09-10 03:27:09--  http://127.0.0.1:5000/login?return_to=%2F
Connecting to 127.0.0.1:5000... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-09-10 03:27:09 ERROR 403: Forbidden.

[root@ip-172-31-15-126 install]# postal logs web
web-1  | Loading config from /config/postal.yml
web-1  | Puma starting in single mode...
web-1  | * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
web-1  | *  Min threads: 5
web-1  | *  Max threads: 5
web-1  | *  Environment: production
web-1  | *          PID: 10
web-1  | * Listening on http://0.0.0.0:5000
web-1  | Use Ctrl-C to stop
web-1  | 2024-09-10 03:26:57 +0000 INFO   GET /login?return_to=%2F (200) event=request transaction=1bd326a3b20519c27916 controller=SessionsController action=new format=*/* method=GET path=/login?return_to=%2F request_id=52bb94bf-e433-40ad-ac54-e65f9c81dbfb ip_address=172.22.0.1 status=200 view_runtime=33.94002598361112 db_runtime=94.09992599487305
web-1  | 172.22.0.1 - - [10/Sep/2024:03:26:57 +0000] "GET /login?return_to=%2F HTTP/1.1" 200 - 0.2009
web-1  | 172.22.0.1 - - [10/Sep/2024:03:27:09 +0000] "GET /login?return_to=%2F HTTP/1.1" 403 - 0.0008
[root@ip-172-31-15-126 install]#

[myster-t]

ok, i went into the web instance and hacked around with the puma instance to dump out debug logs and got this for when the web server 403s:

[ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 127.0.0.1:5000
172.22.0.1 - - [10/Sep/2024:03:52:03 +0000] "GET /login?return_to=%2F HTTP/1.1" 403 - 0.0042

if i update config/environment/production.rb to include:

config.hosts << "127.0.0.1"

then all of a sudden the web server works... but unless i go around hacking the puma docker instance for the postal web server, i'm hoping for a solution that doesn't require this puma config change. any ideas?

[myster-t]

just an update. i gave up. amazon linux isn't (out of the box) compatible with postal.

separately, i migrated to ubuntu and installing docker via snap install docker isn't compatible with the bin/postal wrapper scripts as well. install docker on ubuntu via the convenience script:

 curl -fsSL https://test.docker.com -o test-docker.sh
 sudo sh test-docker.sh

[willpower232]

Yeah not a fan of snaps personally, is Postal working better for you now?

[myster-t]

yea, it's working! Took a lot of tinkering to figure out Postal needs a very specific env, haha.

Thanks for checking back