Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intrusion attempts during image build cause non-empty logs #90

Open
ximon18 opened this issue Feb 1, 2020 · 1 comment
Open

Intrusion attempts during image build cause non-empty logs #90

ximon18 opened this issue Feb 1, 2020 · 1 comment

Comments

@ximon18
Copy link

ximon18 commented Feb 1, 2020 •
edited
Loading

It is possible that the Droplet is attacked between the invocation of the DigitalOcean Marketplace cleanup.sh which truncates the logs and img_check.sh scripts which checks the logs, resulting in img_check.sh warning about non-empty logs due to logged intrusion attempts, and thus failing the Packer build.

Or at least I assume that is what was happening in my case, see below. I could not otherwise explain the sudden appearance of firewall related log entries after the log had been cleared.

Stopping UFW isn't an option as then img_check.sh complains. Stopping UFW logging only prevents some logging, not all. Stopping rsyslog didn't help either.

My solution was to apply a DigitalOcean firewall to the Droplet created by Packer that allowed SSH only from my IP address, thereby preventing the unwanted communication attempts from reaching the VM and thus being logged.

Automating this is non-trivial as the Packer DigitalOcean builder doesn't support enabling a DO firewall on the Droplet, nor does Packer expose the Droplet ID to the template. In theory one can use the cloud-init query instance_id command in a shell provisioner to write the Droplet ID to a file, then download it using a file provisioner and then use a shell-local provisioner to execute doctl compute firewall create to create the firewall, and then delete the firewall at the end. However, Packer doesn't have the notion of steps to always run and so the teardown of the firewall will not happen if any of the prior build steps fail, and running shell-local commands assumes the host has doctl installed and configured with the right DO API token, and brittle assumptions about which command syntax will work on the host have to be made (is it Linux, is it Windows?).

If you have any advice on how to avoid this issue that would be most appreciated. Alternatively, extending the DigitalOcean Packer builder to be able to add a firewall to the created Droplet limiting SSH to "my ip" (perhaps via a Packer user variable) would help work around this problem.

Here is an example of the problem that I encountered:

img_check.sh warnings:

    digitalocean: [WARN] un-cleared log file, /var/log/auth.log found
    digitalocean: [WARN] un-cleared log file, /var/log/kern.log found
    digitalocean: [WARN] un-cleared log file, /var/log/ufw.log found

Actual log file contents at the time:

    digitalocean: /var/log/alternatives.log
    digitalocean: /var/log/apt
    digitalocean: /var/log/auth.log
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: fatal: No supported key exchange algorithms [preauth]
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
    digitalocean: Feb  1 20:33:17 packer-vm-name sshd[2575]: fatal: No supported key exchange algorithms [preauth]
    digitalocean: /var/log/btmp
    digitalocean: /var/log/cloud-init-output.log
    digitalocean: /var/log/dist-upgrade
    digitalocean: /var/log/dpkg.log
    digitalocean: /var/log/journal
    digitalocean: /var/log/kern.log
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/landscape
    digitalocean: /var/log/lastlog
    digitalocean: /var/log/lxd
    digitalocean: /var/log/syslog
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:32:41 packer-vm-name systemd-resolved[664]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/tallylog
    digitalocean: /var/log/ufw.log
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/wtmp
@ximon18
Copy link
Author

ximon18 commented Feb 4, 2020

FYI my current way of automating the work around for this problem is to run the following command after launching Packer:

doctl compute droplet list --format "ID,Name" | fgrep packer | awk '{print $1}' | xargs doctl compute firewall add-droplets XXXX --droplet-ids

Where XXX is the ID of a firewall I created whose only rule is to permit SSH only from my public IP address.

@m90 m90 mentioned this issue Apr 3, 2020
Open
daniellockyer added a commit to TryGhost/digitalocean-1-click that referenced this issue Jun 8, 2020
@daniellockyer
Removed img_check script from list
3503b36 
refs digitalocean/marketplace-partners#90 and digitalocean/marketplace-partners#95

- this script is breaking the build because there are some leftover logs
  from the build process that it fails on
- the logs cannot be easily removed and this issue is waiting on
  upstream resolution
daniellockyer added a commit to TryGhost/digitalocean-1-click that referenced this issue Jun 8, 2020
@daniellockyer
Disabled log checking in img_check
31adbeb 
refs digitalocean/marketplace-partners#90 and
digitalocean/marketplace-partners#95

- there may be some leftover logs from the snapshot process which would
  cause this to fail
- commenting this out until upstream have a solution
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ximon18