Unable to setup - Kibana keeps failing

[ATragicEnding]

Describe the bug
Following instructions on the website, literally copy-pasting every single command. I have not changed any credentials in the .env file as I suspected it might be an issue but no matter what I do, I end up with Kibana failing , rendering the whole stack useless.

OS is Ubuntu Server 22.04 with latest udpates, running in a VM on Proxmox

To Reproduce
Steps to reproduce the behavior:

1. Follow installation instructions on the website to the dot.

Expected behavior
It should work

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information if pertinent):

• Assemblyline Version: Latest (Unsure, I never got to an actual login screen...)
• Browser: Firefox

Additional context
Add any other context about the problem here.

[cccs-rs]

I was able to perform the following (from the documentation) and was able to start up Assemblyline with the ELK components:

git clone https://github.com/CybercentreCanada/assemblyline-docker-compose.git
cd assemblyline-docker-compose/full_appliance/
openssl req -nodes -x509 -newkey rsa:4096 -keyout config/nginx.key -out config/nginx.crt -days 365 -subj "/C=CA/ST=Ontario/L=Ottawa/O=CCCS/CN=assemblyline.local" 
docker-compose pull
sudo docker-compose build
sudo docker-compose up -d --wait

Is there anything you can tell me about the containers that aren't starting up? Is there any error mentioned in their logs?

[Al3xTh3Gr3at]

Not sure if what you are describing is the error I had here : #234,
there is a work around in there, if you can confirm the ip subnets match.

[ATragicEnding]

I was able to perform the following (from the documentation) and was able to start up Assemblyline with the ELK components:

git clone https://github.com/CybercentreCanada/assemblyline-docker-compose.git
cd assemblyline-docker-compose/full_appliance/
openssl req -nodes -x509 -newkey rsa:4096 -keyout config/nginx.key -out config/nginx.crt -days 365 -subj "/C=CA/ST=Ontario/L=Ottawa/O=CCCS/CN=assemblyline.local" 
docker-compose pull
sudo docker-compose build
sudo docker-compose up -d --wait

Is there anything you can tell me about the containers that aren't starting up? Is there any error mentioned in their logs?

I have tested this on a fresh Ubuntu 22.04 container on Proxmox -- Container al-kb_setup-1 exists after about a minute and Container al-kibana-1 has been on waiting for a little over 10 minutes now with no change. URL can be accessed but no UI (I only get a warning about certificates but that's it).

Current status of all containers :
✔ Network al_core Created 0.0s
✔ Network al_registration Created 0.0s
✔ Network al_external Created 0.1s
✔ Volume "al_filestore" Created 0.0s
✔ Volume "al_datastore" Created 0.0s
✔ Volume "al_service_config" Created 0.0s
✔ Volume "al_redis" Created 0.0s
✔ Container al-elasticsearch-1 Healthy 60.8s
✔ Container al-frontend-1 Started 24.1s
✔ Container al-redis-1 Started 24.2s
✔ Container al-minio-1 Started 24.2s
✔ Container al-plumber-1 Started 53.4s
✔ Container al-workflow-1 Started 53.3s
✔ Container al-service_server-1 Started 52.5s
✔ Container al-metrics-1 Started 52.3s
✔ Container al-updater-1 Started 51.9s
✔ Container al-statistics-1 Started 52.4s
✔ Container al-ui-1 Started 52.7s
✔ Container al-socketio-1 Started 52.4s
✔ Container al-scaler-1 Started 52.8s
✔ Container al-expiry-1 Started 51.9s
✔ Container al-archiver-1 Started 52.2s
✔ Container al-dispatcher-1 Started 52.1s
✔ Container al-heartbeat-1 Started 51.6s
✔ Container al-kb_setup-1 Exited 56.3s
✔ Container al-ingester-1 Started 51.7s
✔ Container al-alerter-1 Started 52.4s
⠼ Container al-kibana-1 Waiting 570.5s
✔ Container al-apm_server-1 Created 0.0s
✔ Container al-nginx-1 Started 37.0s
✔ Container al-metricbeat-1 Created 0.0s
✔ Container al-filebeat-1 Created 0.0s

I did not change any informations in the .env for testing purpose but my container is not exposed to the internet.

[ATragicEnding]

Not sure if what you are describing is the error I had here : #234, there is a work around in there, if you can confirm the ip subnets match.

You might be correct as I do get a gateway timeout. I'll check your solution. Thank you !

Edit : Your workaround #1 command does not work for me.

[ATragicEnding]

So after trying again and noticing my VM is at 100% usage on both CPU and RAM/Swap, I've modified resources and allocated much more than is suggested (6C with 16GB RAM)

Installation SEEMS to go fine but I end up with archiver exiting. Based on those logs, it seems to be expected.
{"@timestamp": "2024-06-11 01:13:56,860", "event": { "module": "assemblyline", "dataset": "assemblyline.archiver" }, "host": { "ip": "x.x.x.x", "hostname": "784ce549fcaa" }, "log": { "level": "WARNING", "logger": "assemblyline.archiver" }, "process": { "pid": "1" }, "message": "Archive is not enabled in the config, no need to run archiver."} {"@timestamp": "2024-06-11 01:14:50,120", "event": { "module": "assemblyline", "dataset": "assemblyline.archiver" }, "host": { "ip": "x.x.x.x", "hostname": "784ce549fcaa" }, "log": { "level": "WARNING", "logger": "assemblyline.archiver" }, "process": { "pid": "1" }, "message": "Archive is not enabled in the config, no need to run archiver."}

Trying to login thru the web UI always return a "Wrong Username/Password" no matter the combination of Username/Password I use. Instructions on the website mention logging in with the admin user and password in the .env file but that does not work

[ATragicEnding]

Update #2 : I ran the sudo docker-compose -f bootstrap-compose.yaml up command after everything. I get this error :
failed to register layer: failed to Lchown "/opt/al_service/tools/node_modules/optionator/CHANGELOG.md" for UID 110779, GID 100 (try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): lchown /opt/al_service/tools/node_modules/optionator/CHANGELOG.md: invalid argument

[cccs-rs]

This looks like it's coming from the bootstrap from one of the services (to mind I think only JsJaws uses Node).

You can try running sudo docker-compose -f bootstrap-compose.yaml up first_time_setup to just run the job to initialize the system. You can then install the services later using the UI once you sign in using the admin credentials specified in the .env file.

If you can confirm which container that error is coming from, I can tag the relevant personnel.

(btw thanks for the support @Al3xTh3Gr3at !)

[cccs-kevin]

fix is https://github.com/m9sweeper/m9sweeper/pull/134/files but for the JsJaws service, will do

[kam193]

re:

failed to register layer: failed to Lchown "/opt/al_service/tools/node_modules/optionator/CHANGELOG.md" for UID 110779, GID 100 (try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): lchown /opt/al_service/tools/node_modules/optionator/CHANGELOG.md: invalid argument

@ATragicEnding are you sure you run a Virtual Machine in Proxmox, not an LXC/LXD container? I'm asking because I had very similar troubles in some containers (not only AssemblyLine), when I did one thing without full understanding: I installed Docker in an LXC container (also in Proxmox, but it doesn't matter). It turned out that this is not a recommended configuration because Docker and LXC use the same kernel features under the hood, and it causes problems like this (as well as a few others). The solution was to use a full VM to set up the Docker, not a container.