Skip to content

Commit

Permalink
Merge pull request #3 from CloudStationTeam/dev
Browse files Browse the repository at this point in the history 
Dev to master
  • Loading branch information
@PeterJBurke
PeterJBurke authored Jan 29, 2024
2 parents cc3c735 + 7486c1d commit ef78442
Show file tree
Hide file tree
Showing 4 changed files with 39 additions and 116 deletions.
20 changes: 8 additions & 12 deletions source/contributing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ you just want to quickly set up a development environment, please follow the ins

## General Notes
1. The front end currently uses vanilla JS. It would be best to refactor it in a way that is more sustainable for future development. Using some kind of framework (Angular, React, Vue) should help.
2. **Back End**
* The APIs we implemented don't follow the popular RESTful style. We decided to make them structured in a way that's very similar to MAVLink commands/messages. It makes initial development a lot easier but I imagine it will make it difficult to add some more advanced features. I think having RESTful APIs actually does make sense for this project. An example of the resources would be a JSON document of all the telemetry information of the vehicle. The backend should send the correct command according to the current speed in that document, etc. This design should make it easier to implement the front end and makes it more structured.
3. **Security**

2. **Security**
* This is probably the biggest problem of the current implementation. I don't have any experience in this field so I can't provide much information.
* Currently, CloudStation cannot be used safely for multiple users. User accounts do not actually link drones to their respective accounts, and there are no safeguards in place to prevent one user from entering the drone ID of a different user. Editing the additional telemetry data displayed in "Other Data" will also affect settings for all users.
* You could check who accessed the website by \
Expand All @@ -33,7 +32,7 @@ you just want to quickly set up a development environment, please follow the ins
`less /var/log/nginx/access.log` \
There maybe many hackers (which is common).

4. **Debug**
3. **Debug**
* To reload the server, you could do `bash reload_server.sh`. However, if you updated requirements.txt, you have to do `rm -r ENV` and deploy everything again.
* You could use print statements or Python logging, however, it's recommended to use Python logging because it's a good practice. You could configure the logs in settings.py. The logs are not displayed by Django even if you run it in front-end, it's configured by daphne.service. So after you reload the server, you could do `journalctl -u daphne.service | tail` for print statements. You may output it to a log file or use Splunk for it to be more convenient.
* Debugger Tools: \
Expand All @@ -47,21 +46,18 @@ you just want to quickly set up a development environment, please follow the ins
```
## Feature Improvements
1. Enhancing multi-user support and security (see General Notes - Security)
* Saving drone IDs for each user
* Some sort of safeguard to prevent a user from controlling a drone that is not theirs
* Saving telemetry data displayed in "Other Data" individually for each user
2. There is currently no support for takeoff, landing, etc., so vehicle types other than rovers cannot be controlled through CloudStation.
3. The fly-to feature currently sets all fly-to points to 0 altitude, which means it can only be used for rovers. If support for flying vehicles is added, the fly-to feature will need altitude controls.
4. Waypoint missions are currently partially implemented in the backend but have no front-end interface and have not been tested.
5. Controlling multiple drones at a time feels very unintuitive and clunky. Improvements could include:
2. Waypoint missions are currently partially implemented in the backend but have no front-end interface and have not been tested.
3. Controlling multiple drones at a time feels very unintuitive and clunky. Improvements could include:
* Automatically selecting the tab of a drone when it is clicked on
* Automatically bringing up the popup for a drone when its tab is selected
* Automatically selecting the tab of a drone when a pin belonging to that drone is clicked
* Color-coding or otherwise differentiating pins belonging to different drones
* etc.
6. Adding an HUD (with an artificial horizon, etc.) for the currently selected drone
7. Adding video streaming support for the currently selected drone
8. Using different vehicle icons based on the type of vehicle detected
4. Adding an HUD (with an artificial horizon, etc.) for the currently selected drone
5. Adding video streaming support for the currently selected drone
6. Using different vehicle icons based on the type of vehicle detected
125 changes: 22 additions & 103 deletions source/deployment.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ Before you start, you will need an AWS account (free trial version is OK). The f
You will also need a free [MapBox](https://account.mapbox.com/) account. Make sure you can find your Mapbox public token, located on the home page after logging in.

#### Deployment Steps
For a step by step video guide, see [here](https://www.youtube.com/watch?v=kNkkfPRBoJo). \
Another [video](https://youtu.be/MzEE0EAfrfY) for CloudStation Deployment on Amazon Web Services Linux 22.4.

1. Launch an EC2 instance on AWS with Ubuntu 22.4 LTS
* t2.micro (free-tier eligible) is good enough to test the deployment.
Expand All @@ -33,59 +31,22 @@ Another [video](https://youtu.be/MzEE0EAfrfY) for CloudStation Deployment on Ama
run
```
cd ~
git clone https://github.com/CloudStationTeam/cloud_station_deployment.git
bash ~/cloud_station_deployment/setup_server.sh
git clone https://github.com/CloudStationTeam/cloud_station_deployment.git
bash ~/cloud_station_deployment/setup_server.sh YOUR_MAP_BOX_KEY_HERE_NO_QUOTES
```
This deploys an instance of the CloudStation web server based on the latest commit. Note that the latest commit may be developmental. For a more stable version, you can run setup_server.sh with a flag to indicate a certain release tag, e.g. `bash ~/cloud_station_deployment/setup_server.sh --tag=v3.0`.
* You can see the list of releases [here](https://github.com/CloudStationTeam/cloud_station_web/releases). The tag for the release is indicated by the text following the tag icon to the left of the release icon, e.g. v2.0 or v1.0.
This deploys an instance of the CloudStation web server based on the latest commit. Note that the latest commit may be developmental. For a more stable version, you can run setup_server.sh with a flag to indicate a certain release tag, e.g. `bash ~/cloud_station_deployment/setup_server.sh --tag=v4.0`.
* You can see the list of releases [here](https://github.com/CloudStationTeam/cloud_station_web/releases). The tag for the release is indicated by the text following the tag icon to the left of the release icon, e.g. v4.0.
* setup_server.sh does the following：
1. Update Ubuntu
2. Install NGINX and docker
2. Install NGINX and docker, Django , configures them
3. Clone cloud_station_web source code
4. Set up a Python virtual environment (install dependencies)
5. Modify cloud_station_web/webgms/settings.py
* Add EC2 IP address/DNS to ALLOWED_HOST
* DNS example: "ec2-xx-xx-xxx-xxx.us-west-1.compute.amazonaws.com" (it should be a string, please do not forget the quotation marks)
* Set DEBUG to False
* Set MAPBOX_PUBLIC_KEY to your Mapbox public token
6. Add your Google Map API Public Key to cloud_station_web/.env. \
Note: no space and no quotes. e.g. GOOGLE_MAP_API_KEY=something. \
(You have some free credits for free tier by Google Cloud and it does not cost money under $200 free API limit (which is about several dollars per 1k reqs), and you could opt to Not cost money if it does not exceed the free API limit.)
7. Modify cloud_station_deployment/nginx.conf
* add EC2 IP/DNS address to Line 68: server_name ec2-xx-xx-xxx-xxx.us-west-1.compute.amazonaws.com
8. Configure NGINX, Daphne and Django (run only once)
```
sudo usermod -a -G ubuntu www-data #add nginx user to the ubuntu group
# optional
sudo chgrp -R ubuntu * #change file owners to ubuntu group
sudo chmod -R g+r * #change file permissions of file owners
```
Then run ```bash ~/cloud_station_deployment/configure_web_server.sh```
The script does the following:
1. Write database migrations
2. Collect staticfiles to ~cloud_station_web/static
3. Configure NGINX with nginx.conf
4. Configure systemctl to automatically run Daphne as a service(daphne.service)
5. Download redis and start running redis in a docker container
10. In your web browser, go to your EC2 instance's DNS address (ec2-xx-xx-xxx-xxx.us-west-1.compute.amazonaws.com) and you should see the CloudStation website. \
Note that if it didn't work, reload the server by reload_server.sh.
#### Notes
1. Note: if the CloudStation website link didn't work on Amazon, don't click on it on Amazon but copy and paste it in a new website.
2. Note: public keys for cloud_station_web/.env are all uppercase letters. If you use a lower case (for example, SMT_smt), it won't work. It may work in fg (JS) but not bg (Python).
3. Public keys are in .env not settings.py. \
Because it's easier to manage if you have other APIs, and it's safer because 1. .env is added in .gitignore and 2. we don't use the APIs in frontend because it's not a secure practice. we used the Mapbox API in frontend because it does not cost money so it doesn't matter.
4. Note that you have to modify setup_server.sh (and without the tag) and reload_server.sh if your cloud_station_web repo is a different one.
5. Note that if at some point you did something wrong / you edited requirements.txt, remove all the files and restart the process. \
Refer to cloud_station_deployment/rmv_env.sh and edit the files.
4. Set up a Python virtual environment (install dependencies)
5. Configures mapbox api key
6. Write database migrations (Django)
7. Configure NGINX with nginx.conf
8. Configure systemctl to automatically run Daphne as a service(daphne.service)
8. Download redis and start running redis in a docker container
5. In your web browser, go to your EC2 instance's DNS address (ec2-xx-xx-xxx-xxx.us-west-1.compute.amazonaws.com) and you should see the CloudStation website. Note that modern browsers default to https, but for this use http:// in front of the IP address.
#### Redeployment
To reload the server (after a code update)
Expand All @@ -97,13 +58,18 @@ To reload the server (after a code update)
4. Reload NGINX and Daphne
5. Run django_background_tasks
Note that If frontend didn't work, for example If JS files are not updated, clear browser cache. (Typically control shift R to reload).
#### Testing Using SITL
To test CloudStation with a simulated drone instead of a real drone, you can install and run [SITL](https://ardupilot.org/dev/docs/sitl-simulator-software-in-the-loop.html). If you have a Windows computer running Cygwin, you can also use our [SITL deployment script](https://github.com/CloudStationTeam/sitl_deployment) to automate running multiple SITL instances.
For the example SITL command,
Note that If frontend didn't work, for example If JS files are not updated, clear browser cache.
cd ~/ardupilot/ArduCopter; sim_vehicle.py --console --map --osd --out=udp:35.94.121.200:14550 --custom-location=33.64586111,-117.84275,25,0
#### Testing Using SITL
To test CloudStation with a simulated drone instead of a real drone, you can install and run [SITL](https://ardupilot.org/dev/docs/sitl-simulator-software-in-the-loop.html). If you have a Windows computer running Cygwin, you can also use our [SITL deployment script](https://github.com/CloudStationTeam/sitl_deployment) to automate running multiple SITL instances.
the address of the cloudstation instance is 35.94.121.200. You can use your cloudstation instance IP address instead.
Professor Burke's github has a single script that will deploy SITL to a linux machine, see: https://github.com/PeterJBurke/CreateSITLenv.
#### Restarting Ubuntu
If you need to restart Ubuntu, run the following:
Expand All @@ -118,55 +84,12 @@ There are many ways to configure the drone to do this. The easiest is to have a
[J70] Peter J. Burke “A Safe, Open Source, 4G Connected Self-Flying Plane With 1 Hour Flight Time and All Up Weight (AUW) <300 g: Towards a New Class of Internet Enabled UAVs”
IEEE Access, 7(1), 67833 – 67855 (2019).
## AWS RDS (Aurora engine) - Experimental
Note that the project uses SQLite due to its low cost and ease of use with Django. However, AWS RDS can be configured for scalability and robustness.
1. Launch an RDS instance on AWS with Aurora with MySQL compatibility
* db.r5.large is good enough to test the deployment
* Configure Security Group (AWS EC2 Console)
* MySQL/Aurora(TCP) Port:3306 Source:My RDS endpoint
2. Add the RDS_DB_NAME, RDS_USERNAME, RDS_PASSWORD, RDS_HOSTNAME and RDS_PORT to the environment variables
* You can do this by editing the ~/.env file and adding the variables in the following format.
```
variable_name=value
```
* Add this to the ~/.bash_profile file
```
set -a
. ~/.env
set +a
```
* Make sure to run this command after editing the ~/.bash_profile file.
```
source ~/.bash_profile
```
3. Edit the cloud_station_web/webgms/settings.py file and change the DATABASES field to the following
```
import os
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
```
4. Edit the cloud_station_deployment/backgroundtasks.service and add this line to the [Service] section.
```
EnvironmentFile=/home/ubuntu/.env
```
5. run ```bash ~/cloud_station_deployment/configure_web_server.sh```
6. run ```bash ~/cloud_station_deployment/reload_server.sh```
## Authors
* Lyuyang Hu
* Omkar Pathak
* Peter Burke
## Troubleshooting
1. How do I know my hardware setup is correct (the vehicle is sending mavlink messages to the server)?
Expand All @@ -176,9 +99,7 @@ Note that the project uses SQLite due to its low cost and ease of use with Djang
service nginx status
service daphne status
```
3. How do I know the status of django_background_tasks?
```service backgroundtasks status```
4. The telemetry textbox shows that the websocket connection between server and browser has been disconnected. What do I do?
3. The telemetry textbox shows that the websocket connection between server and browser has been disconnected. What do I do?
* This usually means Redis fails. Following the Django Channels recommendation, we use Redis as the backing store for the channel layer. We use Docker to run Redis.
* To check status of Docker: ```service docker status```
* To show all Docker containers on the machine: ```sudo docker ps -a```
Expand All @@ -188,6 +109,4 @@ Note that the project uses SQLite due to its low cost and ease of use with Djang
sudo systemctl enable docker
sudo docker run -p 6379:6379 -d redis:2.8
```
5. If you are using a Python version past 3.6, Python does not like "pkg-resources" anymore. Go to the requirements.txt file and comment out the following line:
'''pkg-resources==0.0.0'''
5 changes: 4 additions & 1 deletion source/sitl.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# SITL Deployment
# SITL Deployment (Linus)
See https://github.com/PeterJBurke/CreateSITLenv.

# SITL Deployment (Windows)
For SITL deployment ([An Intro Video](https://youtu.be/KfAOsSYtokA)), you may want to start another Windows ec2 ([An Intro Video](https://youtu.be/L0LnWJfgDeM)) and connect to it by rdp, then
1. download https://github.com/ArduPilot/ardupilot/blob/master/Tools/environment_install/install-prereqs-windows.ps1, then right click to run it by ps (then type A) (to download cygwin and mavproxy).

Expand Down
5 changes: 5 additions & 0 deletions source/webapp.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,8 @@
[x] Add customizable telemetry options

[] Link customizable telemetry options to specific users

[x] Waypoints (Cloudstation V 3.0 (Fall 2023)). This was a student project that crashed the drone. So it is on the back burner for now

[x] Cloudstation 4.0 (Released January 2024). Add support for all flight modes, arm/disarm clearly on UI, support land/takeoff, RTL.

0 comments on commit ef78442

Please sign in to comment.