Skip to content

Commit

Permalink
[DOCs/operators]: Create a Nym ISP sheet, write gateway-probe guide &…
Browse files Browse the repository at this point in the history
… clean FnF docs (#4590)

* initialise nym versin of good-bad isps by tor community

* syntax edit

* finalise csv2md

* initialise isp-list page

* add last update info

* add python3 plugis to ci/cd runnners

* remove argparse installation

* python3 syntax edit

* pip install attempt

* pip install attempt

* change python version in cmdrun command

* fix python modules installation

* correct command python version

* workflows python env update

* add probe guide, clean FnF pages

* fix typos
  • Loading branch information
serinko authored May 20, 2024
1 parent a06ae48 commit 0eb6eb8
Show file tree
Hide file tree
Showing 11 changed files with 228 additions and 13 deletions.
6 changes: 5 additions & 1 deletion .github/workflows/cd-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ jobs:
steps:
- uses: actions/checkout@v3
- name: Install Dependencies (Linux)
run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git
run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git python3 && sudo apt-get update --fix-missing
- name: Install pip3
run: sudo apt install -y python3-pip
- name: Install Python3 modules
run: sudo pip3 install pandas tabulate
- name: Install rsync
run: sudo apt-get install rsync
- uses: rlespinasse/[email protected]
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/ci-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@ jobs:
steps:
- uses: actions/checkout@v3
- name: Install Dependencies (Linux)
run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git
run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git python3 && sudo apt-get update --fix-missing
- name: Install pip3
run: sudo apt install -y python3-pip
- name: Install Python3 modules
run: sudo pip3 install pandas tabulate
- name: Install rsync
run: sudo apt-get install rsync
- uses: rlespinasse/[email protected]
Expand Down
9 changes: 6 additions & 3 deletions documentation/operators/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,15 @@
- [Maintenance](nodes/maintenance.md)
- [Manual Node Upgrade](nodes/manual-upgrade.md)
- [Automatic Node Upgrade: Nymvisor Setup and Usage](nodes/nymvisor-upgrade.md)
- [Performance Testing](testing/performance.md)
- [Node Setup](testing/node-setup.md)
- [Metrics Monitoring](testing/templates.md)
- [Performance Monitoring & Testing](testing/performance.md)
<!--- [Node Setup](testing/node-setup.md)-->
- [Gateway Probe](testing/gateway-probe.md)
- [Prometheus & Grafana](testing/prometheus-grafana.md)
- [ExploreNYM scripts](testing/explorenym-scripts.md)
<!-- - [Run in a Docker](testing/docker-monitor.md) -->



# Troubleshooting

- [VPS Setup](troubleshooting/vps-isp.md)
Expand All @@ -56,6 +58,7 @@

- [Exit Gateway](legal/exit-gateway.md)
- [Community Counsel](legal/community-counsel.md)
- [ISP List](legal/isp-list.md)
- [Jurisdictions](legal/jurisdictions.md)
- [Switzerland](legal/swiss.md)
- [United States](legal/united-states.md)
Expand Down
8 changes: 8 additions & 0 deletions documentation/operators/src/data/isp-sheet.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
**ISP**,**Locations**,**Public IPv6**,**Crypto Payments**,**Comments**,**Last Updated**
[Flokinet](https://flokinet.is),"Netherlands, Iceland, Romania,France","Yes, needs a ticket and custom setup","yes, including XMR","Very slow customer support","05/2024"
[BitLaunch](https://bitlaunch.io),"Canada, USA, UK","No","Yes","Expensive. Digial Ocean through BitLanch has IPv6","05/2024"
[Hostinger](https://hostinger.com),"France, Lithuania, India, USA, Brazil","Yes, out of the box","Yes","Crypto payments must be done per each server monthly or annually.","05/2024"
[Linode](https://linode.com),"USA, Canada, Japan, India, Indonesia, Sweden, Netherlands, Germany, Brazil, France, UK, Australia, Italy","Yes out of the box","No, only through [BitLAunch](https://bitlaunch.io)","IPv6 sometimes need to be re-added in Networking tab, no reboot needed","05/2024"
[Cherry Servers](https://www.cherryservers.com),"Lithuania, Netherlands, USA, Singapore","No","Yes","Issued IP doesn’t match the location offered by the provider.","05/2024"
[Njalla](https://nja.la),"Sweden","Yes","Yes","Privacy vandguards! The biggest VPS 45 is 3 cores only, but it works better than many “larger” servers on the market.","05/2024"
[HostSailor](https://hostsailor.com),"USA","Yes, based on ticket","Yes","The IPv6 setup needs custom research and is not documented","05/2024"
25 changes: 25 additions & 0 deletions documentation/operators/src/legal/isp-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Where to host your `nym-node`?

```admonish info
The entire content of this page is under [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/).
```

Inspired by a valuable resource, done by Tor community - [*Good Bad ISPs*](https://community.torproject.org/relay/community-resources/good-bad-isps/), LunarDAO squad initiated a table customised for Nym Exit Gateways operators.

This ISP list is fully managed by Nym operator community and it serves as a space to share their experience of running Exit Gateways on various Internet Service Providers (ISPs). The ISPs greatly differ in regards to services they offer as well as to their openess of hosting exit routing software.

Please share any experiences running a node like policies, complains, legal issues and solutions, discrepancy between offers and reality (bandwidth, IP range, locations) or anything regarding pricing or customer support.

If you came across any legal findings, please share them in our [list of jurisdictions](jurisdictions.md).

While we trust that Nym node operators are honest, we would like to ask everyone to do your own research.

```admonish caution title=""
To edit or add information to the ISP list, make changes to the csv file located [here](https://github.com/nymtech/nym/blob/develop/documentation/operators/src/data/isp-sheet.csv) and submit your edits as a pull request according to [this guide](add-content.md).
```

```admonish note title=""
As of now the list is quite short. When it grows, we can divide it according the localities of the listed ISPs.
```

<!--cmdrun python3 ../../../scripts/csv2md.py ../data/isp-sheet.csv -s 0 -->
4 changes: 2 additions & 2 deletions documentation/operators/src/nodes/vps-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ A suboptimally configured VPS often results in a non-functional node. To follow

You will need to rent a VPS to run your node on. One key reason for this is that your node **must be able to send TCP data using both IPv4 and IPv6** (as other nodes you talk to may use either protocol).

Tor community created a very helpful table called [*Good Bad ISPs*](https://community.torproject.org/relay/community-resources/good-bad-isps/), use that one as a guideline for your choice of ISP for your VPS.
Tor community created a very helpful table called [*Good Bad ISPs*](https://community.torproject.org/relay/community-resources/good-bad-isps/), you can use that one as a guideline for your choice of ISP for your VPS.

Currently we run [performance testing](../testing/performance.md) events to find out the best optimization. Sphinx packet decryption is CPU-bound, so more fast cores the better throughput.
**Update:** Nym community started an ISP table called [*Where to host your nym node?*](../legal/isp-list.md), check it out and add your findings!

### `nym-node`

Expand Down
79 changes: 79 additions & 0 deletions documentation/operators/src/testing/gateway-probe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Nym Gateway Probe

Nym Node operators running Gateway functionality are already familiar with the monitoring tool [Harbourmaster.nymtech.net](https://harbourmaster.nymtech.net). Under the hood of Nym Harbourmaster runs iterations of `nym-gateway-probe` doing various checks and displaying the results on the interface. Operators don't have to rely on the probe ran by Nym and wait for the data to refresh. With `nym-gateway-probe` everyone can check any Gateway's networking status from their own computer at any time. In one command the client queries data from:

- [`nym-api`](https://validator.nymtech.net/api/)
- [`explorer-api`](https://explorer.nymtech.net/api/)
- [`harbour-master`](https://harbourmaster.nymtech.net/)


## Preparation

We recommend to have install all [the prerequisites](../binaries/building-nym.md#prerequisites) needed to build `nym-node` from source including latest [Rust Toolchain](https://www.rust-lang.org/tools/install).

## Installation

`nym-gateway-probe` source code is in [`nym-vpn-client`](https://github.com/nymtech/nym-vpn-client) repository. The client needs to be build from source.

1. Clone the repository:

```sh
git clone https://github.com/nymtech/nym-vpn-client.git
```

2. Build `nym-gateway-probe`:

```sh
cd nym-vpn-client

cargo build --release -p nym-gateway-probe
```

## Running the client

```sh
./target/release/nym-gateway-probe --help
```
~~~admonish collapsible=true
```
Usage: nym-gateway-probe [OPTIONS]
Options:
-c, --config-env-file <CONFIG_ENV_FILE> Path pointing to an env file describing the network
-g, --gateway <GATEWAY>
-n, --no-log
-h, --help Print help
-V, --version Print version
```
~~~

To run the client, simply add a flag `--gateway` with a targeted gateway identity key.

```sh
./target/release/nym-gateway-probe --gateway <GATEWAY_IDENTITY_KEY>
```

For any `nym-node --mode exit-gateway` the aim is to have this outcome:
```sh
{
"gateway": "<GATEWAY_IDENTITY_KEY>",
"outcome": {
"as_entry": {
"can_connect": true,
"can_route": true
},
"as_exit": {
"can_connect": true,
"can_route_ip_v4": true,
"can_route_ip_external_v4": true,
"can_route_ip_v6": true,
"can_route_ip_external_v6": true
}
}
}
```

If you don't provide a `--gateway` flag it will pick a random one to test.


6 changes: 5 additions & 1 deletion documentation/operators/src/testing/node-setup.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# Node Setup for Performance Testing Event

```admonish info
For the moment we paused Fast and Furious `perf` environment. Nym Mainnet environment will be used for future tests, please wait for further instructions.
```

To join the [Performance testing event]({{performance_testing_webpage}}) node operators need to do proceed with the following tasks:

1. **[Sign their node]({{performance_testing_webpage}}) into the testing environment**
2. **[Configure their node](#node-configuration) for the test**
3. (*Not mandatory*) [Setup metric monitoring system](templates.md) to observe node performance at any time
3. (*Not mandatory*) [Setup metric monitoring system](performance.md#monitoring) to observe node performance at any time

## Node Configuration

Expand Down
37 changes: 32 additions & 5 deletions documentation/operators/src/testing/performance.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,34 @@
# Performance Testing
# Performance Monitoring & Testing

> To configure your node for a testing event, visit [node setup page](node-setup.md).
Nym Mixnet has been running on mainnet for quite some time. There is still work to be done in order for the network to meet its full potential - mass adoption of privacy through fully distributed Mixnet.

Nym Mixnet has been running on mainnet for quite some time. There is still work to be done in order for the network to meet its full potential - mass adoption of privacy through fully distributed Mixnet.
As developers we need to be constantly improving the software. Operators have as much important role, keep their nodes up to date, monitor their performance and share their feedback with the rest of the community and core developers.

Therefore [monitoring](#monitoring) and [testing](#testing) are essential pieces of our common work. We call out all Nym operators to join the efforts!

## Monitoring

There are multiple ways to monitor performance of nodes and the machines on which they run. For the purpose of maximal privacy and decentralisation of the data - preventing Nym Mixnet from any global adversary takeover - we created these pages as a source of mutual empowerment, a place where operators can share and learn new skills to **setup metrics monitors on their own infrastructure**.

### Guides to Setup Own Metrics

A list of different scripts, templates and guides for easier navigation:

* [`nym-gateway-probe`](gateway-probe.md) - a useful tool used under the hood of [harbourmaster.nymtech.net](https://harbourmaster.nymtech.net)
* [Prometheus and Grafana](prometheus-grafana.md) self-hosted setup
* [Nym-node CPU cron service](https://gist.github.com/tommyv1987/97e939a7adf491333d686a8eaa68d4bd) - an easy bash script by Nym core developer [@tommy1987](https://gist.github.com/tommyv1987), designed to monitor a CPU usage of your node, running locally
* Nym's script [`prom_targets.py`](https://github.com/nymtech/nym/blob/develop/scripts/prom_targets.py) - a useful python program to request data from API and can be run on its own or plugged to more sophisticated flows

### Collecting Testing Metrics

For the purpose of the performance testing Nym core developers plan to run instances of Prometheus and Grafana connected to Node explorer in the house. The network overall key insights we seek from these tests are primarily internal. We're focused on pinpointing bottlenecks, capacity loads, and monitoring cpu usage on the nodes' machines.


## Testing

```admonish info
For the moment we paused Fast and Furious `perf` environment. Nym Mainnet environment will be used for future tests, please wait for further instructions.
```

Nym asks its decentralised community of operators to join a series of performance testing events in order to **increase the overall quality of the Mixnet**. The main takeaways of such event are:

Expand All @@ -21,7 +47,7 @@ Visit [Fast and Furious web page]({{performance_testing_webpage}}) and [Nym Harb

* Nym runs a paralel network environment [validator.performance.nymte.ch]({{performance_validator}}) with a chain ID `perf`
* Operators of Nym Nodes join by following easy steps on [performance testing web page]({{performance_testing_webpage}}), including simplified node authentication signature (while keep running their nodes on the mainnet)
* Once signed in, operators will be asked to swap their binary for the modified version with metrics endpoint to be able to connect their own [monitoring system](templates.md)
* Once signed in, operators will be asked to swap their binary for the modified version with metrics endpoint to be able to connect their own [monitoring system](#monitoring)
* Core node data will be fed to a unique mixnet contract for the `perf` side chain
* Nym starts a new API and start packet transition in high load through these nodes in both settings
* Nym tracks packet flow using Prometheus and Grafana
Expand All @@ -31,4 +57,5 @@ Visit [Fast and Furious web page]({{performance_testing_webpage}}) and [Nym Harb
## More Information

* What happens after the test or what operators get for participating is shared up to date on the [performance testing web page]({{performance_testing_webpage}})
* Visit our guides to [setup metrics template](templates.md) and learn how to operate them in self-custodial way


2 changes: 2 additions & 0 deletions documentation/operators/src/troubleshooting/vps-isp.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ Begin with the steps listed in [*Connectivity Test and Configuration*](../nodes/
2. Checkout your VPS dashboard and make sure your IPv6-public enabled.
3. If you are able to add IPv6 address `/64` range, do it.

**Update:** Nym community started an ISP table called [*Where to host your nym node?*](../legal/isp-list.md), check it out and add your findings!

![](../images/ipv6_64.png)

4. Search or ask your ISP for additional documentation related to IPv6 routing and ask them to provide you with `IPv6 IP address` and `IPv6 IP gateway address`
Expand Down
59 changes: 59 additions & 0 deletions documentation/scripts/csv2md.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/python3

"""CLI to display .csv files as markdown"""

import argparse
import pandas as pd
import sys
import csv

def create_table(args):
"""Imports csv and creates a table"""
file = args.file
csv = pd.read_csv(file)
if args.sort != None:
csv = csv.sort_values(csv.columns[args.sort])
if args.table:
table = csv.to_markdown(tablefmt="grid", index=args.index)
else:
table = csv.to_markdown(index=args.index)
return table

def display_file(args):
"""Display csv file as a table"""
table = create_table(args)
print(table)

def panic(msg):
"""Error message print"""
print(f"error: {msg}", file=sys.stderr)
sys.exit(-1)

def parser_main():
"""Main function initializing ArgumentParser, storing arguments and executing commands."""
# Top level parser
parser = argparse.ArgumentParser(
prog='CSV2MD',
description='''Displays .csv files in markdown''',
epilog='''Code is power!'''
)

# Parser arguments
parser.add_argument("-V","--version", action="version", version='%(prog)s 1.1.0')
parser.add_argument("file", help="path/to/file.csv")
parser.add_argument("-t","--table", default=False, action="store_true", help="output with a tabulate option for terminal reading - does not render in mdbook")
parser.add_argument("-i","--index", default=False, action="store_true", help="output with an index column")
parser.add_argument("-s","--sort", type=int, help="supply with column index to sort your output accordingly (ascending way)")

parser.set_defaults(func=display_file)
args = parser.parse_args()

try:
args.func(args)
except AttributeError as e:
msg = f"{e}.\nPlease run with --help or read the error message in case your .csv file is corrupted."
panic(msg)


if __name__ == '__main__':
parser_main()

0 comments on commit 0eb6eb8

Please sign in to comment.