Standalone Egress NAT

Signed-off-by: Patryk Strusiewicz-Surmacki <[email protected]> Update v2/cmd/coil-egress-controller/sub/root.go Update v2/runners/coild_server.go Update v2/pkg/nodenet/pod.go Update v2/pkg/cnirpc/cni.proto Co-authored-by: Tomoya Terashima <[email protected]>
cybozu-go · Sep 24, 2024 · df10c75 · df10c75
1 parent ca263f3
commit df10c75
Show file tree

Hide file tree

Showing 55 changed files with 1,177 additions and 365 deletions.
diff --git a/docs/cmd-coil-egress-controller.md b/docs/cmd-coil-egress-controller.md
@@ -0,0 +1,27 @@
+coil-egress-controller
+===============
+
+`coil-egress-controller` is a Kubernetes controller for Coil custom resources related to on-demand NAT egress.
+It is intended to be run as a Pod in `kube-system` namespace.
+
+
+## Egress
+
+`coil-egress-controller` creates **Deployment** and **Service** for each Egress.
+
+It also creates `coil-egress` **ServiceAccount** in the namespace of Egress,
+and binds it to the **ClusterRoles** for `coil-egress`.
+
+## Command-line flags
+
+```
+Flags:
+      --cert-dir string        directory to locate TLS certs for webhook (default "/certs")
+      --egress-port int32      UDP port number used by coil-egress (default 5555)
+      --gc-interval duration   garbage collection interval (default 1h0m0s)
+      --health-addr string     bind address of health/readiness probes (default ":9387")
+  -h, --help                   help for coil-egress-controller
+      --metrics-addr string    bind address of metrics endpoint (default ":9386")
+  -v, --version                version for coil-egress-controller
+      --webhook-addr string    bind address of admission webhook (default ":9443")
+```
diff --git a/docs/cmd-coil-controller.md → docs/cmd-coil-ipam-controller.md b/docs/cmd-coil-controller.md → docs/cmd-coil-ipam-controller.md
@@ -1,41 +1,33 @@
-coil-controller
+coil-ipam-controller
 ===============
 
-`coil-controller` is a Kubernetes controller for Coil custom resources.
+`coil-ipam-controller` is a Kubernetes controller for Coil IPAM related custom resources.
 It is intended to be run as a Pod in `kube-system` namespace.
 
 ## AddressPool and AddressBlock
 
-`coil-controller` has an in-memory database of address pools and
+`coil-ipam-controller` has an in-memory database of address pools and
 address blocks to allocate address blocks quickly.
 
 ## BlockRequest
 
-`coil-controller` watches newly created block requests and carve out
+`coil-ipam-controller` watches newly created block requests and carve out
 address blocks from the requested pool.
 
-## Egress
-
-`coil-controller` creates **Deployment** and **Service** for each Egress.
-
-It also creates `coil-egress` **ServiceAccount** in the namespace of Egress,
-and binds it to the **ClusterRoles** for `coil-egress`.
-
 ## Garbage collection
 
-`coil-controller` periodically checks orphaned address blocks and deletes them.
+`coil-ipam-controller` periodically checks orphaned address blocks and deletes them.
 
 ## Command-line flags
 
 ```
 Flags:
       --cert-dir string        directory to locate TLS certs for webhook (default "/certs")
-      --egress-port int32      UDP port number used by coil-egress (default 5555)
       --gc-interval duration   garbage collection interval (default 1h0m0s)
       --health-addr string     bind address of health/readiness probes (default ":9387")
-  -h, --help                   help for coil-controller
+  -h, --help                   help for coil-ipam-controller
       --metrics-addr string    bind address of metrics endpoint (default ":9386")
-  -v, --version                version for coil-controller
+  -v, --version                version for coil-ipam-controller
       --webhook-addr string    bind address of admission webhook (default ":9443")
 ```
 

diff --git a/docs/cni-grpc.md b/docs/cni-grpc.md
@@ -7,6 +7,7 @@
     - [AddResponse](#pkg-cnirpc-AddResponse)
     - [CNIArgs](#pkg-cnirpc-CNIArgs)
     - [CNIArgs.ArgsEntry](#pkg-cnirpc-CNIArgs-ArgsEntry)
+    - [CNIArgs.InterfacesEntry](#pkg-cnirpc-CNIArgs-InterfacesEntry)
     - [CNIError](#pkg-cnirpc-CNIError)
 
     - [ErrorCode](#pkg-cnirpc-ErrorCode)
@@ -57,6 +58,8 @@ https://pkg.go.dev/github.com/containernetworking/[email protected]/pkg/skel?tab=doc#Cm
 | args | [CNIArgs.ArgsEntry](#pkg-cnirpc-CNIArgs-ArgsEntry) | repeated | Key-Value pairs parsed from CNI_ARGS |
 | path | [string](#string) |  |  |
 | stdin_data | [bytes](#bytes) |  |  |
+| ips | [string](#string) | repeated |  |
+| interfaces | [CNIArgs.InterfacesEntry](#pkg-cnirpc-CNIArgs-InterfacesEntry) | repeated |  |
 
 
 
@@ -79,6 +82,22 @@ https://pkg.go.dev/github.com/containernetworking/[email protected]/pkg/skel?tab=doc#Cm
 
 
 
+<a name="pkg-cnirpc-CNIArgs-InterfacesEntry"></a>
+
+### CNIArgs.InterfacesEntry
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| key | [string](#string) |  |  |
+| value | [bool](#bool) |  |  |
+
+
+
+
+
+
 <a name="pkg-cnirpc-CNIError"></a>
 
 ### CNIError

diff --git a/docs/design.md b/docs/design.md
@@ -73,7 +73,8 @@ Now that we have learned how to do these things, and want to add rich features s
 
 Coil v2 will consist of the following programs:
 
-- `coil-controller`: Kubernetes controller managing custom resources.
+- `coil-ipam-controller`: Kubernetes controller managing IPAM related custom resources.
+- `coil-egress-controller`: Kubernetes controller managing on-demand NAT egress related custom resources.
 - `coild`: Daemon program running on nodes.
 - `coil`: CNI interface that delegates requests from `kubelet` to `coild`.
 - `coil-egress`: Administration program running in Egress pods.
@@ -102,11 +103,11 @@ To make things simple, the default pool is the pool whose name is `default`.
 To reduce the number of advertised routes, addresses in an address pool are divided into fixed-size blocks.
 These blocks are called _address blocks_, and assigned to nodes.  Since all IP addresses in an address block are routed to the same node, only one route per address block need to be advertised.
 
-For example, if an address pool defines that the size of an address block is 2<sup>5</sup>, `coil-controller` will carve an address block for IPv4 with `/27` subnet mask out of the pool, and assigns it to a node.
+For example, if an address pool defines that the size of an address block is 2<sup>5</sup>, `coil-ipam-controller` will carve an address block for IPv4 with `/27` subnet mask out of the pool, and assigns it to a node.
 
 
 In general, avoiding immediate reuse of IP addresses is better not to confuse other software or components.
-To avoid such immediate reuse, `coil-controller` remembers the last used address, and it assigns the address from the next address.
+To avoid such immediate reuse, `coil-ipam-controller` remembers the last used address, and it assigns the address from the next address.
 
 The same problem may occur when we use address blocks of the size `/32`.
 In this case, there is a high chance of reusing the same address immediately.
@@ -333,7 +334,7 @@ Therefore, when the deletion of the owning `AddressPool` is directed, all `Addre
 That said, an `AddressBlock` should not be deleted until there are no more Pods with an address in the block.
 For this purpose, Coil adds a finalizer to each `AddressBlock`.  **`coild` checks the usage of addresses in the block**, and once there are no more Pods using the addresses, it removes the finalizer to delete the `AddressBlock`.
 
-`AddressBlock` should also be deleted when `Node` that acquired the block is deleted.  Since `coild` running as a DaemonSet pod cannot do this, **`coil-controller` watches Node deletions and removes `AddressBlocks`**.  `coil-controller` periodically checks dangling `AddressBlocks` and removes them.
+`AddressBlock` should also be deleted when `Node` that acquired the block is deleted.  Since `coild` running as a DaemonSet pod cannot do this, **`coil-ipam-controller` watches Node deletions and removes `AddressBlocks`**.  `coil-ipam-controller` periodically checks dangling `AddressBlocks` and removes them.
 
 `coild` also deletes `AddressBlock` when it frees the last IP address used in the block.  At startup, `coild` also checks each `AddressBlock` for the Node, and if no Pod is using the addresses in the block, it deletes the `AddressBlock`.
 
@@ -342,7 +343,7 @@ Note that Coil does not include `Node` in the list of owner references of an `Ad
 ### AddressPool
 
 Similar to an `AddressBlock` and its addresses, an `AddressPool` should not be deleted until there are no more `AddressBlock`s derived from the pool.
-For this purpose, Coil adds a finalizer to each `AddressPool`.  `coil-controller` checks the usage of blocks in the pool.
+For this purpose, Coil adds a finalizer to each `AddressPool`.  `coil-ipam-controller` checks the usage of blocks in the pool.
 
 Note that `blockOwnerDeletion: true` in `AddressBlock`'s `ownerReferences` does not always block the deletion of the owning `AddressPool`.
 This directive has effect only when foreground cascading deletion is adopted.
@@ -397,7 +398,7 @@ skinparam rectangle {
 together {
 actor User
 package Deployment {
-    [coil-controller] as controller
+    [coil-ipam-controller] as controller
 }
 database kube-apiserver as apiserver #lightblue {
     [AddressPool] as pool1

diff --git a/docs/setup.md b/docs/setup.md
@@ -17,6 +17,8 @@ You can tweak optional parameters by editing [`kustomization.yaml`](../v2/kustom
   - [IPv4/v6 dual stack pool](#ipv4v6-dual-stack-pool)
 - [(Option) Configure BIRD](#option-configure-bird)
 - [Note on CRI runtime compatibility](#note-on-cri-runtime-compatibility)
+- [Standalone egress](#standalone-egress)
+  - [Configuration](#configuration)
 
 ## Install `kustomize`
 
@@ -37,7 +39,8 @@ This should generate the following PEM files:
 
 ```console
 $ ls config/default/*.pem
-config/default/cert.pem  config/default/key.pem
+config/default/cert.pem         config/default/egress-key.pem  config/default/ipam-key.pem
+config/default/egress-cert.pem  config/default/ipam-cert.pem   config/default/key.pem
 ```
 
 ## Edit `kustomization.yaml`
@@ -57,7 +60,7 @@ $ vi kustomization.yaml
 (actually, the example is a network configuration list).
 
 You may edit the file to, say, add Cilium for network policies or to tune MTU.
-Note that `coil` must be the first in the plugin list.
+Note that `coil` must be the first in the plugin list if IPAM is enabled.
 
 ```console
 vi netconf.json
@@ -76,7 +79,9 @@ The following example adds `tuning` and `bandwidth` plugins.
   "plugins": [
     {
       "type": "coil",
-      "socket": "/run/coild.sock"
+      "socket": "/run/coild.sock",
+      "ipam": true,
+      "egress": true
     },
     {
       "type": "tuning",
@@ -231,3 +236,45 @@ host directory.
 
 [netconf]: https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md#network-configuration
 [BIRD]: https://bird.network.cz/
+
+## Standalone egress
+
+Coil can be run as standalone egress NAT controller, using CNI chaining with another CNI providing base connectivity. This chapter will guide you on how to achieve this.
+
+### Configuration
+
+To deploy Coil with only egress feature enabled the following changes are required in the configuration files:
+
+1. Comment all IPAM related pieces in the following `kustomization.yaml` files:
+    - `v2/config/crd/kustomization.yaml`
+    - `v2/config/default/kustomization.yaml`
+    - `v2/config/pod/kustomization.yaml`
+    - `v2/config/rbac/kustomization.yaml`
+
+2. Comment unnecessary resources in `config/crd/patches/remove_status.yaml`.
+3. Add following arguments to the `coild` contianer executable in `config/pod/coild.yaml`
+    ```yaml
+    containers:
+      - name: coild
+        image: coil:dev
+        command: ["coild"]
+        args:
+          - --zap-stacktrace-level=panic
+          - --enable-ipam=false
+          - --enable-egress=true
+          - --pod-table-id=0 # 255 if IPv6 is being used
+          - --protocol-id=2
+    ```
+4. Set coil capabilites in `v2/netconf.json` to:
+    ```json
+    {
+      "type": "coil",
+      "socket": "/run/coild.sock",
+      "capabilities": {
+        "ipam": false,
+        "egress": true
+      }
+    },
+    ```
+5. Add configuration of your chosen CNI to `v2/netconf.json` before `coil` related configuration.
+6. Deploy `coil` to existing cluster as described in [Compile and apply the manifest](#compile-and-apply-the-manifest).
diff --git a/v2/Makefile b/v2/Makefile
@@ -15,7 +15,8 @@ CONTROLLER_GEN := $(shell pwd)/bin/controller-gen
 SETUP_ENVTEST := $(shell pwd)/bin/setup-envtest
 YQ := $(shell pwd)/bin/yq
 CRD_OPTIONS = "crd:crdVersions=v1"
-ROLES = config/rbac/coil-controller_role.yaml \
+ROLES = config/rbac/coil-ipam-controller_role.yaml \
+	config/rbac/coil-egress-controller_role.yaml \
 	config/rbac/coild_role.yaml \
 	config/rbac/coil-router_role.yaml \
 	config/rbac/coil-egress_role.yaml
@@ -89,25 +90,32 @@ $(YQ):
 	rm -f yq.tar.gz
 	chmod +x $@
 
-COIL_CONTROLLER_ROLE_DEPENDS = controllers/addresspool_controller.go \
+COIL_IPAM_CONTROLLER_ROLE_DEPENDS = controllers/addresspool_controller.go \
 	controllers/blockrequest_controller.go \
-	controllers/egress_controller.go \
-	controllers/clusterrolebinding_controller.go \
 	pkg/ipam/pool.go \
 	runners/garbage_collector.go
 
-config/rbac/coil-controller_role.yaml: $(COIL_CONTROLLER_ROLE_DEPENDS)
+config/rbac/coil-ipam-controller_role.yaml: $(COIL_IPAM_CONTROLLER_ROLE_DEPENDS)
 	-rm -rf work
 	mkdir work
 	sed '0,/^package/s/.*/package work/' controllers/addresspool_controller.go > work/addresspool_controller.go
 	sed '0,/^package/s/.*/package work/' controllers/blockrequest_controller.go > work/blockrequest_controller.go
-	sed '0,/^package/s/.*/package work/' controllers/egress_controller.go > work/egress_controller.go
-	sed '0,/^package/s/.*/package work/' controllers/clusterrolebinding_controller.go > work/clusterrolebinding_controller.go
 	sed '0,/^package/s/.*/package work/' pkg/ipam/pool.go > work/pool.go
 	sed '0,/^package/s/.*/package work/' runners/garbage_collector.go > work/garbage_collector.go
-	$(CONTROLLER_GEN) rbac:roleName=coil-controller paths=./work output:stdout > $@
+	$(CONTROLLER_GEN) rbac:roleName=coil-ipam-controller paths=./work output:stdout > $@
 	rm -rf work
 
+COIL_EGRESS_CONTROLLER_ROLE_DEPENDS = controllers/egress_controller.go \
+	controllers/clusterrolebinding_controller.go
+
+config/rbac/coil-egress-controller_role.yaml: $(COIL_EGRESS_CONTROLLER_ROLE_DEPENDS)
+	-rm -rf work
+	mkdir work
+	sed '0,/^package/s/.*/package work/' controllers/egress_controller.go > work/egress_controller.go
+	sed '0,/^package/s/.*/package work/' controllers/clusterrolebinding_controller.go > work/clusterrolebinding_controller.go
+	$(CONTROLLER_GEN) rbac:roleName=coil-egress-controller paths=./work output:stdout > $@
+	# rm -rf work
+
 COILD_DEPENDS = controllers/blockrequest_watcher.go \
 	pkg/ipam/node.go \
 	runners/coild_server.go
@@ -142,10 +150,16 @@ config/rbac/coil-egress_role.yaml: controllers/pod_watcher.go
 
 # TLS certificates for webhook
 .PHONY: certs
-certs: config/default/cert.pem config/default/key.pem config/default/webhook_manifests_patch.yaml
+certs: config/default/cert.pem config/default/key.pem config/default/ipam-cert.pem config/default/ipam-key.pem config/default/egress-cert.pem config/default/egress-key.pem config/default/webhook_manifests_patch.yaml
 
 config/default/cert.pem config/default/key.pem:
-	go run ./cmd/gencert -outdir=$(PWD)/config/default
+	go run ./cmd/gencert -outdir=$(PWD)/config/default -cn=coilv2-webhook-service -host=coilv2-webhook-service.kube-system.svc -certname=cert.pem -keyname=key.pem
+
+config/default/ipam-cert.pem config/default/ipam-key.pem:
+	go run ./cmd/gencert -outdir=$(PWD)/config/default -cn=coilv2-ipam-webhook-service -host=coilv2-ipam-webhook-service.kube-system.svc -certname=ipam-cert.pem -keyname=ipam-key.pem -ca=cert.pem -cakey=key.pem
+
+config/default/egress-cert.pem config/default/egress-key.pem:
+	go run ./cmd/gencert -outdir=$(PWD)/config/default -cn=coilv2-egress-webhook-service -host=coilv2-egress-webhook-service.kube-system.svc -certname=egress-cert.pem -keyname=egress-key.pem -ca=cert.pem -cakey=key.pem
 
 config/default/webhook_manifests_patch.yaml: config/default/cert.pem config/default/webhook_manifests_patch.yaml.tmpl
 	sed "s/%CACERT%/$$(base64 -w0 < $<)/g" $@.tmpl > $@
@@ -171,7 +185,8 @@ pkg/cnirpc/cni_grpc.pb.go: pkg/cnirpc/cni.proto
 .PHONY: build
 build:
 	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil -ldflags="-s -w" cmd/coil/*.go
-	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-controller -ldflags="-s -w" cmd/coil-controller/*.go
+	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-ipam-controller -ldflags="-s -w" cmd/coil-ipam-controller/*.go
+	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-egress-controller -ldflags="-s -w" cmd/coil-egress-controller/*.go
 	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-egress -ldflags="-s -w" cmd/coil-egress/*.go
 	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-installer -ldflags="-s -w" cmd/coil-installer/*.go
 	GOARCH=$(GOARCH) CGO_ENABLED=0 go build -o work/coil-router -ldflags="-s -w" cmd/coil-router/*.go

diff --git a/v2/cmd/coil-controller/main.go b/v2/cmd/coil-controller/main.go
diff --git a/v2/cmd/coil-egress-controller/main.go b/v2/cmd/coil-egress-controller/main.go
@@ -0,0 +1,7 @@
+package main
+
+import "github.com/cybozu-go/coil/v2/cmd/coil-egress-controller/sub"
+
+func main() {
+	sub.Execute()
+}