Increasing security posture of HTC Grid by enforcing and fixing relevant encryption, authentication and RBAC issues #78

fgogolli · 2023-12-06T00:45:33Z

Description

This pull request fixes all the relevant security issues in the current code base, as detected by cfn_lint, trivy, checkov and ScoutSuite.

Terraform State:

Encrypt and secure init_grid state and Lambda buckets.
Limit the scope of KMS Key policy for State Buckets.
Remove AccessControls and use BucketPolicy to keep the bucket private.
Configure all Makefiles to use encrypted S3 Buckets for TF State, non-root Dockerfiles, fix HTCGRID_ECR_REPO, name CloudFormation stack outputs, and support updating existing init_grid stack.
Improve init_grid Makefile to handle initial and deletion cases better.
Add support for cleaning up S3 object versions and standardize bucket variable naming.

HTC Grid Containers:

Configure all Dockerfiles to run non-root containers and fix builds.
Configure all HTC K8S resources to run with runAsNonRoot, default seccompProfile, and disabled allowPrivilegeEscalation.
Rename components, add readOnlyFileSystem and seccomp profile to HTC Agent, fix and cleanup code.
Remove file system write dependencies for the agent.
Harden K8S manifests and enforce further chekov rules.
Configure Grafana Ingress to drop invalid HTTP Header fields.

HTC Grid Control Plane:

Configure CMK KMS Key encryption for VPC Flow Logs, ECR Repositories, SQS, DynamoDB, S3, EKS Cluster, EKS MNG EBS Volumes, and all CloudWatch Logs.
Add encrypted CloudWatch Logging for API Gateway.
Create S3 via TF Module, add encryption support for S3 Data Plane in the agent, fix AWS partition, and DNS Suffix usage.
Simplify code and move all lambdas and auth to the control_plane.
Configure and consolidate least-privilege permissions on KMS, Lambda, and Agent IAM policies.
Add KMS Decrypt and GenerateDataKey permissions to Lambda and Agent permissions.
Move installation of jq onto lambda images and fix the bootstrap script.
Convert EC Redis to a single replica cluster mode and add encryption.
Add AUTH for ElastiCache Redis Cluster.
Enable XRay tracing for Lambda functions and adjust Redis config.
Add an explicit ASG Service Linked Role declaration to enable KMS support for ASG EBS Volumes.
Handle cases where AWSServiceRoleForAutoScaling already exists.
Add S3 and SQS Resource Policies to enforce HTTPS and create separate CMK KMS Keys for DLQs per each SQS Queue.
Configure the DLQs to be used with the respective SQS Queues and fix naming/references.
Add security group and ACL controls where possible.
Configure securityContext for OpenAPI.

General:

Add GitHub workflows for cfn_lint, trivy, and checkov.
Standardize, fix, and simplify tests.
Standardize the naming of TF resources.
Fix docs and random_password to align with pipelines.
Add auto deploy & destroy stages for images.

Cloud9:

Fix Cloud9 deployment script to target correct instances.
Fix Cloud9 bootstrap race condition and adjust to WS.
Force a reinstall at bootstrap time to fix virtualenv issues.
Add support for specifying a Git repo/branch for HTCGridSource.
Remove Admin role from KMS Admins as it doesn't exist in WS.

Checklist

Added tests that cover your change (if possible)
Added/modified documentation as required (such as the README.md, or the docs directory)
Manually tested
Made sure the title of the PR is a good description that can go into the release notes
(Core team) Added labels for change area (e.g. area/controlplane) and kind (e.g. kind/improvement)

BONUS POINTS checklist:

Backfilled missing tests for code in same general area
Refactored something and made the world a better place

…compProfile and disabled allowPrivilegeEscalation

…-root Dockerfiles, fixed HTCGRID_ECR_REPO, named CloudFormation stack outputs and support for updating existing init_grid stack

…ermissions

…rmissions

… CMK KMS Keys for the DLQs per each SQS Queue

…naming/references

…Logs

…olicies

…agent, fix AWS partition and DNS Suffix usage

… variable naming

deployment/grid/terraform/control_plane/openapi_public.tf

deployment/grid/terraform/control_plane/lambda_ttl_checker.tf

deployment/grid/terraform/control_plane/openapi_private.tf

deployment/grid/terraform/compute_plane/aws_iam.tf

…alidate issues

deployment/grid/scripts/python/create_ddb_item.py

examples/client/python/portfolio_pricing_client.py

source/client/python/api-v0.1/api/state_table_dynamodb.py

source/control_plane/python/lambda/submit_tasks/submit_tasks.py

deployment/grid/terraform/control_plane/aws_iam.tf

+  policy_arn = "arn:${local.partition}:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"
+}
+
+resource "aws_api_gateway_account" "apigateway_account" {


examples/workloads/python/quant_lib/portfolio_pricing_engine.py


        for i, r in enumerate(results):
            # print(i, (abs(all_expected_results[i] - results[i])), all_expected_results[i], results[i])
-            assert (abs(all_expected_results[i] - results[i]) < 0.000001)
+            assert abs(all_expected_results[i] - results[i]) < 0.000001


source/compute_plane/python/agent/agent.py

-        stdout=f_stdout,
-        stderr=f_stderr,
-        shell=False)
+    proc = subprocess.Popen(command, stdout=f_stdout, stderr=f_stderr, shell=False)


source/compute_plane/python/agent/agent.py

-            logging.info("Could not acquire a task from the queue, backing off for {}".
-                         format(timeout)
-                         )
+            timeout = random.uniform(


source/compute_plane/python/lambda/drainer/handler.py


 logger = logging.getLogger(__name__)
 logger.setLevel(logging.DEBUG)

-KUBE_FILEPATH = '/tmp/kubeconfig'
-region = os.environ['AWS_REGION']
+KUBE_FILEPATH = "/tmp/kubeconfig"


source/control_plane/python/lambda/cancel_tasks/cancel_tasks.py


    else:
        errlog.log("Unimplemented path, exiting")
-        assert(False)
+        assert False


source/control_plane/python/lambda/get_results/get_results.py



 def get_time_now_ms():
    return int(round(time.time() * 1000))


 def get_tasks_statuses_in_session(session_id):
-
-    assert(session_id is not None)
+    assert session_id is not None


source/control_plane/python/lambda/get_results/get_results.py


    else:
        errlog.log("Uniplemented path, exiting")
-        assert(False)
+        assert False


deployment/init_grid/cloudformation/Makefile

clementrey-dev

Hello Flamur,

Thank you for your work.

I been running the happy path section (from my laptop) and I got error from multiple sections:

REGION not set after different make command
Mallformed policy during the deployment
bucket not fully emptied during the cleaning process

Can you please take a look at this ?

deployment/grid/terraform/compute_plane/aws_iam.tf

deployment/init_grid/cloudformation/Makefile

kirillsc · 2024-02-06T09:33:41Z

lgtm

kirillsc

Looks good to me

deployment/init_grid/cloudformation/grid_state.yaml

+  ImageTFStateBucket:
    Type: 'AWS::S3::Bucket'
+    DeletionPolicy: Delete
    Properties:
      BucketName: !Sub
-        - '${BucketTag}-tfstate-htc-grid-${RANDOM}'
+        - '${BucketTag}-htc-grid-image-tfstate-${RANDOM}'
        - RANDOM: !Select [0, !Split ['-', !Select [2, !Split ['/', !Ref 'AWS::StackId' ]]]]
-    DeletionPolicy: Delete
-  LambdaUnitHtcGrid:
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        BlockPublicPolicy: true
+        IgnorePublicAcls: true
+        RestrictPublicBuckets: true
+      VersioningConfiguration:
+        Status: Enabled
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+        - BucketKeyEnabled: true
+          ServerSideEncryptionByDefault:
+            SSEAlgorithm: 'aws:kms'
+            KMSMasterKeyID: !Sub 'arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:${HTCStateS3KeyAlias}'


deployment/init_grid/cloudformation/grid_state.yaml

+  LambdaLayerBucket:
    Type: 'AWS::S3::Bucket'
+    DeletionPolicy: Delete
    Properties:
      BucketName: !Sub
-        - '${BucketTag}-lambda-unit-htc-grid-${RANDOM}'
+        - '${BucketTag}-htc-grid-lambda-layer-${RANDOM}'
        - RANDOM: !Select [0, !Split ['-', !Select [2, !Split ['/', !Ref 'AWS::StackId' ]]]]
-    DeletionPolicy: Delete
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        BlockPublicPolicy: true
+        IgnorePublicAcls: true
+        RestrictPublicBuckets: true
+      VersioningConfiguration:
+        Status: Enabled
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+        - BucketKeyEnabled: true
+          ServerSideEncryptionByDefault:
+            SSEAlgorithm: 'aws:kms'
+            KMSMasterKeyID: !Sub 'arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:${HTCStateS3KeyAlias}'


deployment/init_grid/cloudformation/grid_state.yaml

+  GridTFStateBucket:
    Type: 'AWS::S3::Bucket'
+    DeletionPolicy: Delete
    Properties:
      BucketName: !Sub
-        - '${BucketTag}-image-tfstate-htc-grid-${RANDOM}'
+        - '${BucketTag}-htc-grid-tfstate-${RANDOM}'
        - RANDOM: !Select [0, !Split ['-', !Select [2, !Split ['/', !Ref 'AWS::StackId' ]]]]
-    DeletionPolicy: Delete
-  TfstateHtcGrid:
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        BlockPublicPolicy: true
+        IgnorePublicAcls: true
+        RestrictPublicBuckets: true
+      VersioningConfiguration:
+        Status: Enabled
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+        - BucketKeyEnabled: true
+          ServerSideEncryptionByDefault:
+            SSEAlgorithm: 'aws:kms'
+            KMSMasterKeyID: !Sub 'arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:${HTCStateS3KeyAlias}'


examples/submissions/k8s_jobs/Dockerfile.Submitter

@@ -1,20 +1,22 @@
-FROM python:3.7.7-slim-buster
+ARG HTCGRID_ECR_REPO


examples/workloads/c++/mock_computation/Dockerfile.Build

-RUN yum install -d1 -y  make gcc-c++ zip
-RUN mkdir -p /app
-WORKDIR /app
+ARG HTCGRID_ECR_REPO


source/compute_plane/python/agent/Dockerfile.Lambda

@@ -1,21 +1,26 @@
-FROM python:3.7.7-slim-buster
+ARG HTCGRID_ECR_REPO


source/compute_plane/python/agent/Dockerfile.Local

@@ -1,41 +1,54 @@
-FROM python:3.7.7-slim-buster
+#Builder Container


source/compute_plane/shell/attach-layer/Dockerfile

-ENV LAYER_NAME lambda
-ENV LAYER_VERSION 1
-ENV LAYER_ROOT .
+ARG HTCGRID_ECR_REPO


deployment/grid/terraform/control_plane/openapi_public.tf

+resource "aws_api_gateway_method_settings" "htc_public_api_method_settings" {
+  #checkov:skip=CKV_AWS_308: API Gateway method setting caching encryption wouldn't work for this API
+  #checkov:skip=CKV_AWS_225: API Gateway method setting caching wouldn't work for this API
+
+  rest_api_id = aws_api_gateway_rest_api.htc_public_api.id
+  stage_name  = aws_api_gateway_stage.htc_public_api_stage.stage_name
+
+  method_path = "*/*"
+
+  settings {
+    metrics_enabled = true
+    logging_level   = "ERROR"
+  }
+}


deployment/grid/terraform/control_plane/openapi_private.tf

+resource "aws_api_gateway_method_settings" "htc_private_api_method_settings" {
+  #checkov:skip=CKV_AWS_308: API Gateway method setting caching encryption wouldn't work for this API
+  #checkov:skip=CKV_AWS_225: API Gateway method setting caching wouldn't work for this API
+
+  rest_api_id = aws_api_gateway_rest_api.htc_private_api.id
+  stage_name  = aws_api_gateway_stage.htc_private_api_stage.stage_name
+
+  method_path = "*/*"
+
+  settings {
+    metrics_enabled = true
+    logging_level   = "ERROR"
+  }
+}


fgogolli added 30 commits October 31, 2023 11:35

Encrypt and secure init_grid state and lambda buckets

07947a0

Configure all Dockerfiles to run non-root containers and fix builds

d5a4075

Configure all HTC K8S resources to run with runAsNonRoot, default sec…

4295e2a

…compProfile and disabled allowPrivilegeEscalation

Configure Grafana Ingress to drop invalid HTTP Header fields

0aca22d

Configure all Makefiles to use encrypted S3 Buckets for TF State, non…

e9a6b5f

…-root Dockerfiles, fixed HTCGRID_ECR_REPO, named CloudFormation stack outputs and support for updating existing init_grid stack

Encrypt EKS MNG EBS Volumes with CMK KMS Key

f59ed8f

Fix docs and random_password to align with pipelines

5624eb5

Add KMS Decrypt and GenerateDataKey permissions to Lambda and Agent p…

5f040a8

…ermissions

Move installation of jq onto lambda images and fix bootstrap script

6ca4be0

Add CMK KMS Encryption to SQS, DynamoDB and S3

c0e8157

Add encryption to CloudWatch logs and standardise Lambda roles and pe…

71c1c74

…rmissions

Enable XRay tracing for Lambda functions and adjust Redis config

8b71b93

Add encryption to EC Redis and convert to single replica cluser mode

9b1f196

Add CMK KMS encryption to EKS Cluster and EKS CloudWatch Logs

5762e28

Add S3 and SQS Resource Policies to enforce HTTPS and create seperate…

d7c703d

… CMK KMS Keys for the DLQs per each SQS Queue

Configure the DLQs to be used with the respective SQS Queues and fix …

a0e2162

…naming/references

Standardise the naming of TF resources

676dd32

Add encrypted CloudWatch Logging for API Gateway

f5fe27e

Add CMK KMS Encrypted CloudWatch LogGroup configued for the VPC Flow …

0700751

…Logs

Add CMK KMS Encryption to ECR Repositories

7437a5d

Configure least-priviledge permissions on KMS, Lambda and Agent IAM p…

179eac8

…olicies

Configure securityContext for OpenAPI

3f29229

Consolidate IAM Policies

dab697f

Add AUTH for ElastiCache Redis Cluster

33984c3

Create S3 via TF Module, add encryption support for S3 Data Plane in …

15390e2

…agent, fix AWS partition and DNS Suffix usage

Enforce further chekov rules and harden K8S manifests

6301f28

Fix Cloud9 deployment script to target correct instances

15f9861

Limit the scope of KMS Key policy for the State Buckets

0c68717

Add security group and acl controls where possible

e403f53

Add support for cleaning up S3 object versions and standardise bucket…

b58b847

… variable naming

fgogolli force-pushed the v043_fixes branch from 3b1628b to 930bcef Compare December 12, 2023 11:42

github-advanced-security bot found potential problems Dec 12, 2023

View reviewed changes

deployment/grid/terraform/control_plane/openapi_public.tf Fixed Show fixed Hide fixed

fgogolli force-pushed the v043_fixes branch from 930bcef to 3d4348d Compare December 12, 2023 11:49

github-advanced-security bot found potential problems Dec 12, 2023

View reviewed changes

deployment/grid/terraform/control_plane/lambda_ttl_checker.tf Fixed Show fixed Hide fixed

fgogolli force-pushed the v043_fixes branch 2 times, most recently from d050e97 to 93a029d Compare December 12, 2023 12:00

github-advanced-security bot found potential problems Dec 12, 2023

View reviewed changes

deployment/grid/terraform/control_plane/openapi_private.tf Fixed Show fixed Hide fixed

fgogolli force-pushed the v043_fixes branch from 93a029d to ddc80ac Compare December 12, 2023 12:04

github-advanced-security bot found potential problems Dec 12, 2023

View reviewed changes

deployment/grid/terraform/compute_plane/aws_iam.tf Fixed Show fixed Hide fixed

fgogolli force-pushed the v043_fixes branch from ddc80ac to c9754f4 Compare December 12, 2023 12:06

Add types to all TF vars, remove unused vars and fix all tflint and v…

9d85378

…alidate issues

fgogolli force-pushed the v043_fixes branch from c9754f4 to 9d85378 Compare December 13, 2023 10:42

github-advanced-security bot found potential problems Dec 13, 2023

View reviewed changes

Fix flake8 formatting issues

3bba529

fgogolli force-pushed the v043_fixes branch 2 times, most recently from 395e14d to 4689e66 Compare December 13, 2023 14:18

Add trivyignore and fix tfchecks

4dc021b

fgogolli force-pushed the v043_fixes branch from 4689e66 to 4dc021b Compare December 15, 2023 13:56

clementrey-dev reviewed Jan 10, 2024

View reviewed changes

deployment/init_grid/cloudformation/Makefile Outdated Show resolved Hide resolved

clementrey-dev requested changes Jan 11, 2024

View reviewed changes

deployment/grid/terraform/compute_plane/aws_iam.tf Show resolved Hide resolved

deployment/init_grid/cloudformation/Makefile Outdated Show resolved Hide resolved

kirillsc approved these changes Feb 6, 2024

View reviewed changes

fgogolli requested a review from clementrey-dev February 6, 2024 12:59

clementrey-dev approved these changes Feb 6, 2024

View reviewed changes

Change versioned State S3 Buckets deletion to use a Python script

b4da2e8

fgogolli force-pushed the v043_fixes branch from b6462c6 to b4da2e8 Compare February 6, 2024 16:57

github-advanced-security bot found potential problems Feb 6, 2024

View reviewed changes

fgogolli merged commit 4813f07 into finos:main Feb 6, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increasing security posture of HTC Grid by enforcing and fixing relevant encryption, authentication and RBAC issues #78

Increasing security posture of HTC Grid by enforcing and fixing relevant encryption, authentication and RBAC issues #78

fgogolli commented Dec 6, 2023

clementrey-dev left a comment

kirillsc commented Feb 6, 2024

kirillsc left a comment

		@@ -1,20 +1,22 @@
		FROM python:3.7.7-slim-buster
		ARG HTCGRID_ECR_REPO

		@@ -1,21 +1,26 @@
		FROM python:3.7.7-slim-buster
		ARG HTCGRID_ECR_REPO

		@@ -1,41 +1,54 @@
		FROM python:3.7.7-slim-buster
		#Builder Container

Increasing security posture of HTC Grid by enforcing and fixing relevant encryption, authentication and RBAC issues #78

Increasing security posture of HTC Grid by enforcing and fixing relevant encryption, authentication and RBAC issues #78

Conversation

fgogolli commented Dec 6, 2023

Description

Terraform State:

HTC Grid Containers:

HTC Grid Control Plane:

General:

Cloud9:

Checklist

BONUS POINTS checklist:

clementrey-dev left a comment

Choose a reason for hiding this comment

kirillsc commented Feb 6, 2024

kirillsc left a comment

Choose a reason for hiding this comment