Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build the docker and push #33

Merged
merged 43 commits into from
Jan 23, 2024
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c43536c
Build the docker and push
Vasilije1990 Dec 16, 2023
dfb13a2
Build the docker and push
Vasilije1990 Dec 16, 2023
6e47c1f
Automate summary function
Vasilije1990 Dec 17, 2023
5ed04de
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
985c459
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
9293c74
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
cbde083
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
078d697
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
fa1df60
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
523f893
Make sure that secrets would work on aws
Vasilije1990 Dec 21, 2023
98383b7
Make a change to test neo4j
Vasilije1990 Dec 21, 2023
118613c
Make a change to test neo4j
Vasilije1990 Dec 21, 2023
2928f51
Add all functions for architecture to work + for regular user flows, …
Vasilije1990 Dec 27, 2023
be3f179
Add all functions for architecture to work + for regular user flows, …
Vasilije1990 Dec 27, 2023
60d5c74
Add all functions for architecture to work + for regular user flows, …
Vasilije1990 Dec 27, 2023
3632a85
Add all functions for architecture to work + for regular user flows, …
Vasilije1990 Dec 28, 2023
0eae4e9
Add all functions for architecture to work + for regular user flows, …
Vasilije1990 Dec 28, 2023
37e7a78
Test to fix deployment
Vasilije1990 Dec 28, 2023
a8d9ba2
Fix database issue
Vasilije1990 Dec 28, 2023
305bd67
Fix database issue
Vasilije1990 Dec 28, 2023
400e78b
Fix database issue
Vasilije1990 Dec 28, 2023
88de713
Fix database issue
Vasilije1990 Dec 28, 2023
9971cac
Optimize the cognitive function a bit
Vasilije1990 Dec 30, 2023
42ca604
Optimize the cognitive function a bit
Vasilije1990 Jan 4, 2024
f7d2dd4
Optimize the cognitive function a bit
Vasilije1990 Jan 4, 2024
0120421
Add specific driver closing to avoid issues with neo4
Vasilije1990 Jan 5, 2024
8aeef5c
Add specific driver closing to avoid issues with neo4
Vasilije1990 Jan 9, 2024
7afe9a7
Add fuzzy matchin for neo4j
Vasilije1990 Jan 9, 2024
0323ab6
Fix context retireval
Vasilije1990 Jan 10, 2024
50e4d7c
Fix context retireval
Vasilije1990 Jan 10, 2024
f41f091
Fix context retireval
Vasilije1990 Jan 11, 2024
fb542a3
Fix context retireval
Vasilije1990 Jan 11, 2024
7b39813
Reduce size of context, hack for now
Vasilije1990 Jan 13, 2024
9b2a639
Reduce size of context, hack for now
Vasilije1990 Jan 14, 2024
3d6d0a0
Reduce size of context, hack for now
Vasilije1990 Jan 14, 2024
db18bd8
Reduce size of context, hack for now
Vasilije1990 Jan 14, 2024
eee6049
Reduce size of context, hack for now
Vasilije1990 Jan 14, 2024
444d4b3
Reduce size of context, hack for now
Vasilije1990 Jan 17, 2024
d8f621f
Reduce size of context, hack for now
Vasilije1990 Jan 17, 2024
94ef465
Reduce size of context, hack for now
Vasilije1990 Jan 17, 2024
f9f7d4a
Reduce size of context, hack for now
Vasilije1990 Jan 19, 2024
0b1cefb
Added timestamps to be able to filter and merge the nodes based on re…
Vasilije1990 Jan 19, 2024
dab514f
Added timestamps to be able to filter and merge the nodes based on re…
Vasilije1990 Jan 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
12 changes: 8 additions & 4 deletions level_4/.env.template → .env.template
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
OPENAI_API_KEY=sk
WEAVIATE_URL =
WEAVIATE_API_KEY =
ENVIRONMENT = docker
ENV = docker
POSTGRES_USER = bla
POSTGRES_PASSWORD = bla
POSTGRES_DB = bubu
POSTGRES_HOST = localhost
POSTGRES_HOST_DOCKER = postgres
SEGMENT_KEY = Etl4WJwzOkeDPAjaOXOMgyU16hO7mV7B
COG_ARCH_DIR = cognitive_architecture
GRAPH_DB_URL =
GRAPH_DB_PW =
GRAPH_DB_USER =
GRAPH_DB_URL =
GRAPH_DB_PW =
GRAPH_DB_USER =
AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =
QDRANT_API_KEY
QDRANT_API_URL
Comment on lines +4 to +18
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 NOTE
Codebase verification is a beta feature.

The search results indicate that the ENVIRONMENT variable is still being used in several places within the codebase:

  • entrypoint.sh
  • README.md (twice, in instructions)
  • iterations/level_3/create_database.py
  • iterations/level_3/entrypoint.sh
  • iterations/level_3/database/database.py
  • iterations/level_2/entrypoint.sh

Given this information, the renaming of ENVIRONMENT to ENV in the .env.template file is not consistently applied across the entire project. The references to ENVIRONMENT in the README.md are likely instructional and may need to be updated to reflect the changes in the .env.template file. The occurrences in the scripts and Python files suggest that the code still expects an ENVIRONMENT variable to be set.

Based on these findings, the following issues have been identified:

  • The ENVIRONMENT variable is still being used in multiple places, which could lead to inconsistencies or errors if the environment variable is expected to be ENV now.
  • The README.md file contains outdated instructions regarding the ENVIRONMENT variable.
Analysis chain

Ensure that the new AWS and QDRANT API keys are handled securely and are not exposed in the codebase or version control. Verify that the renaming of ENVIRONMENT to ENV is consistently applied across the entire project.


Scripts Executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Verify that ENV is used instead of ENVIRONMENT throughout the codebase
rg --type env 'ENVIRONMENT'
# Expecting no results for ENVIRONMENT, only ENV should be present

Length of output: 55



Script:

#!/bin/bash
# Verify that ENV is used instead of ENVIRONMENT throughout the codebase
# This time, search all text files without specifying a file type
rg 'ENVIRONMENT'
# Expecting no results for ENVIRONMENT, only ENV should be present

Length of output: 495

1 change: 0 additions & 1 deletion .github/actions/image_builder/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,3 @@ runs:
image_name="${{ inputs.ecr_image_repo_name }}" docker_login="true" version="$VERSION" account="${{ inputs.aws_account_id }}" app_dir="$APP_DIR" publish="${{ inputs.should_publish }}" ./bin/dockerize
echo "Docker tag is: $VERSION"
echo $VERSION > /tmp/.DOCKER_IMAGE_VERSION

36 changes: 18 additions & 18 deletions .github/workflows/cd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ jobs:
stage: dev
aws_account_id: ${{ env.AWS_ACCOUNT_ID_DEV }}
should_publish: true
ecr_image_repo_name: promethai-dev-backend-promethai-backend-memory
dockerfile_location: level_2/.
ecr_image_repo_name: promethai-dev-backend-promethai-backend
dockerfile_location: ./
- name: Export Docker image tag
id: export-promethai-docker-tag
run: |
Expand All @@ -58,19 +58,19 @@ jobs:
outputs:
promethai_docker_tag_backend: ${{ steps.export-promethai-docker-tag.outputs.promethai_docker_tag_backend }}

# apply_tf:
# name: Trigger terraform apply workflow
# runs-on: ubuntu-latest
# needs: publish_docker_to_ecr
# steps:
# - name: TF apply workflow triggers step
# uses: actions/github-script@v6
# with:
# github-token: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
# script: |
# await github.rest.actions.createWorkflowDispatch({
# owner: 'topoteretes',
# repo: 'PromethAI-Infra',
# workflow_id: 'terraform.apply.yml',
# ref: 'main'
# })
apply_tf:
name: Trigger terraform apply workflow
runs-on: ubuntu-latest
needs: publish_docker_to_ecr
steps:
- name: TF apply workflow triggers step
uses: actions/github-script@v6
with:
github-token: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
script: |
await github.rest.actions.createWorkflowDispatch({
owner: 'topoteretes',
repo: 'PromethAI-Infra',
workflow_id: 'terraform.apply.yml',
ref: 'main'
})
100 changes: 50 additions & 50 deletions .github/workflows/cd_prd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,38 +23,38 @@ jobs:
steps:
- name: Take code from repo
uses: actions/checkout@v3
# - name: Set environment variable for stage
# id: set-env
# run: |
# if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
# echo "STAGE=prd" >> $GITHUB_ENV
# echo "::set-output name=stage::prd"
# else
# echo "STAGE=dev" >> $GITHUB_ENV
# echo "::set-output name=stage::dev"
# fi
# - name: Use output
# run: echo "The stage is ${{ steps.set-env.outputs.stage }}"
# - name: Configure AWS credentials
# uses: aws-actions/configure-aws-credentials@v1
# with:
# role-to-assume: ${{ env.AWS_ROLE_DEV_CICD }}
# aws-region: eu-west-1
# - name: Create Docker image and push to ECR
# uses: ./.github/actions/image_builder
# id: generate-promethai-docker
# with:
# stage: prd
# aws_account_id: ${{ env.AWS_ACCOUNT_ID_DEV }}
# should_publish: true
# ecr_image_repo_name: promethai-prd-backend-promethai-backend-memory
# dockerfile_location: ./level_2
# - name: Export Docker image tag
# id: export-promethai-docker-tag
# run: |
# export DOCKER_TAG=$(cat /tmp/.DOCKER_IMAGE_VERSION)
# echo "Docker tag is: $DOCKER_TAG"
# echo "promethai_docker_tag_backend=$DOCKER_TAG" >> $GITHUB_OUTPUT
- name: Set environment variable for stage
id: set-env
run: |
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
echo "STAGE=prd" >> $GITHUB_ENV
echo "::set-output name=stage::prd"
else
echo "STAGE=dev" >> $GITHUB_ENV
echo "::set-output name=stage::dev"
fi
- name: Use output
run: echo "The stage is ${{ steps.set-env.outputs.stage }}"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ env.AWS_ROLE_DEV_CICD }}
aws-region: eu-west-1
- name: Create Docker image and push to ECR
uses: ./.github/actions/image_builder
id: generate-promethai-docker
with:
stage: prd
aws_account_id: ${{ env.AWS_ACCOUNT_ID_DEV }}
should_publish: true
ecr_image_repo_name: promethai-prd-backend-promethai-backend
dockerfile_location: ./
- name: Export Docker image tag
id: export-promethai-docker-tag
run: |
export DOCKER_TAG=$(cat /tmp/.DOCKER_IMAGE_VERSION)
echo "Docker tag is: $DOCKER_TAG"
echo "promethai_docker_tag_backend=$DOCKER_TAG" >> $GITHUB_OUTPUT

# - name: Create Tag and Release
# runs-on: ubuntu-latest
Expand All @@ -78,22 +78,22 @@ jobs:
# with:
# tag_name: ${{ steps.bump_version_and_push_tag.outputs.tag }}
# release_name: Release ${{ steps.bump_version_and_push_tag.outputs.tag }}
# outputs:
# promethai_docker_tag_backend: ${{ steps.export-promethai-docker-tag.outputs.promethai_docker_tag_backend }}
outputs:
promethai_docker_tag_backend: ${{ steps.export-promethai-docker-tag.outputs.promethai_docker_tag_backend }}

# apply_tf:
# name: Trigger terraform apply workflow
# runs-on: ubuntu-latest
# needs: publish_docker_to_ecr
# steps:
# - name: TF apply workflow triggers step
# uses: actions/github-script@v6
# with:
# github-token: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
# script: |
# await github.rest.actions.createWorkflowDispatch({
# owner: 'topoteretes',
# repo: 'PromethAI-Infra',
# workflow_id: 'terraform.apply.yml',
# ref: 'main'
# })
apply_tf:
name: Trigger terraform apply workflow
runs-on: ubuntu-latest
needs: publish_docker_to_ecr
steps:
- name: TF apply workflow triggers step
uses: actions/github-script@v6
with:
github-token: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
script: |
await github.rest.actions.createWorkflowDispatch({
owner: 'topoteretes',
repo: 'PromethAI-Infra',
workflow_id: 'terraform.apply.yml',
ref: 'main'
})
Comment on lines 78 to +99
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [59-80]

The commented-out code for creating tags and releases is still present in the workflow file. If these steps have been implemented elsewhere, this commented-out code should be removed to avoid confusion and maintain cleanliness of the workflow file.

- #      - name: Create Tag and Release
- #        runs-on: ubuntu-latest
- #        uses: actions/checkout@v3
- #        needs: publish_docker_to_ecr  # ensure this job runs after Docker image is pushed
- #        steps:
- #          - name: Check out code
- #            uses: actions/checkout@v3
- #          - name: Bump version and push tag
- #            id: bump_version_and_push_tag
- #            uses: anothrNick/[email protected]
- #            env:
- #              GITHUB_TOKEN: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
- #              WITH_V: true
- #              DEFAULT_BUMP: 'minor'  # or 'minor' or 'major'
- #          - name: Create Release
- #            id: create_release
- #            uses: actions/create-release@v1
- #            env:
- #              GITHUB_TOKEN: ${{ secrets.PAT_FOR_CROSS_REPOS_CICD_TRIGGERING }}
- #            with:
- #              tag_name: ${{ steps.bump_version_and_push_tag.outputs.tag }}
- #              release_name: Release ${{ steps.bump_version_and_push_tag.outputs.tag }}

4 changes: 2 additions & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ jobs:
export SHA_SHORT="$(git rev-parse --short HEAD)"
export CUR_DATE="$(date +%Y%m%d%H%M%S)"
export VERSION="dev-$CUR_DATE-$SHA_SHORT"
image_name="backend-memory" docker_login="false" version="$VERSION" account="${{ env.AWS_ACCOUNT_ID_DEV }}" app_dir="level_2" publish="false" ./bin/dockerize
export DOCKER_TAG=$(cat level_2/tmp/.DOCKER_IMAGE_VERSION)
image_name="backend" docker_login="false" version="$VERSION" account="${{ env.AWS_ACCOUNT_ID_DEV }}" app_dir="backend" publish="false" ./bin/dockerize
export DOCKER_TAG=$(cat /tmp/.DOCKER_IMAGE_VERSION)
echo "Successfully built PromethAI backend Docker tag is: $DOCKER_TAG"
File renamed without changes.
1 change: 1 addition & 0 deletions level_4/Dockerfile → Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ RUN apt-get update -q && \
curl \
zip \
jq \
# libgl1-mesa-glx \
netcat-traditional && \
pip install poetry && \
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
Expand Down
Loading
Loading