Skip to content

Latest commit

 

History

History
126 lines (87 loc) · 3.74 KB

PROJECT_SETUP.MD

File metadata and controls

126 lines (87 loc) · 3.74 KB

Setup

URL Component
Debezium UI Debezium UI - http://localhost:8080/
KafkaUI Kafka UI - http://localhost:9099/

We will use docker compose for a fast deployment of the demo. The shell script deploy.sh is designed to initialize the environment when triggered using sh deploy.sh.

deploy

Alternatively, you may choose to follow the steps listed below, depending on your preference.

  1. Clone the repository
git clone https://github.com/AnanthaRajuC/Streaming_ETL_pipeline_MySQL.git
cd Streaming_ETL_pipeline_MySQL

Running the application via docker compose

Validate the docker-compose file

docker-compose -f docker-compose.yaml config
  1. Pull all required docker images
docker compose -f docker-compose.yaml pull
  1. Start up the environment

The first time that you do this, the Docker images will be pulled down from the remote server. This may take a while!

docker compose -f docker-compose.yaml up
Creating network "streaming_etl_pipeline_mysql_webproxy" with driver "bridge"
Creating zookeeper ... done
Creating kafka     ... done
Creating debezium           ... done
Creating cp-schema-registry ... done
Creating kafka-connect-01   ... done
Creating ksqldb-server      ... done
Creating ksqldb-cli         ... done

Accessing Kafka Topics via Kakfka-UI

  1. Optionally, start Kafka UI, an open-source web UI for Apache Kafka Management
docker run --name=kafka-ui --network=streaming_etl_pipeline_mysql_webproxy -p 9099:8080 -e KAFKA_CLUSTERS_0_NAME=local -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 -d provectuslabs/kafka-ui:latest

URL to access Kakfka-UI console: http://localhost:9099

Kafka UI

  1. Make sure everything is up and running
docker ps

Docker Compose Up

IMPORTANT: If any components do not show "Up" under the Status column (e.g., they say "Exit") then you must rectify this before continuing. As a first solution, try re-issuing the docker-compose up -d command.

Accessing ksqlDb via ksqldb-cli

  1. Launch the KSQL CLI in another terminal window.
docker exec -it ksqldb-cli ksql http://ksqldb-server:8088

ksqlDB CLI

Tear down the stack

The shell script terminate.sh is designed to tear down the environment when triggered using sh terminate.sh.

terminate

Alternatively, you may choose to follow the steps listed below, depending on your preference.

docker stop kafka-ui

docker rm kafka-ui
docker compose -f docker-compose.yaml down
Stopping ksqldb-cli       ... done
Stopping ksqldb-server    ... done
Stopping kafka-connect-01 ... done
Stopping debezium         ... done
Stopping kafka            ... done
Stopping zookeeper        ... done
Removing ksqldb-cli         ... done
Removing ksqldb-server      ... done
Removing kafka-connect-01   ... done
Removing cp-schema-registry ... done
Removing debezium           ... done
Removing kafka              ... done
Removing zookeeper          ... done
Removing network streaming_etl_pipeline_mysql_webproxy

If you want to preserve the state of all containers, run docker-compose stop instead.