DSA3101-2410 Group 8 OpenAYEye

Table of Contents

1. Project Overview
2. Instructions for Setting Up the Environment
3. Description of the Repository Structure
   • Jupyter Notebooks for EDA and Model Development
   • Production-Ready Python Code
4. Data Sources and Data Preparation Steps
5. Instructions for Building and Running the Docker Containers
6. API Documentation (Endpoints, Request/Response Formats)
7. API Documentation Using Swagger/OpenAPI Specification
   • Web App
8. Data Pipelining for MLOps
   • Recommendation System Pipeline
   • ROI Model Pipeline

Project overview

In today’s competitive financial sector, banks face challenges in effectively targeting customers with personalized marketing efforts. This often results in low engagement rates, inefficient use of resources, and missed opportunities to connect with customers in meaningful ways. The core challenge is the lack of personalization due to underutilization of vast customer data, which often leaves marketing campaigns generic and misaligned with individual preferences.

Our project addresses this challenge by developing an AI-driven, personalized marketing system that leverages machine learning to analyze customer data in depth. By integrating customer insights, predictive analytics, and real-time optimization, our system aims to create highly tailored marketing campaigns that enhance customer engagement, improve conversion rates, and support data-driven decision-making for marketing teams.

Instructions for setting up the environment

1. Clone the Repository

git clone https://github.com/ChinSekYi/OpenAYEye.git
cd OpenAYEye

Description of the repository structure

This is the top 2 levels of our repository structure. Unimportant files and folders are omitted below.

OpenAYEye/
├── docker-run.bash                # Bash script to run Docker
├── requirements.txt               # Python dependencies
├── Makefile                       # Automates tasks for project setup
├── test.py                        # Test script for CI/CD Github Actions
├── README.md                      # Project README
├── docker-compose.yml             # Docker Compose file for container orchestration
├── server/                        # Server-side code and ML models
│   ├── requirements.txt           # Server dependencies
│   ├── Dockerfile                 # Docker setup for server
│   ├── README.md                  # Server README
│   └── src/                       # Analytics source files
├── notebook/                      # Jupyter notebooks for EDA, model training etc
│   ├── segment/                   # Segment model notebooks
│   ├── roi-model/                 # ROI model notebooks 
│   ├── reco_system/               # Recommendation system notebooks 
│   └── src/                           # Data pipelining
│       ├── exception.py               # Custom exception handling classes
│       ├── pipeline/                  # Pipeline for prediction
│       ├── components/                # Pipeline for data ingestion and model training
│       ├── logger.py                  # Logging utility 
│       ├── utils.py                   # Utility functions
│       └── main.py                    # To run Pipeline
├── data/                          # Data processing and database code
│   ├── requirements.txt           # Data dependencies
│   ├── Dockerfile                 # Docker setup for data service
│   ├── README.md                  # Data README
│   └── src/                       # Data source files
├── client/                        # Client-side code and setup
│   ├── index.html                 # Client HTML file
│   ├── Dockerfile                 # Docker setup for client
│   ├── README.md                  # Client README
│   ├── public/                    # Public assets
│   └── src/                       # Client source files
└── sql/                           # SQL setup and initialization scripts
    ├── Dockerfile                 # Docker setup for SQL
    └── init.sql                   # SQL initialization script

Jupyter notebooks for EDA and Model development

Can be found under notebook/
Subgroup A worked on: segment
Subgroup B worked on: roi-model and reco_system

notebook/
├── segment/                             # code for RFM, customer segmentation
├── roi-model/    
│   └── roi-model-with-wiki.ipynb        # EDA & Model development for ROI model
└── reco_system/  
    └── logistic_regression              # Model development
    └── New_data_creation_covid.ipynb    # EDA & data manipulation Notebook for Covid event
    └── New_data_creation_meteor.ipynb   # EDA & data manipulation Notebook for Meteor event 
    └── santender_cleaning.ipynb         # EDA Notebook 

Production-ready Python code

Main Python File

server/src/main.py          # The main entry point for the server-side application.
notebook/segment/main.py    # Script for FastAPI app with endpoints for data queries and ML predictions.
notebook/src/main.py        # Script for running data pipelines for recommendation system and ROI model. 
                            # Refer to "Data Pipelining for MLOps Overview" section at the end of this README.md

Other relevant files such as utils.py, logger.py etc can also be found in the repository.

Data sources and data preparation steps

• For data sources, refer to the Wiki 3. Overall Data Understanding Database Schema.

• For data preparation steps, refer to the wiki for relevant models such as RFM, Customer Behaviour EDA, Recommendation System and ROI model.
  Link to wiki: OpenAYEye Wiki

• For detailed coding steps for data preparation, please refer to the relevant Jupyter notebooks, as they include more detailed explanations and methods. Refer to "Jupyter notebooks for EDA and Model development" section above.

API documentation

Main Python File Found in

server/src/main.py          # The main entry point for the server-side application.

Dockerized Web App / API

• Run

bash docker-run.bash

• or

docker compose up

• For API, Wait for the following messages to appear

app   | INFO:     Started server process [1]
app   | INFO:     Waiting for application startup.
app   | INFO:     Application startup complete.
app   | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

• Frontend Webapp served on: localhost:5173
• Backend API served on: localhost:8000
• Webapp will be served when app container is ready

Data Pipelining for MLOps

This repository includes two main data pipeline systems: the Recommendation System (Reco) and the ROI Model. These pipelines automate the steps of data ingestion, transformation, model training, and prediction, crucial for machine learning workflows in business applications such as customer recommendation and return on investment (ROI) analysis.

One example:

Recommendation System Pipeline

• Purpose: To ingest, clean, transform, and generate personalized recommendations based on historical data.
• Pipelines:
  • Data ingestion and transformation pipeline.
  • Model training pipeline to generate insights.
  • Prediction pipeline for generating ranked recommendations.
• Location: /src/components/reco_sys/ for the components, /src/pipeline/reco_sys/ for the prediction pipeline.

ROI Model Pipeline

• Purpose: To process and analyze data to predict the ROI for marketing campaigns, based on features such as cost and category.
• Pipelines:
  • Data ingestion and transformation pipeline.
  • Model training pipeline for predicting ROI-related metrics.
  • Prediction pipeline for new input data.
• Location: /src/components/roi-model/ for the components, /src/pipeline/roi-model/ for the prediction pipeline.

For detailed instructions on running each pipeline, please refer to src/pipeline README.md.