Tal Mizrahi's Final Project for Computer Science Bachelor's, Recognized as Outstanding for the Class of 2024
Description • Architecture • API • Setup • Author
I developed a distributed task management system designed to efficiently assign tasks within user groups. The system includes a React.js client where users can create and manage groups, and a NestJS backend connected to MongoDB for securely storing user data. A second client, built in Python, will allow users to download and connect to the system with their existing credentials. Tasks will be distributed across group members, with only metadata and task statistics stored on the server, ensuring minimal data storage while maintaining robust task coordination.
You can browse to the web managment page here.
The Firmlight system consists of the following components:
- NestJS Server: Manages users, groups, and task distribution.
- Reactjs Client: Web interface for users to manage groups and initiate tasks.
- Python Worker Client: Downloadable client for processing distributed tasks.
- MongoDB: Database for storing user data, group information, and task metadata.
- Redis: In-memory data structure store used for caching and as a message broker.
- BullMQ: Queue system for handling distributed job processing.
- cron-jobs: Scheduled tasks for monitoring and maintenance.
- JSON Web Tokens: An authentication for representing claims securely between The server and clients.
- Users register/login via the React client, with credentials stored in MongoDB.
- Users create or join groups through the React interface.
- When a task is sent to a group, the server uses BullMQ to queue and distribute it among group members.
- Python worker clients process the tasks and send results back to the server.
- The server updates task status in real-time using WebSockets.
- Task results are sent to the appropriate client (React or Python).
- Only task metadata and statistics are stored on the server to maintain efficiency.
- Cron jobs periodically monitor task progress and perform system maintenance.
For detailed API documentation, please refer to our API Docs in the Swagger interface.
# Clone the repository
git clone https://github.com/TalMizrahii/firmlight.git
# Navigate to the project directory
cd firmlight/FirmlightWorker
# Install dependencies
pip install
# Sign up using the web interface
Browse to https://firmlight.onrender.com
# Login to the worker and start to work!
Using the same credentials of the web managment page