Inconsistent AI Data Parser Load Times Based on Dataset Breadth

[JulieHoaglandSorensen]

Description

The AI data parser in Bottlenetes shows significant variability in load times when processing datasets with different attribute breadth. Broader datasets result in disproportionately longer wait times, disrupting user workflows.

Expected Behavior:
The AI Data Parser should handle datasets consistently, with load times remaining proportional to dataset size, regardless of attribute breadth.

Actual Behavior:
Datasets with broader attributes take disproportionately longer to load. For example:

• Dataset A (5 attributes, 10,000 rows): ~2 seconds
• Dataset B (50 attributes, 10,000 rows): ~15 seconds

This pattern holds even when dataset size (row count) is constant.

Reproduction

Reproduction:

• Fork the prepository](https://github.com/oslabs-beta/BottleNetes.git.)

• Start the application using the quickstart direction portion in the ReadMe. Logs and sample datasets from testing the demo app are attached for further investigation and analysis of the AI data parser's inconsistent performance. Alternatively, you may test with an application of your choice.

• Upload datasets with varying attribute counts through the frontend:

  • Dataset A: 5 attributes, 10,000 rows.
  • Dataset B: 50 attributes, 10,000 rows.

• Test questions on the Ai Chatbot of varying attribute breadth (e.g., questions about a dataset with 5 fields versus one with 50+ fields).

• Measure the load time for each dataset.

System information

Full Dependencies for Bottlenetes

Dependencies:

@emotion/react: v11.14.0 – A library for writing CSS styles with JavaScript.
@emotion/styled: v11.14.0 – Styled components library built for Emotion.
@kubernetes/client-node: v0.22.3 – Kubernetes client for Node.js.
@mui/material: v6.3.0 – Material-UI, a React UI framework for building styled components.
axios: v1.7.9 – HTTP client for making API requests.
bcrypt: v5.1.1 – Library for hashing passwords.
body-parser: v1.20.3 – Middleware to parse incoming request bodies in Express.
chart.js: v4.4.7 – JavaScript library for creating data visualizations.
concurrently: v9.1.0 – Utility to run multiple commands concurrently.
cookie-parser: v1.4.7 – Middleware for parsing cookies in Express.
cors: v2.8.5 – Middleware for enabling Cross-Origin Resource Sharing.
dotenv: v16.4.7 – Module to load environment variables from a .env file.
express: v4.21.2 – Node.js framework for building web applications.
express-session: v1.18.1 – Middleware for managing sessions in Express.
express-validator: v7.2.0 – Middleware for validating and sanitizing inputs in Express.
jsonwebtoken: v9.0.2 – Implementation of JSON Web Tokens for secure data transmission.
lucide-react: v0.462.0 – Icon library for React.
moment: v2.30.1 – Library for parsing, validating, and manipulating dates and times.
openai: v4.74.0 – Client library for OpenAI API integration.
passport: v0.7.0 – Middleware for authentication in Node.js.
passport-github2: v0.1.12 – GitHub OAuth strategy for Passport.
passport-google-oauth20: v2.0.0 – Google OAuth 2.0 strategy for Passport.
pg: v8.13.1 – PostgreSQL client for Node.js.
pg-format: v1.0.4 – Library for safely creating SQL queries.
pg-hstore: v2.3.4 – Library for serializing and deserializing PostgreSQL hstore data.
react: v18.3.1 – Library for building user interfaces.
react-chartjs-2: v5.2.0 – React wrapper for Chart.js.
react-dom: v18.3.1 – React package for DOM-specific methods.
react-draggable: v4.4.6 – Component for making elements draggable.
react-icons: v5.4.0 – Icon library for React.
react-markdown: v9.0.1 – Component for rendering markdown as React elements.
react-router-dom: v7.0.2 – Router library for React.
sequelize: v6.37.5 – ORM for managing SQL databases.
wait-on: v8.0.1 – Utility for waiting on files, ports, sockets, and HTTP(s) resources.
zustand: v5.0.2 – State management library for React.

DevDependencies:

@eslint/js: v9.15.0 – ESLint utility for linting JavaScript.
@tailwindcss/typography: v0.5.15 – Tailwind CSS plugin for typography styling.
@types/bcrypt: v5.0.2 – TypeScript types for bcrypt.
@types/chart.js: v2.9.41 – TypeScript types for Chart.js.
@types/cookie-parser: v1.4.8 – TypeScript types for cookie-parser.
@types/cors: v2.8.17 – TypeScript types for CORS.
@types/express: v5.0.0 – TypeScript types for Express.
@types/jsonwebtoken: v9.0.7 – TypeScript types for jsonwebtoken.
@types/passport: v1.0.17 – TypeScript types for Passport.
@types/passport-github2: v1.2.9 – TypeScript types for passport-github2.
@types/passport-google-oauth20: v2.0.16 – TypeScript types for passport-google-oauth20.
@types/react: v18.3.12 – TypeScript types for React.
@types/react-dom: v18.3.1 – TypeScript types for react-dom.
@vitejs/plugin-react: v4.3.4 – React plugin for Vite.
autoprefixer: v10.4.20 – Tool for parsing and adding vendor prefixes to CSS.
eslint: v9.15.0 – Linter for JavaScript and TypeScript.
eslint-plugin-react: v7.37.2 – ESLint plugin for React.
eslint-plugin-react-hooks: v5.0.0 – ESLint plugin for React hooks.
eslint-plugin-react-refresh: v0.4.14 – ESLint plugin for React refresh.
globals: v15.12.0 – Global variable definitions for ESLint.
nodemon: v3.1.9 – Tool for automatically restarting Node.js applications.
postcss: v8.4.49 – Tool for transforming styles with JS plugins.
prettier: v3.4.1 – Code formatter.
prettier-plugin-tailwindcss: v0.6.9 – Prettier plugin for Tailwind CSS.
tailwindcss: v3.4.17 – Utility-first CSS framework.
ts-node: v10.9.2 – TypeScript execution environment for Node.js.
typescript: v5.7.2 – Superset of JavaScript for static typing.
typescript-eslint: v8.19.0 – Tooling for ESLint with TypeScript.
vite: v6.0.6 – Build tool for frontend projects.

Additional information

Priority

High – This variability impacts scalability and user experience, particularly for broader applications.

👨‍👧‍👦 Contributing

• 🙋‍♂️ Yes, I'd love to make a PR to fix this bug!