Feature Request: Minify Option to Reduce Output Size

[escorciav]

Thank you for the great tool! While using Gitingest, I encountered a challenge with large repositories containing irrelevant or extraneous information in the .txt output. This includes files such as:

• Jupyter Notebooks with embedded images.
• CSV files with results or large datasets.
• (possibly) Binaries or other non-textual data.

These often inflate the file size unnecessarily, making it difficult for downstream tools (e.g., ChatGPT) to process them efficiently.

Feature Request:

• Add a minify option to exclude or summarize such files in the output. This option could work by:
  • Stripping or summarizing Jupyter Notebooks (e.g., excluding image data or non-code cells).
  • Skipping large files like CSVs or binaries entirely.
  • Allowing configurable file-type or size exclusions (e.g., via an argument like --exclude ".csv,.ipynb,*.bin" or --max-file-size 5MB).

Use Case:
This would be particularly useful for repositories where only code and textual content are relevant, improving the usability and performance of gitingest outputs.

Proposed Approach:
A potential implementation could involve:

• Scanning file extensions or MIME types to exclude or process specific formats.
• Adding a size threshold to skip overly large files.
• Leveraging libraries like nbconvert for Jupyter Notebook minification.

Thank you for considering this enhancement!

[cyclotruc]

Thank you for your feedback

At the moment the non text files should be ignored by default, could you provide an example link where you this issue?

[escorciav]

Try out with this repo. I got a file around 8MB. I manually minify it to 568Kb.

mlfoundations-open_clip.txt

[cyclotruc]

Ok this makes sense, i'm going to look into that

[escorciav]

I guess someone is hammering at this ⚒️ 🚀

[cyclotruc]

Don't hesitate to tell me if there's anything you whish you could do that those two settings don't permit.

I plan to add a "minify" as part of an advanced settings revamp but some issues are higher priority right now

[cyclotruc]

@filipchristiansen made this #105

And 3 weeks after my last answer I realise that the priorities are now shifting towards improving the points you mentioned