This repository has been archived by the owner on Feb 29, 2020. It is now read-only.

download python package git repos #1

Open

tbonza opened this issue Jun 5, 2019 · 1 comment

Assignees

Labels

Contributor

tbonza commented Jun 5, 2019

No description provided.

tbonza self-assigned this

tbonza added the enhancement label

tbonza added a commit that referenced this issue


          updated readme with first release support, #1

368208a

Contributor Author

tbonza commented Jun 16, 2019

We want to first prioritize by collecting python packages. When I review pypi.org/robots.txt

Parse the PyPi sitemap.xml to get all the Python package names on PyPI
Use python package ids from previous step to make API calls to PyPI (see pypi api reference docs)
Store API results as JSON files for now with project name and datetime as file name so okra-20190616.json

We may be able to make parallel calls to the API by using Kubernetes Jobs

treble37 added a commit that referenced this issue


          Initial golang setup and basic download

399210f

Setup with new golang modules and golang 1.12

Install guide here: https://gist.github.com/rubencaro/5ce32fb30bbfa70e7db6be14cf42a35c

Also, download json files from pypi; addresses #1

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.