Skip to content
This repository has been archived by the owner on Sep 15, 2022. It is now read-only.

Possible issue pulling from a repository with a large number of tags. #22

Closed
bshi opened this issue Sep 10, 2014 · 6 comments · May be fixed by #28
Closed

Possible issue pulling from a repository with a large number of tags. #22

bshi opened this issue Sep 10, 2014 · 6 comments · May be fixed by #28

Comments

@bshi
Copy link

bshi commented Sep 10, 2014

I have an internal GCS-backed repository, lets call it "acmecorp/FOOPROJECT", that has accumulated quite a few tags (~200+). I am unable to fetch a particular image tag from it using google's docker-registry.

By the way this might be an upstream problem but it appears it's been several months since google's registry was sync'ed with the official docker registry project so I'm filing the bug here for lack of a better alternative. Please advise if there is a more appropriate place for this.

When I attempt to pull "acmecorp/FOOPROJECT:some-specific-tag" it appears the registry performs a linear scan over ALL tag files with several thousand lines of output like:

Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,966 DEBUG: path=/acmecorp-docker-repository/repositories/acmecorp/FOOPROJECT/tag_gff622ff
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,966 DEBUG: auth_path=/acmecorp-docker-repository/repositories/acmecorp/FOOPROJECT/tag_gff622ff
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,967 DEBUG: Method: HEAD
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,967 DEBUG: Path: /acmecorp-docker-repository/repositories/acmecorp/FOOPROJECT/tag_gff622ff
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,968 DEBUG: Data:
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,969 DEBUG: Headers: {}
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,969 DEBUG: Host: storage.googleapis.com
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,970 DEBUG: Port: 443
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,970 DEBUG: Params: {}
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,970 DEBUG: Token: None
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,971 DEBUG: GetAccessToken: checking cache for key da39a3ee5e6b4b0d3255bfef95601890afd80709
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,971 DEBUG: InMemoryTokenCache.GetToken: key=da39a3ee5e6b4b0d3255bfef95601890afd80709 present
Sep 10 20:43:20 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:20,971 DEBUG: GetAccessToken: token from cache: AccessToken(token=ya29.fADHyBWAPKdgzNZC2mM6RjUy5ipa5LothrWQ6Va4_bZFwMCwMYOx59a9, expiry=2014-09-10 21:02:08.971254Z)

Finally, the repository chokes with the following error and responds with a 500

Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:21 [18] [ERROR] Error handling request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: Traceback (most recent call last):
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 39, in handle
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: self.handle_request(req, client, addr)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 86, in handle_request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: super(GeventWorker, self).handle_request(*args)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 74, in handle_request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: resp.write(item)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 275, in write
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: util.write(self.sock, arg, self.chunked)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 240, in write
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: sock.sendall(data)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 504, in sendall
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: data_sent += self.send(_get_memory(data, data_sent), flags)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 478, in send
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: return sock.send(data, flags)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: error: [Errno 32] Broken pipe
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:21,563 ERROR: Error handling request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: Traceback (most recent call last):
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 39, in handle
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: self.handle_request(req, client, addr)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 86, in handle_request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: super(GeventWorker, self).handle_request(*args)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 74, in handle_request
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: resp.write(item)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 275, in write
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: util.write(self.sock, arg, self.chunked)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 240, in write
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: sock.sendall(data)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 504, in sendall
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: data_sent += self.send(_get_memory(data, data_sent), flags)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 478, in send
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: return sock.send(data, flags)
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: error: [Errno 32] Broken pipe
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: "172.17.42.1 - - [10/Sep/2014:20:43:21] "GET /v1/repositories/acmecorp/FOOPROJECT/tags HTTP/1.1" 500 0 "-" "-"
Sep 10 20:43:21 co-FOOPROJECT-00.c.blue-fire.internal docker[1268]: 2014-09-10 20:43:21,568 INFO: "172.17.42.1 - - [10/Sep/2014:20:43:21] "GET /v1/repositories/acmecorp/FOOPROJECT/tags HTTP/1.1" 500 0 "-" "-"

I have confirmed that the tag I'm attempting to pull does indeed exist.
@proppy
Copy link
Member

proppy commented Sep 10, 2014

I wonder if solving #14 would fix this.

This was referenced Oct 3, 2014
Closed
Merged
@bshi
Copy link
Author

bshi commented Oct 7, 2014

Tested #14 and #23 and they do not resolve the issue. The repository is still scanning all tags which takes a long time and triggers an IO timeout.

@proppy
Copy link
Member

proppy commented Oct 7, 2014

@bshi are you able to reproduce this with another backend (local? s3?), if that the case we should probably report it upstream.

@bshi
Copy link
Author

bshi commented Oct 7, 2014

Not familiar with setting up any other backend. I did a cursory scan of the issues in docker/docker-registry and it doesn't seem anyone else has reported the issue. I'll report it upstream at any rate and link to this issue.

@bshi bshi mentioned this issue Oct 7, 2014
Open
@bshi
Copy link
Author

bshi commented Oct 22, 2014

I have a patch for this issue (and docker-archive#614) but it's against the current master. The offending function(s) haven't changed much from Google's version to Docker's current head of master but I'll wait for #23 to go in as any patch against current master will probably be out of date.

@bshi
Copy link
Author

bshi commented Oct 22, 2014

Closing in favor of upstream bug referenced above.

@bshi bshi closed this as completed Oct 22, 2014
@bshi bshi mentioned this issue Nov 6, 2014
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update upstream docker/docker-registry repository
2 participants
@proppy @bshi