Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indexing fails on newly fetched gems #38

Open
mayurva opened this issue Sep 28, 2016 · 6 comments
Open

Indexing fails on newly fetched gems #38

mayurva opened this issue Sep 28, 2016 · 6 comments
Labels
bug

Comments

@mayurva
Copy link
Contributor

mayurva commented Sep 28, 2016

Hello,

We notice a weird issue where if we fetch a gem which does not exist on Gemirro, it gets fetched fine from Rubygems.org and also gets served to the client. However the index_update seems to fail (https://github.com/PierreRambaud/gemirro/blob/master/lib/gemirro/server.rb#L151). Any new gem fetches after this point fail until the server is restarted. When server is restarted, again it works for 1 new fetch, but index update fails and so on.
@PierreRambaud
Copy link
Owner

Hi and thanks for your feedback.
I'm not able to reproduce, can you tell me more about your rubygems versions? (gem --version)
Regards

@hemmrich
Copy link

hemmrich commented Oct 3, 2016

Same version as in the other issue (#39) that I created:

bash-4.1$ gem --version
2.5.1
bash-4.1$ ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

@PierreRambaud
Copy link
Owner

Hum, really strange, this part of code really bothers me. Look like there is a bug when updating indexes, but there is also a bug when trying to fetch a gem with many dependencies (consuming all memory)...
If you have an idea @hemmrich i'm interested :)

@stdietrich
Copy link

Hello,

I think we are hitting the same issue now. Installing a gem works, but installing a different gem fails afterwards:

# gem install ntail
Fetching: ntail-1.3.2.gem (100%)
Successfully installed ntail-1.3.2
Parsing documentation for ntail-1.3.2
Installing ri documentation for ntail-1.3.2
1 gem installed
# gem install typrtail
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    bad response Not Found 404 (http://<server>:2000/gems/typrtail-0.0.3.gem)

I, [2016-12-01T12:09:15.940345 #22491]  INFO -- : Try to download ntail with version 1.3.2
I, [2016-12-01T12:09:15.941074 #22491]  INFO -- : Fetching ntail-1.3.2.gem
I, [2016-12-01T12:09:15.941116 #22491]  INFO -- : Fetching gem ntail-1.3.2.gem on http://rubygems.org
I, [2016-12-01T12:09:16.161869 #22491]  INFO -- : Generating indexes
I, [2016-12-01T12:09:16.175377 #22491]  INFO -- : [1/1]: Processing ntail-1.3.2.gem
E, [2016-12-01T12:09:16.208925 #22491] ERROR -- : undefined method `name' for "/srv/repo/extra/rubygems/public/gems/ntail-1.3.2.gem":String (NoMethodError)
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/server.rb:171:in `block in update_indexes'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/pmap-1.0.2/lib/pmap.rb:35:in `block (2 levels) in process_core'
131.169.72.98 - - [01/Dec/2016:12:09:16 +0100] "GET /gems/ntail-1.3.2.gem HTTP/1.1" 200 27136 0.2699

I, [2016-12-01T12:09:48.904943 #22491]  INFO -- : Try to download typrtail with version 0.0.3
I, [2016-12-01T12:09:48.905138 #22491]  INFO -- : Fetching typrtail-0.0.3.gemspec.rz
I, [2016-12-01T12:09:48.905186 #22491]  INFO -- : Fetching gemspec typrtail-0.0.3.gemspec.rz on http://rubygems.org
I, [2016-12-01T12:09:49.151309 #22491]  INFO -- : Generating indexes
I, [2016-12-01T12:09:49.161387 #22491]  INFO -- : No new gems
I, [2016-12-01T12:09:49.161495 #22491]  INFO -- : Exiting RubyGems with exit_code 0
<client IP> - - [01/Dec/2016:12:09:49 +0100] "GET /quick/Marshal.4.8/typrtail-0.0.3.gemspec.rz HTTP/1.1" 200 355 0.2593
I, [2016-12-01T12:09:49.166105 #22491]  INFO -- : Try to download typrtail with version 0.0.3
I, [2016-12-01T12:09:49.166630 #22491]  INFO -- : Generating indexes
I, [2016-12-01T12:09:49.174168 #22491]  INFO -- : No new gems
I, [2016-12-01T12:09:49.174239 #22491]  INFO -- : Exiting RubyGems with exit_code 0
<client IP> - - [01/Dec/2016:12:09:49 +0100] "GET /gems/typrtail-0.0.3.gem HTTP/1.1" 404 1164 0.0133

# ruby --version
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux-gnu]
# gem --version
2.6.8
# gem list | grep gemirro
gemirro (0.13.5)

fetch_gem is set to true in the config.rb. Any ideas?
Restarting the server works as a workaround, until the next gem is encountered.

PierreRambaud added a commit that referenced this issue Dec 1, 2016
@PierreRambaud
Failed to flush cache
719f97a 
Issue: #38
@stdietrich
Copy link

Hello,

I applied the changes from commit 719f97a in our instance, but the problem is still appearing.

Gemirro server

I, [2017-01-16T13:18:15.653129 #16363]  INFO -- : Try to download rubygems-update with version 1.3.6
I, [2017-01-16T13:18:15.755997 #16363]  INFO -- : Fetching rubygems-update-1.3.6.gemspec.rz
I, [2017-01-16T13:18:16.091686 #16363]  INFO -- : Fetching gemspec rubygems-update-1.3.6.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:18:17.254837 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:18:17.362592 #16363]  INFO -- : No new gems
I, [2017-01-16T13:18:17.362734 #16363]  INFO -- : Exiting RubyGems with exit_code 0
127.0.0.1 - - [16/Jan/2017:13:18:17 +0100] "GET / HTTP/1.1" 200 4908 3.6063
I, [2017-01-16T13:19:00.412711 #16363]  INFO -- : Try to download vagrant-hiera with version 0.1.4
I, [2017-01-16T13:19:00.413108 #16363]  INFO -- : Fetching vagrant-hiera-0.1.4.gemspec.rz
I, [2017-01-16T13:19:00.413150 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.1.4.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.414406 #16363]  INFO -- : Try to download vagrant-hiera with version 0.1.20
I, [2017-01-16T13:19:00.415296 #16363]  INFO -- : Try to download vagrant-hiera with version 0.1.30
I, [2017-01-16T13:19:00.416043 #16363]  INFO -- : Try to download vagrant-hiera with version 0.1.31
I, [2017-01-16T13:19:00.416766 #16363]  INFO -- : Try to download vagrant-hiera with version 0.2.1
I, [2017-01-16T13:19:00.417639 #16363]  INFO -- : Try to download vagrant-hiera with version 0.3.0
I, [2017-01-16T13:19:00.418450 #16363]  INFO -- : Try to download vagrant-hiera with version 0.3.2
I, [2017-01-16T13:19:00.419325 #16363]  INFO -- : Try to download vagrant-hiera with version 0.3.3
I, [2017-01-16T13:19:00.420166 #16363]  INFO -- : Try to download vagrant-hiera with version 0.3.4
I, [2017-01-16T13:19:00.420923 #16363]  INFO -- : Try to download vagrant-hiera with version 0.3.5
I, [2017-01-16T13:19:00.421659 #16363]  INFO -- : Fetching vagrant-hiera-0.1.20.gemspec.rz
I, [2017-01-16T13:19:00.421893 #16363]  INFO -- : Fetching vagrant-hiera-0.1.30.gemspec.rz
I, [2017-01-16T13:19:00.426290 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.1.30.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.422360 #16363]  INFO -- : Fetching vagrant-hiera-0.2.1.gemspec.rz
I, [2017-01-16T13:19:00.422906 #16363]  INFO -- : Fetching vagrant-hiera-0.3.0.gemspec.rz
I, [2017-01-16T13:19:00.425443 #16363]  INFO -- : Fetching vagrant-hiera-0.3.2.gemspec.rz
I, [2017-01-16T13:19:00.425618 #16363]  INFO -- : Fetching vagrant-hiera-0.3.3.gemspec.rz
I, [2017-01-16T13:19:00.425811 #16363]  INFO -- : Fetching vagrant-hiera-0.3.4.gemspec.rz
I, [2017-01-16T13:19:00.426101 #16363]  INFO -- : Fetching vagrant-hiera-0.3.5.gemspec.rz
I, [2017-01-16T13:19:00.426191 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.1.20.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.422124 #16363]  INFO -- : Fetching vagrant-hiera-0.1.31.gemspec.rz
I, [2017-01-16T13:19:00.426637 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.2.1.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.426729 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.3.0.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.426805 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.3.2.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.426873 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.3.3.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.426941 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.3.4.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.427004 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.3.5.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:00.428158 #16363]  INFO -- : Fetching gemspec vagrant-hiera-0.1.31.gemspec.rz on http://rubygems.org
I, [2017-01-16T13:19:01.170349 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.292680 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:01.292815 #16363]  INFO -- : Exiting RubyGems with exit_code 0
I, [2017-01-16T13:19:01.524803 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.531396 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:01.531500 #16363]  INFO -- : Exiting RubyGems with exit_code 0
I, [2017-01-16T13:19:01.669682 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.670582 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.830949 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:01.833211 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.869831 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.869930 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:01.876371 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:01.876561 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:02.069603 #16363]  INFO -- : Exiting RubyGems with exit_code 0
I, [2017-01-16T13:19:02.176125 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:02.176747 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:02.291797 #16363]  INFO -- : Generating indexes
I, [2017-01-16T13:19:02.298260 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:02.376389 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:02.676955 #16363]  INFO -- : No new gems
I, [2017-01-16T13:19:02.677043 #16363]  INFO -- : Exiting RubyGems with exit_code 0
E, [2017-01-16T13:19:02.971023 #16363] ERROR -- : closed stream (IOError)
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/user_interaction.rb:699:in `close'
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/user_interaction.rb:699:in `close'
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/user_interaction.rb:388:in `terminate_interaction'
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/user_interaction.rb:158:in `terminate_interaction'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/indexer.rb:255:in `update_index'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/server.rb:169:in `update_indexes'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/server.rb:152:in `fetch_gem'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/server.rb:259:in `spec_for'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/gemirro-0.13.5/lib/gemirro/server.rb:219:in `block (2 levels) in gem_dependencies'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/pmap-1.1.1/lib/pmap.rb:82:in `call'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/pmap-1.1.1/lib/pmap.rb:82:in `block (2 levels) in pmap'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/pmap-1.1.1/lib/pmap/thread_pool.rb:58:in `call'
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/pmap-1.1.1/lib/pmap/thread_pool.rb:58:in `block in spawn_worker'

gem install

# gem install vagrant-hiera
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    bad response Not Found 404 (http://it-fs4.desy.de:2000/gems/vagrant-hiera-0.4.0.gem

@nike38rus
Copy link

nike38rus commented Mar 27, 2017 •
edited
Loading

I'm experiencing the same issue as well

Server-side:

ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
gemirro (0.13.5)

Client-side

$ ruby --version
ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
$ gem --version
2.5.1

The first gem fetched and installed successfully and second and next is failing with error until gemirror webserver will be restarted:

$ sudo gem update
Updating installed gems
Updating bundler
Fetching: bundler-1.14.6.gem (100%)
Successfully installed bundler-1.14.6
Parsing documentation for bundler-1.14.6
Installing ri documentation for bundler-1.14.6
Installing darkfish documentation for bundler-1.14.6
Done installing documentation for bundler after 7 seconds
Parsing documentation for bundler-1.14.6
Done installing documentation for bundler after 4 seconds
Updating childprocess
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    bad response Not Found 404 (http://rpm.ftc.ru/gems/childprocess-0.6.3.gem)

On the server side I have the same errors as @stdietrich had

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mayurva @PierreRambaud @hemmrich @stdietrich @nike38rus