Implement Scrapy style concurrency controls?

[philbudne]

Currently rss-fetcher has limits on concurrent requests and minimum interval between connections to rss servers.

My reading of Scrapy (and what I implemented in the queue-based story fetcher under control of SCRAPY_LATENCY) is to keep a moving average of page fetch time for each destination server, and to use AVG_FETCH_TIME/CONCURRENT_CONNECTION_GOAL to calculate the connection interval.

The only time this might matter is when the server has been off-line (down, or off the Internet) and there is a large backlog of feeds overdue for fetching.