Skip to content

VMRuiz/hcf-backend

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HCF (HubStorage Frontier) Backend for Frontera

When used with scrapy, use it with Scrapy Scheduler provided by scrapy-frontera. Scrapy scheduler provided by Frontera is not supported

See usage instructions at module and class docstrings at backend.py.

Package also installs a convenient command line tool for hubstorage frontier handling and manipulation: hcfpal.py. It supports dumping, count, deletion, moving, listing, etc.

Another tool provided is hcfmanager.py. It facilitates the scheduling of consumer spider jobs.

About

Crawl Frontier HCF backend

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 100.0%