[EPIC] Rewrite

[jonblack]

This projects is unwieldly:

• The core update-databanks script is difficult to understand and poorly commented;
• No automated tests. Manual testing involves running it in an isolated environment and checking the results by hand;
• Too many responsibilities. In particular, it is responsible for backing itself up and updating whynot (see [EPIC] Clarify project boundaries whynot2#15).
• It uses whynot which is by default configured to update the production database, so testing is doubly difficult.

I propose we rewrite the process of updating and generating the databanks to solve the issues above.

This requires a lot of thought to determine what responsibilities should be in this process and which can be managed by separate services. For example, should generated databanks (e.g. hssp) be a separate service that watches for changes in its dependencies/checks to see what's missing?

[jonblack]

I've started https://github.com/cmbi/databanks2 since this issue was made but recently I'm wondering if it's the right approach. It uses Makefiles as in this repository (albeit in a much better way).

The problem is that it offers no API, which is useful for solving issues like cmbi/mrs#44 and #3.

[jonblack]

The current databanks scripts do not scale at all. Distributed storage and processing platforms offer a much nicer way to process PDB and mmCIF files to create the other databanks. The problem is that all we have are a couple of large supermicrocomputers rather than many commodity servers. Moreover, the network speed is only 100Mbps. The market leader in distributed processing is Hadoop with HDFS; however, this works better for fewer small files, whereas the databanks are composed of many small files.

I'm in favour of moving to a distributed platform like Hadoop.