db: use native SQL

[ellemouton]

Overview:

We should follow the example set by LND and migrate our kvdb stores to SQL. Luckily, LiTs set of tables is quite small and so this is quite a contained task. We also have never supported the kv schema in other backend options like sql or postgres and don't currently have any tables in any sql backend. This is nice because it keeps set of possible starting points to a size of 1: Migrate bbolt kvstore data to native sql stores (either postgres or sqlite). I think that since currently we only store LNC data in LiT's DB, nothing is critical and so I think we should force the migration at the end of the day and not necessarily let uses opt to not run the migration.

Getting started:

We can get started by:

1. defining the new SQL tables (dont actually trigger them in anything other than tests)
2. defining the SQL queries & putting them behind an interface such that all current unit tests can be run against both the kvdb backends and new sql backends.

The above can be done for the accounts, actions, session and firewalldb stores.

We can get started on the work for the above and merge things well before we need to worry about the final switch to sql.

End goal:

By the end of the project, we want a single SQL backend with the following relations.

Action Plan/Progress tracking

Phase 1: Defining Schemas

By the end of phase 1, we want to be able to spin up LiT in dev mode with a full SQL backend.
To do this, we will follow the following conversion strategy for each DB store:

1. massage the given Store interface into one that is more SQL compatible
2. Make any changes to the unit tests and KVDB impl that will make that unit tests runnable against all DB backends.
3. Define the SQL schemas & queries
4. Implement the SQL version of the Store interface (CRUD layer)
5. Get the CI to run all that DB's unit tests against all possible backends.
6. get the itests running against the new backends

We will do the above steps for each of the following and in the following order:

A) The Accounts store
B) Sessions store
C) rules kvstore
D) rules privacy mapper store
E) actions store.

Phase 2: Migration code

Here we will write the migration functions for each of the tables plus thorough tests for them. At the end of this phase, we are still not exposing any new functionality to a prod release.

Phase 3: Calling the migration code & handling user config

Finally, we will call the migration code & add some configuration values for switching to a new backend.

Progress Checklist

Phase 1 (Schemas):

Accounts:

• Prep/Cleanup
• Prepare the accounts Store interface to be ready for a SQL impl
• Prep unit tests to be ready to pass against a different backend
• Define the SQL schemas and queries
• implement the SQL CRUD (ie, the SQL version of the Store interface).
• Run the above impl against the existing unit tests (part of CI)
• CI step for getting itests to run against LiT with SQL store for accounts

Sessions:

• Prep/Cleanup
• Prepare the sessions Store interface to be ready for a SQL impl
• Prep unit tests to be ready to pass against a different backend
• Define the SQL schemas and queries
• implement the SQL CRUD (ie, the SQL version of the Store interface).
• Run the above impl against the existing unit tests (part of CI)
• CI step for getting itests to run against LiT with SQL store for sessions

KV stores

• Prep/Cleanup
• Prepare the kvstores interface to be ready for a SQL impl
• Prep unit tests to be ready to pass against a different backend
• Define the SQL schemas and queries
• implement the SQL CRUD (ie, the SQL version of the Store interface).
• Run the above impl against the existing unit tests (part of CI)
• CI step for getting itests to run against LiT with SQL store for kvstores

Privacy Pairs

• Prep/Cleanup
• Prepare the privacy map store interface to be ready for a SQL impl
• Prep unit tests to be ready to pass against a different backend
• Define the SQL schemas and queries
• implement the SQL CRUD (ie, the SQL version of the Store interface).
• Run the above impl against the existing unit tests (part of CI)
• CI step for getting itests to run against LiT with SQL store for privacy pairs

Actions

• Prep/Cleanup
• Prepare the actions store interface to be ready for a SQL impl
• Prep unit tests to be ready to pass against a different backend
• Define the SQL schemas and queries
• implement the SQL CRUD (ie, the SQL version of the Store interface).
• Run the above impl against the existing unit tests (part of CI)
• CI step for getting itests to run against LiT with SQL store for actions

Phase 2

TBC

Phase 3

TBC

[ellemouton]

cc @levmi for input re the two approach options

EDIT: realised we do actually need to migrate. can discuss offline