Ingest BOR appeal data into lake #553
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@ccao-jardine this is in athena now if you want to take a look |
dfsnow
requested changes
Jul 24, 2024
Co-authored-by: Dan Snow <[email protected]>
wrridgeway
commented
Jul 24, 2024
| @@ -1,6 +1,3 @@ | |||
| **Primary Key**: `reascd` | |||
| {% enddocs %} | |||
|
|
|||
There was a problem hiding this comment.
Nothing to do with this PR, just random extraneous code that needed to be removed.
dfsnow
reviewed
Jul 26, 2024
| @@ -0,0 +1,9 @@ | |||
| # appeals | |||
|
|
|||
| {% docs table_appeals %} | |||
There was a problem hiding this comment.
thought: Realizing that maybe we made a mistake by not prefixing these table names in the docs with their respective database, since some stuff like appeals could definitely refer to multiple tables in the future. Fine for now, but something to fix in a later PR. Can you make an issue @wrridgeway?
Co-authored-by: Dan Snow <[email protected]>
Co-authored-by: Dan Snow <[email protected]>
dfsnow
approved these changes
Jul 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ETL pipeline to take data from BOR appeals open data asset and load it into athena. Data is more-or-less unaltered. Glue crawler for the
ccbordatabase was also added.See #548 for more details.
As of right now using the
RSocratapackage (sincejsonlitecan't handle the amount of data being gathered from the open data portal) which needs credentials. I've added them to.Renviron, but that's a file we track with git in this repo. So we'll need to figure out a solution for this.