Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New static downloads #76

Merged
merged 92 commits into from
Jul 23, 2024
Merged
Changes from 1 commit
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
201bc50
rebase
wbagge Aug 18, 2023
6106b94
commit soil download
wbagge Aug 24, 2023
d6410a5
saving progress
wbagge Oct 20, 2023
fbe73e9
elevation download and processing
wbagge Nov 3, 2023
7d733c3
changes to soil preprocessing
wbagge Nov 3, 2023
4f91ff9
first pass at building augmented dataset. getting error that some (no…
emmamendelsohn Nov 21, 2023
bccfd6f
closes #68 but still throws error on join "Last error: IOError: Could…
emmamendelsohn Nov 27, 2023
6bdd257
regenerated forecast anomaly files with arrow 14.0.0.2 and seems to b…
emmamendelsohn Dec 5, 2023
c85a0fd
tar_prune
emmamendelsohn Dec 5, 2023
0304ded
snapshot renv
emmamendelsohn Dec 5, 2023
f7dd0cd
rerunning forecast anomalies to replace NAs in files
emmamendelsohn Dec 8, 2023
f252bc5
finish forecast anomaly regeneration
emmamendelsohn Dec 11, 2023
3bf2083
complete anomaly dataset
emmamendelsohn Dec 15, 2023
5ad5f45
copy .rprofile from repel
emmamendelsohn Dec 15, 2023
3e2118b
add .Renvignore
emmamendelsohn Dec 15, 2023
894097b
first working shiny version
emmamendelsohn Dec 18, 2023
cb88281
working side by side maps with ndvi and ndvi anomalies
emmamendelsohn Dec 18, 2023
cbb1ad5
working slider bar
emmamendelsohn Dec 18, 2023
c334a37
some aesthetic improvements
emmamendelsohn Dec 18, 2023
5286d92
remove raw data - using anomalies only
emmamendelsohn Dec 19, 2023
b06c4ef
thee anomaly lags
emmamendelsohn Dec 19, 2023
d402686
add labels
emmamendelsohn Dec 19, 2023
8969f67
add dataset toggle
emmamendelsohn Dec 19, 2023
fc3b308
working temperature map
emmamendelsohn Dec 19, 2023
9a40eb9
working function to create maps
emmamendelsohn Dec 19, 2023
60773db
this works
emmamendelsohn Dec 19, 2023
636f674
works for temp too
emmamendelsohn Dec 19, 2023
83ca3e6
fix labels and color scale
emmamendelsohn Dec 19, 2023
c337178
add dynamic text
emmamendelsohn Dec 19, 2023
79c9f66
improve dynamic rendering
emmamendelsohn Dec 19, 2023
9e45018
more code improvements
emmamendelsohn Dec 19, 2023
4f91aa4
add other weather datasets
emmamendelsohn Dec 20, 2023
0821939
fix color scales
emmamendelsohn Dec 20, 2023
0342f9d
use augmented data instead of invidual anomaly sets
emmamendelsohn Dec 20, 2023
49144fa
tweaks
emmamendelsohn Dec 20, 2023
f659095
add forecasts but the layout is incorrect
emmamendelsohn Dec 20, 2023
d0f786a
small tweak
emmamendelsohn Dec 20, 2023
e6c6275
fix typo
emmamendelsohn Dec 20, 2023
4e13d5d
saving prgress
wbagge Dec 28, 2023
6d5e342
dynamic UI works
emmamendelsohn Dec 29, 2023
f3b941d
script to download augmented data from aws
emmamendelsohn Dec 29, 2023
187376f
this works to functionalize leaflet output but throws quick error bef…
emmamendelsohn Dec 29, 2023
005ec84
rename function
emmamendelsohn Dec 29, 2023
60b401c
separate tabs for forecasted and recorded data
emmamendelsohn Dec 29, 2023
e4467b0
add todo notes
emmamendelsohn Dec 29, 2023
7d6dee1
switch from tabs to radio buttons
emmamendelsohn Dec 30, 2023
87e9b1b
create target for comparison of validation/forecast data against reco…
emmamendelsohn Jan 3, 2024
8cf4903
run forecast anomaly validation
emmamendelsohn Jan 5, 2024
c6e6991
add select choice boxes - not yet connected
emmamendelsohn Jan 5, 2024
459a238
yay dynamic UI for maps no more error loading messages
emmamendelsohn Jan 5, 2024
7bc11d5
add dynamic tags
emmamendelsohn Jan 5, 2024
95687d2
cleanup, fix width
emmamendelsohn Jan 5, 2024
67af52d
get dynamic tags working
emmamendelsohn Jan 12, 2024
c0a7619
useful script for testing the code within the app
emmamendelsohn Jan 12, 2024
c954ac2
first pass at displaying comparisons
emmamendelsohn Jan 12, 2024
427e62d
working app with TODO list
emmamendelsohn Jan 12, 2024
bd1c6a3
quick code cleanup
emmamendelsohn Jan 12, 2024
7353eb5
Merge branch 'main' into feature/augment
emmamendelsohn Feb 2, 2024
720aeed
Merge pull request #74 from ecohealthalliance/feature/augment
emmamendelsohn Feb 2, 2024
5c3b138
Merge pull request #80 from ecohealthalliance/feature/data-viz
emmamendelsohn Feb 2, 2024
1542f94
filter augmented data to South Africa
emmamendelsohn Feb 2, 2024
2dca924
Take 2 - doing spatial aggregation by district at same time as filter…
emmamendelsohn Feb 2, 2024
67f94ac
Merge pull request #81 from ecohealthalliance/feature/rsa-data
emmamendelsohn Mar 4, 2024
956d4a7
full run for spatial aggregation
emmamendelsohn Mar 4, 2024
93e31ce
Add 1 git-crypt collaborator
noamross Jun 4, 2024
0cde870
remove .gitkeep
n8layman Jun 4, 2024
69d9241
rebase
wbagge Aug 18, 2023
b611fa8
commit soil download
wbagge Aug 24, 2023
824a092
saving progress
wbagge Oct 20, 2023
046da65
elevation download and processing
wbagge Nov 3, 2023
a800aed
changes to soil preprocessing
wbagge Nov 3, 2023
5640633
saving prgress
wbagge Dec 28, 2023
cee9c00
Finish rebase
noamross Jun 24, 2024
7de0fb2
Testing and taking notes on static pipeline
n8layman Jun 24, 2024
3e57b10
Update gitignore
n8layman Jun 26, 2024
cc548aa
Capture progress on working through static layer pipeline
n8layman Jun 26, 2024
3ec79f9
Add interim outbreak history function
noamross Jun 26, 2024
2c56cd4
Merge remote-tracking branch 'origin/new_static_downloads' into new_s…
noamross Jun 26, 2024
f14b830
Work on adding in outbreak histories. Currently incomplete
n8layman Jun 26, 2024
76b2670
Work on calc_outbreak_history. Remove circle geometry and define max …
n8layman Jun 27, 2024
f9ac4c7
Load targets in .Rprofile for interactive sessions only
n8layman Jul 16, 2024
b2d37e6
Comment out orphaned target cache_aws_branched_target
n8layman Jul 16, 2024
b7c8366
Run static_targets till failure
n8layman Jul 16, 2024
d174ab0
Add note on where to pick back up with slope_aspect download
n8layman Jul 16, 2024
a8c90a6
Split aspect and slope into re-usable get_remote_rasters function and…
n8layman Jul 20, 2024
e1891e6
add in bioclim and landcover data
n8layman Jul 20, 2024
cce599b
Finish statics except prior exposure to disease
n8layman Jul 20, 2024
4db41b0
Start on calc_outbreak_history
n8layman Jul 20, 2024
442fe88
Add daily outbreak history for both old and recent outbreaks
n8layman Jul 21, 2024
a5a5f8f
Finish static variables and outbreak history
n8layman Jul 22, 2024
6c61462
Add animations
n8layman Jul 22, 2024
bd55210
Update animation
n8layman Jul 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
elevation download and processing
wbagge committed Nov 3, 2023
commit fbe73e9fdb750659e665f3ade1f634268a5f104e
30 changes: 25 additions & 5 deletions R/get_elevation.R
Original file line number Diff line number Diff line change
@@ -7,11 +7,31 @@
#' @return
#' @author Whitney Bagge
#' @export
get_glw <- function() {
library(paws)
get_elevation<- function(elevation_directory_raw, overwrite=FALSE) {

url_cattle <- "https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/LHBICE#"
url_cattle_out <- GET(url_cattle)
unzipped_glw_cattle <- unzip(url_cattle_out)
Aw_layer <- if_else(str_detect(names(unzipped_glw_cattle), "Aw")==TRUE, )
existing_files <- list.files(elevation_directory_raw)

download_filename <- tools::file_path_sans_ext(existing_files)

save_filename <- paste0(download_filename, ".tif")

message(paste0("Downloading ", download_filename))

if(save_filename %in% existing_files & !overwrite) {
message("file already exists, skipping download")
return(file.path(elevation_directory_raw, save_filename)) # skip if file exists
}

s3_svc <- s3(config = list(region = "af-south-1", credentials = list(anonymous = TRUE)))
# Download the file directly to disk in the current working directory
s3_svc$download_file(
Bucket = "deafrica-input-datasets",
Key = "srtm_dem/srtm_africa.tif",
Filename = "data/elevation/srtm_africa.tif"
)


return(elevation_directory_raw)

}
17 changes: 13 additions & 4 deletions R/process_elevation.R
Original file line number Diff line number Diff line change
@@ -7,11 +7,20 @@
#' @return
#' @author Whitney Bagge
#' @export
process_elevation <- function(elevation_layer_raw, bounding_boxes) {
process_elevation<- function(elevation_directory_raw, elevation_directory_dataset, elevation_downloaded, continent_raster_template) {

extent_object <- extent(bounding_boxes)
elevation_layer_raw_out <- crop(elevation_layer_raw, extent_object)

return(elevation_layer_raw_out)
transformed_raster <- transform_raster(raw_raster = rast("data/elevation/srtm_africa.tif"),
template = rast(continent_raster_template))

# Convert to dataframe
dat_out2 <- as.data.frame(transformed_raster, xy = TRUE) |>
as_tibble()

# Save as parquet
write_parquet(dat_out2, "data/elevation_dataset/elevation_dataset", compression = "gzip", compression_level = 5)


return(elevation_directory_dataset)

}