Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Miovision: Extremely slow aggregation for single intersection, many days #862

Closed
gabrielwol opened this issue Feb 7, 2024 · 0 comments · Fixed by #863 or #864
Closed

Miovision: Extremely slow aggregation for single intersection, many days #862

gabrielwol opened this issue Feb 7, 2024 · 0 comments · Fixed by #863 or #864
Assignees
@gabrielwol

Comments

@gabrielwol
Copy link
Collaborator

When backfilling new intersections, there is still a lot of room to improve the speed of the aggregation process for single intersections:

  1. volumes_15min_mvt aggregation is extremely slow (hours) when run for many days, likely due to large cross join.
  2. unacceptable_gaps aggregation is missing an intersection parameter. 20+ minutes for 5 months.
  3. volumes_daily aggregation is missing an intersection parameter
  4. agg_zero_volume_anomalous_ranges is missing an intersection parameter. This could interfere with manual edits to the automated anomalous_ranges.
@gabrielwol gabrielwol self-assigned this Feb 7, 2024
gabrielwol added a commit that referenced this issue Feb 7, 2024
@gabrielwol
#862 add intersection param to 3 remaining aggregate functions + adju…
c8cc0e7 
…st dag and intersections_tmc
gabrielwol added a commit that referenced this issue Feb 7, 2024
@gabrielwol
#862 much faster version of aggregate_15_min_mvt
7484744
@gabrielwol gabrielwol mentioned this issue Feb 7, 2024
Merged
@gabrielwol gabrielwol linked a pull request Feb 7, 2024 that will close this issue
#862 much faster version of aggregate_15_min_mvt #863
Merged
gabrielwol added a commit that referenced this issue Feb 7, 2024
@gabrielwol
#862 fix small gt to gte bug
ea0d519
gabrielwol added a commit that referenced this issue Feb 7, 2024
@gabrielwol
#862 remove already aggregated filter - we use clear function instead…
8089230 
… now
@gabrielwol gabrielwol mentioned this issue Feb 8, 2024
Merged
@gabrielwol gabrielwol linked a pull request Feb 8, 2024 that will close this issue
#862 add intersection param to 3 remaining aggregate functions + adju… #864
Merged
gabrielwol added a commit that referenced this issue Feb 8, 2024
@gabrielwol
#862 make alter statements specific to params
bce252a
chmnata pushed a commit that referenced this issue Feb 9, 2024
@gabrielwol
#862 add intersection param to 3 remaining aggregate functions + adju… (
3a523e6 
#864)

* #862 add intersection param to 3 remaining aggregate functions + adjust dag and intersections_tmc

* #862 make alter statements specific to params
chmnata pushed a commit that referenced this issue Feb 13, 2024
@gabrielwol
#862 much faster version of aggregate_15_min_mvt (#863)
085a020 
* #862 much faster version of aggregate_15_min_mvt

* #862 fix small gt to gte bug

* #862 remove already aggregated filter - we use clear function instead now
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gabrielwol gabrielwol

Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@gabrielwol