Btdag

bluetooth/README.md

@@ -2,39 +2,66 @@
 
 ## Table of Contents
 
-- [Table of Contents](#table-of-contents)
-- [1. Overview](#1-overview)
-- [2. Table Structure](#2-table-structure)
+- [Bluetooth - Bliptrack](#bluetooth---bliptrack)
+  - [Table of Contents](#table-of-contents)
+  - [1. Overview](#1-overview)
+  - [2. Table Structure](#2-table-structure)
     - [Open Data Tables](#open-data-tables)
-        - [Live Feed](#live-feed)
-        - [Historical Data](#historical-data)
-        - [Geography](#geography)
+      - [Live Feed](#live-feed)
+      - [Historical Data](#historical-data)
+      - [Geography](#geography)
     - [Internal Tables](#internal-tables)
-        - [Observations](#observations)
-            - [Filtering devices](#filtering-devices)
-        - [all_analyses](#all_analyses)
-        - [ClassOfDevice](#classofdevice)
-- [3. Technology](#3-technology)
-- [4. Bliptrack UI](#4-bliptrack-ui)
+      - [Observations](#observations)
+        - [Filtering devices](#filtering-devices)
+      - [all_analyses](#all_analyses)
+      - [reader_history](#reader_history)
+      - [reader_locations](#reader_locations)
+      - [routes](#routes)
+      - [reader_status_history](#reader_status_history)
+      - [ClassOfDevice](#classofdevice)
+  - [3. Technology](#3-technology)
+  - [4. Bliptrack UI](#4-bliptrack-ui)
     - [Accessing Bliptrack](#accessing-bliptrack)
-        - [Terms](#terms)
-        - [Downloading travel time data](#downloading-travel-time-data)
-        - [Common Issues](#common-issues)
-- [5. Bliptrack API](#5-bliptrack-api)
+      - [Terms](#terms)
+      - [Downloading travel time data](#downloading-travel-time-data)
+      - [Common Issues](#common-issues)
+  - [5. Bliptrack API](#5-bliptrack-api)
     - [Pulling travel time data](#pulling-travel-time-data)
-        - [Under the Hood](#under-the-hood)
-        - [The `analysisId`](#the-analysisid)
-- [6. Bliptrack API OD Data](#6-bliptrack-api-od-data)
+      - [Under the Hood](#under-the-hood)
+      - [The `analysisId`](#the-analysisid)
+  - [6. Bliptrack API OD Data](#6-bliptrack-api-od-data)
     - [Start-End Data](#start-end-data)
-        - [Some notes on `measuredTime` and records](#some-notes-on-measuredtime-and-records)
-        - [Dictionary Structure](#dictionary-structure)
+      - [Some notes on `measuredTime` and records](#some-notes-on-measuredtime-and-records)
+      - [Dictionary Structure](#dictionary-structure)
     - [Others Data](#others-data)
     - [deviceClass and outlierLevel](#deviceclass-and-outlierlevel)
-        - [For the Start-End Data](#for-the-start-end-data)
-        - [For the Others Data](#for-the-others-data)
-        - [outliersLevel](#outlierslevel)
-- [7. Adding New Segments to the Database](#7-adding-new-segments-to-the-database)
-- [8. Open Data Releases](#8-open-data-releases)
+      - [For the Start-End Data](#for-the-start-end-data)
+      - [For the Others Data](#for-the-others-data)
+      - [outliersLevel](#outlierslevel)
+  - [7. Adding New Segments to the Database](#7-adding-new-segments-to-the-database)
+  - [8. Open Data Releases](#8-open-data-releases)
+
+  - [9. Technology](#3-technology)
+  - [10. Bliptrack UI](#4-bliptrack-ui)
+    - [Accessing Bliptrack](#accessing-bliptrack)
+      - [Terms](#terms)
+      - [Downloading travel time data](#downloading-travel-time-data)
+      - [Common Issues](#common-issues)
+  - [11. Bliptrack API](#5-bliptrack-api)
+    - [Pulling travel time data](#pulling-travel-time-data)
+      - [Under the Hood](#under-the-hood)
+      - [The `analysisId`](#the-analysisid)
+  - [12. Bliptrack API OD Data](#6-bliptrack-api-od-data)
+    - [Start-End Data](#start-end-data)
+      - [Some notes on `measuredTime` and records](#some-notes-on-measuredtime-and-records)
+      - [Dictionary Structure](#dictionary-structure)
+    - [Others Data](#others-data)
+    - [deviceClass and outlierLevel](#deviceclass-and-outlierlevel)
+      - [For the Start-End Data](#for-the-start-end-data)
+      - [For the Others Data](#for-the-others-data)
+      - [outliersLevel](#outlierslevel)
+  - [13. Adding New Segments to the Database](#7-adding-new-segments-to-the-database)
+  - [14. Open Data Releases](#8-open-data-releases)
 
 ## 1. Overview
 
@@ -134,6 +161,72 @@ The script pulls the route configurations nightly from the Blip server. These ar
 |pull_data|boolean| (defaults to false) whether the script should pull observations |
 `outcomes` are set for different routes for purposes like: filtering BT and WiFi, or tracking Origin Destination points.
 
+
+#### reader_history
+
+This is a table of bluetooth readers that have been installed at each locations at different times in the past. Readers that have been installed at any point of time in the past are listed in this table irrespective of whether the reader is still physically present at the installed location or not. This table is the sum-total of all the readers irrespective of their current status.
+
+
+ `reader_history` table contains the following fields:
+
+|Column|Type|Notes|
+|------|----|-----|
+|`reader_history_id`|integer| Unique ID for each reader|
+|`reader_id`|varchar|foreign key to `reader_locations`. This is a location id. Multiple readers could have been installed in a same location at different times.|  
+|`serial_no_bluetooth`|integer|This is a four digit number that is assigned to each bluetooth reader. This serial number corresponds to the **Zone** in the bliptrack table.| 
+|`serial_no_wifi`|integer|Some readers have both wifi and bluetooth sensors. For those readers which has the wifi sensor its serial number is populated.|
+|`date_installed`|date|Date the reader is installed at the location|
+|`date_uninstalled`|date|Date the reader is uninstalled at the location|
+
+Except the `reader_history_id` all other fields in this table has to be updated manually.
+
+#### reader_locations 
+
+This is a table of all locations at which Bliptrack readers have been installed and are physically existing. The installed readers could be online or offline but has NOT been removed physically. Each intersection has only **one** reader that is assigned to a route/routes. Therefore, if there are more than one readers in a locations that have not been removed, such detectors are listed in the `reader_history` table. The function [`sql/functions/reader_status_history`](sql/functions/) updates the field `date_last_received`. This table consists of the following fields:
+|Column|Type|Notes|
+|------|----|-----|
+|`reader_id`|integer|Unique ID for a unique reader that corresponds to the `reader_id` in the `reader_history` table.|
+|`name`|varchar|Name of location consisting of two characters for E/W street, two characters for N/S street. for example QU_DF for Queen st and Dufferin St. Whatever name is already existing has been retained for example, A, B, C or Beechwood, Castlefield etc has been retained)|
+|`int_id`|integer|Centreline intersection id for closest intersection or pseudo intersections in case of an expressway. A logical location closest to a reader that would be an intersection (Node)|
+|`date_active`|date|The date this reader was installed.|
+|`date_inactive`|date|NULL unless this field is updated manually to reflect the date when this reader is deemed inactive.|
+|`date_last_received`|timestamp without time zone|Latest date when the `aggr_5min` table has data aggregated for this reader. This field is updated daily by the function [`sql/functions/reader_status_history`](sql/functions/).|  
+|`project_name`|varchar|Name of the project by which the detector is installed. 
+|`geom`|geometry| Geometry of the location.|
+
+
+#### routes 
+
+This is a table of all the routes that pass through the locations (which are either intersections or pseudo intersections) where readers are installed. It corresponds to a unique segments on which data is collected from the network of readers. For a two way street, routes are created for both directions such as Eastbound (EB) - Westbound (WB) or Northbound (NB) - Southbound (SB). In the City of Toronto, bluetooth readers are installed at various locations at different times by different projects. Thus, new routes were created accordingly as more readers are added in new locations. Easy way to create and update routes is [described here](https://github.com/CityofToronto/bdit_data-sources/blob/btdag/bluetooth/update/README.md). 
+
+The `routes` table has the following fields:
+
+|Column|Type|Notes|
+|------|----|-----|
+|`analysis_id`|bigint|analysis_id from the `bluetooth.all_analyses` table. For new routes that are added lately, new analysis_id starting from 1600000 is assigned. _`all_analyses` table has to be updated to include these new routes for data aggregation_.|
+|`name`|varchar|name of the route. This generally contains a detail name explaining the route start and end points. For example, `DVP-J to DVP-I` is a route along Don Valley Parkway between detector **J** and **I**.|  
+|`start_street_name`|varchar|This is the name of the street along which the route is created at the start point of the route.|
+|`start_cross_street`|varchar|The street that crosses the start street at the start point of the route.|
+|`start_reader_id`|varchar|Corresponding reader_id from the reader_locations table at the start point of the route.|
+|`end_street_name`|varchar|At times the route can start and end at different street name thus the name of the street along the route where the route ends.|
+|`end_cross_street`|varchar|This is the name of the street where the route ends.|
+|`end_reader_id`|varchar| Corresponding reader_id from the reader_locations table.|
+|`date_active`|date|The date when the `aggr_5min` table started aggregating data from this reader.|
+|`date_inactive`|date|The date when the reader stopped sending the readings. This field has to be updated manually as either one or both of the readers in a route may temporarily stop aggregating data for few days and come back again.|
+|`date_last_received`|date|Last day data on the route is aggregated. This field is updated everyday by the function [`sql/functions/insert_report_date`](sql/functions/). For the routes that are active, the last reported date will be yesterday.|
+|`geom`|geometry||
+
+#### reader_status_history
+
+This is a table that logs the `last_active_date` for each reader daily. This table is used as a lookup table to identify the readers from which data aggregation did not occur aka `broken_readers` as of yesterday. This table is updated by the function [`sql/functions/reader_status_history`](sql/functions/). The function runs daily in `Airflow`. The function [`sql/functions/broken_readers`](sql/functions/) depends on this look up table to identify the readers that were `not active` yesterday but were `active` the day before as `broken_readers`. It has the following four fields.
+
+|Column|Type|Notes|
+|------|----|-----|
+|`reader_id`|integer|The reader_id unique for readers in each location.|
+|`last_active_date`|date|The latest date when the data from the reader was aggregated.|
+|`active`|boolean|Boolean true or false. If the data_aggregation for the reader occured yesterday, this field is true. Else false. |
+|`dt`|date|This is the same date that is used as the parameter in the function to update the table. As the function runs daily in airflow, this field contains the most recent date.|
+
 #### ClassOfDevice
 
 |Column|Type|Notes|
@@ -167,6 +260,7 @@ substring(cod::bit(24) from 17 for 6) as minor_device_class,
 substring(cod::bit(24) from 12 for 5) as major_device_class
 ```
 
+
 ## 3. Technology
 
 The BlipTrack sensors have two directional Bluetooth antennas and an omnidirectional WiFi antenna.
@@ -228,7 +322,9 @@ The script pulls a day of data for each [analysisID](#the-analysisid) and upload
 Two companion scripts send alerts after this script runs:
 
 - [notify_routes.py](api/notify_routes.py) sends an email if new route configurations appear in the database.
-- [brokenreaders.py](readersdown/) sends an email if a sensor stopped producing data the previous day.
+- [bluetooth_check_readers.py](../dags/bluetooth_check_readers.py) This script runs at 08:00 hrs everyday and sends a slack message in `data_talk` channel if the data pipeline fails as of the day before. If the data pipeline is ok, then the script checks if there are any bluetooth readers which is not sending the data. In case of an occurance of a broken reader, it will then log the list of broken readers in `broken_readers_log` table and also sends a slack message. This script also updates the `date_last_received` field in the `routes` table and `reader_locations` table and add new rows to the `reader_status_history` table.  
+
+
 
 #### Under the Hood
 
@@ -325,3 +421,4 @@ For the [King St. Transit Pilot](toronto.ca/kingstreetpilot), the team has relea
 
 - [King St. Transit Pilot - Detailed Bluetooth Travel Time](https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-catalogue/#739f4e47-737c-1b32-3a0b-45f80e8c2951) contains travel times collected during the King Street Pilot in the same format as the 5-min data set. [Here](sql\analysis\open_data_ksp_travel_times.sql) is the SQL code producing these data.
 - [King St. Transit Pilot – Bluetooth Travel Time Summary](https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-catalogue/#a85f193a-4910-f155-6cb9-49f9dedd1392) contains monthly averages of corridor-level travel times by time periods. [Here](sql\analysis\open_data_ksp_agg_travel_times.sql) is the SQL code producing these summaries.
+