Merge pull request #152 from m-appel/148-ooni-scrubbed-ips

Rework OONI crawlers

ACKNOWLEDGMENTS.md

@@ -111,6 +111,15 @@ We use the [extended allocation and assignment
 reports](https://www.nro.net/about/rirs/statistics/) provided by the [Number Resource
 Organization](https://www.nro.net/).
 
+## Open Observatory of Network Interference
+
+We use [Internet censorship measurements](https://explorer.ooni.org/) provided by the
+[Open Observatory of Network Interference](https://ooni.org/).
+
+This data is licensed under [CC BY-NC-SA
+4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). The data is aggregated for
+display in the graph.
+
 ## OpenINTEL
 
 We use several datasets from [OpenINTEL](https://www.openintel.nl/), a joint project of

documentation/data-sources.md

@@ -22,11 +22,11 @@
 | Cisco                       | Umbrella Popularity List                     | https://s3-us-west-1.amazonaws.com/umbrella-static/index.html   |
 | Citizen Lab                 | URL testing lists                            | https://github.com/citizenlab/test-lists                        |
 | Cloudflare                  | Cloudflare Radar API endpoints radar/dns/top/ases, radar/dns/top/locations, radar/ranking/top, radar/datasets   | https://radar.cloudflare.com                                   |
-|                             |                                   |
 | Emile Aben                  | AS names                                     | https://github.com/emileaben/asnames                                 |
 | IHR                         | Country Dependency, AS Hegemony, ROV         | https://ihr.iijlab.net                                               |
 | Internet Intelligence Lab   | AS to Organization Mapping                   | https://github.com/InetIntel/Dataset-AS-to-Organization-Mapping      |
 | NRO                         | Extended allocation and assignment reports   | https://www.nro.net/about/rirs/statistics                            |
+| OONI                        | Internet censorship measurements             | https://ooni.org/                                               |
 | OpenINTEL                   | tranco1m, umbrella1m, ns                     | https://data.openintel.nl/data                                       |
 |                             | DNS Dependency Graph                         | https://dnsgraph.dacs.utwente.nl                                     |
 | Packet Clearing House       | Daily routing snapshots                      | https://www.pch.net/resources/Routing_Data                          |

documentation/node-types.md

@@ -25,6 +25,7 @@
 | PeeringdbOrgID          | Unique identifier for an Organization as assigned by PeeringDB.                                                                   |
 | Prefix                  | An IPv4 or IPv6 prefix uniquely identified by the **prefix** property. The **af** property (address family) provides the IP version of the prefix.|
 | Ranking                 | Represent a specific ranking of Internet resources (e.g. CAIDA's ASRank or Tranco ranking). The rank value for each resource is given by the RANK relationship.|
+| Resolver                | An additional label added to IP nodes if they are a DNS resolver. |
 | Tag                     | The output of a classification. A tag can be the result of a manual or automated classification. Uniquely identified by the **label** property.|
 | URL                     | The full URL for an Internet resource, uniquely identified by the **url** property.                                               |
 

documentation/relationship-types.md

@@ -8,6 +8,7 @@
 | ASSIGNED                   | Represent the allocation by a RIR of a network resource (AS, Prefix) to a resource holder (see OpaqueID). Or represent the assigned IP address of an AtlasProbe.                                                 |
 | AVAILABLE                  | Relate ASes and Prefixes to RIRs (in the form of an OpaqueID) meaning that the resource is not allocated and available at the related RIR.                                                                       |
 | CATEGORIZED                | Relate a network resource (AS, Prefix, URL) to a Tag, meaning that the resource has been classified accordingly to the Tag. The **reference_name** property provide the name of the original dataset/classifier. |
+| CENSORED                   | Relate ASes to an OONI censorship test in form of a Tag or a network resource (IP, URL), meaning that there exists a censorship test result from a probe in this AS to the connected node. The **reference_name** property provides the name of the test.|
 | COUNTRY                    | Relate any node to its corresponding country. This relation may have different meaning depending on the original dataset (e.g. geo-location or registration).                                                    |
 | DEPENDS_ON                 | Relate an AS or Prefix to an AS, meaning the reachability of the AS/Prefix depends on a certain AS.                                                                                                              |
 | EXTERNAL_ID                | Relate a node to an identifier commonly used by an organization. For example, PeeringDB assigns unique identifiers to IXPs (see PeeringdbIXID).                                                                  |