DNS remodeling #119
Merged
DNS remodeling #119
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…be run after openintel.umbrella1m
|
I have done most of the changes we discussed. Only things left are the cloudflare crawlers. |
|
Forgot to lower umbrella in the config (readme should also be updated) |
|
I have adjusted all crawlers for the new Still left: add |
- Batch create new nodes (happens more often than expected) - Add logging output - Do not use builtins as variable names
We could not decide on a name, so we will deal with this later.
|
I did a full run and got two errors:
These are completely unrelated to the modifications so I think we can now merge this PR. @m-appel are you planning any other changes here? |
m-appel
approved these changes
Feb 7, 2024
|
No, I think we can merge it (actually squash and merge, since it's many commits). I had the MANRS problem yesterday as well, but worked on rerun, I think they had some problems. |
MAVRICK-1
added a commit
to MAVRICK-1/internet-yellow-pages
that referenced
this pull request
Feb 7, 2024
…operty DNS remodeling (InternetHealthReport#119) * update url2domain to url2hostname * remove iana root zone file and dns hierarchy from config file * Atlas measurement targets are now hostnames * update openintel crawlers to the new DNS model * umbrella now ranks a mix of DomainName and HostName nodes and should be run after openintel.umbrella1m * Add explanation for cloudflare DNS modeling * lower umbrella crawler in config file * update READMEs with the new DNS modeling * add (:Service {name:'DNS'}) node and link it to authoritative name servers * Nodes do not have reference properties * Normalize IPv6 addresses * Fix wrong crawler name * Typos and formatting * Remove infra_mx crawler since it does not do anything at the moment * Update Cisco Umbrella crawler - Batch create new nodes (happens more often than expected) - Add logging output - Do not use builtins as variable names * Remove redundant set and parameters * Remove Service node for now We could not decide on a name, so we will deal with this later. --------- Co-authored-by: Malte Tashiro <[email protected]> Add OpenINTEL DNS dependency crawler Integrate with existing files and remove some unnecessary stuff. Co-authored-by: Raffaele Sommese <[email protected]> precommit error rectified Update __init__.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remodeling of DNS data
Description
After multiple discussions we decided to improve the way we model DNS data. The main changes include de-coupling the DomainName and HostName nodes.
Types of changes
Checklist: