It is a simple web crawler tool which takes initial urls(Url Seed), crawls all the pages of the same domain and finds outs the dead-links and stale-links across the website.
While using the documentation SDK Tools at https://developer.amazon.com, developers often end up at dead-links. These dead-links are links which might point towards repositories which might have been moved somewhere else. Therefore, Dex-Crawler stores all those links so that they can be removed.