A definitive guide to implementing, using, and understanding all aspects of RDAP by Andy Newton.
The Registration Data Access Protocol (RDAP) is the successor protocol to the Whois protocol. It was first ratified by the Internet Engineering Task Force (IETF) in March 2015 by their WEIRDS working group, and initial server and client implementations were released shortly thereafter by the many Regional Internet Registries (RIRs) in June 2015.
In the years since RDAP became standardized, extensions have been added and profiles have been specified. While this is a clear sign of the success of the protocol, the amount of information spread across RFCs, IANA registries and other documents makes specification information more difficult to acquire and implementations harder to develop and deploy.
This book is intended to describe RDAP a in way the RFCs do not, and in many cases cannot describe the protocol and its ecosystem through the use of mdbook, the many mdbook plugins, annotated examples, easier to read language and references to other materials.
The Registration Data Access Protocol (RDAP) is the successor to WHOIS. Like WHOIS, RDAP provides access to information about Internet resources (domain names, autonomous system numbers, and IP addresses). Unlike WHOIS, RDAP provides:
- A machine-readable representation of registration data;
- Differentiated access;
- Structured request and response semantics;
- Internationalisation;
- Extensibility.
RDAP.org aims to support users and developers of RDAP clients by providing a "bootstrap server", i.e. single end point for RDAP queries. RDAP.org aggregates information about all known RDAP servers. RDAP clients can send RDAP queries to RDAP.org, which will then redirect requests to the appropriate RDAP service.
This is a database of Internet places. Mostly domains. Sometimes other things. Think of it as Internet meta database. This repository contains link metadata: title, description, publish date, etc.
The entire Internt is in one file! Just unzip internet.zip!
FBI Watchdog is a threat intelligence tool that monitors domain DNS changes in real-time, specifically detecting law enforcement seizures (ns1.fbi.seized.gov and ns2.fbi.seized.gov). It alerts users via Telegram and Discord and captures screenshots of seized domains.
Only alerts over Telegram or Discord, though.
Sublist3r is a python tool designed to enumerate subdomains of websites using OSINT. It helps penetration testers and bug hunters collect and gather subdomains for the domain they are targeting. Sublist3r enumerates subdomains using many search engines such as Google, Yahoo, Bing, Baidu and Ask. Sublist3r also enumerates subdomains using Netcraft, Virustotal, ThreatCrowd, DNSdumpster and ReverseDNS.
subbrute was integrated with Sublist3r to increase the possibility of finding more subdomains using bruteforce with an improved wordlist. The credit goes to TheRook who is the author of subbrute.
Subdomain Finder is a scanner that scans an entire domain to find as many subdomains as possible.
The RapidBlock Project is a grassroots initiative to make Fediverse domain blocking more effective through collective action.
Moderation on the Fediverse is unevenly distributed. Some instance admins devotedly follow the #FediBlock hashtag, blocking abusive servers within hours of their first appearance on the network. Others wait until their own users file a report. Still others do nothing at all.
This uneven distribution of moderation allows abusive instances to do significant psychological harm. Abusive instances are a fast-moving target; setting up a new Mastodon instance takes only an hour or two, as does resetting an instance to give it a new domain name. This gives abusers a substantial time window in which there are a lot of available victims to target.
The RapidBlock Project is something different: humans are in the loop at every step of the decision-making process, and the only thing that is automated is the actual propagation of the decisions. Moderation is hard, especially good moderation. Moderation is a full-time job, and many Fediverse admins aren't taking up that mantle of responsibility. We are trying to build a central moderation team with a clear, published rationale for our blocking criteria and a clear dispute process for remediating mistaken blocks.
A collection of several hundred online tools for OSINT.
BBOT is a recursive, modular OSINT framework written in Python. It is capable of executing the entire OSINT process for entire domains in a single command, including subdomain enumeration, port scanning, web screenshots (with its gowitness module), vulnerability scanning (with nuclei), and much more.
BBOT currently has over 50 modules and counting.
Requires Python v3.9.x or later.
A blocklist for QAnon, conspiracy, fake news, nazi websites for multiple applications, including web browser adblockers, DNSes, and even /etc/hosts. It looks like the lists (which are substantially identical in content) could be used to compile a database of known-bad domains. IPv4 and IPv6 supported.
A site that has all of the TLDs (traditional as well as kitchy new-school), what registrars you can get them through, and how much you can expect to pay at each of them so you can shop around.
Transparent domain information, from AAAA records to WHOIS. Free DNS record, IP address hostname, and WHOIS lookups.
A directory of tools for domain, network, and PII reconaissance. Includes some Google searching tricks.
Intelligence X differentiates itself from other search engines in these unique ways:
The search works with selectors, i.e. specific search terms such as email addresses, domains, URLs, IPs, CIDRs, Bitcoin addresses, IPFS hashes, etc.
It searches in places such as the darknet, document sharing platforms, whois data, public data leaks and others.
It keeps a historical data archive of results, similar to how the Wayback Machine from archive.org stores historical copies of websites.
You can use Intelligence X to perform any kind of open source intelligence. We deliver fast, high-quality results and make the deepest parts of the internet accessible with a few clicks. Intelligence X searches billions of selectors in a matter of milliseconds. Combined with our data archive this is a powerful new tool.
To get interesting data you have to sign up for an account.
They host e-mail and provide IMAP, POP3, and webmail. No storage limits. Pay by the e-mail address. MFA. Collects as little user data as they can because they don't want to secure it. Hosted on AWS. Fairly decent privacy policy and procedures. Everything is encrypted where it won't break the user experience.
A repository for monitoring attack vectors mentioned in the billion-dollar disinformation campaign to reelect the president in 2020. Includes some Python code for analyzing the data.
A Python module that implements a WHOIS client. Requires the futures module as a dependency. Returns what it finds as a Python data structure.
An organization of hobbyists who run an alternative DNS network, also provides access to domains not administered by ICANN.
A curated blocklist of known fake news sites, suitable for use with adblockers or other countermeasures. Still updated fairly frequently.
The list itself, suitable for adding to a Pi-Hole or adblocking addon: https://raw.githubusercontent.com/StevenBlack/hosts/master/extensions/fakenews/hosts