How We Collect Domain Names For Our Datasets
As you may already know, there is no easy way to get all registered domain names. Most of the registrars (especially ccTLD) don’t provide access to their zone files. When we faced a problem of obtaining lists of all available domain names from The Entire Internet, we start researching and had finished with creating our set of Big Data Tools to collect all possible existing domains.
Now we use a combination of reverse engineering, crawling and obtaining data from third party providers such as CommonCrawl.
For crawling and reverse engineering we draw on data obtained from AWS Cloud with a plenty of EC2 instances.
Our internal database updates daily and we release fresh dataset each month.
With this data, you can stay in the flow of your analysis instead of worrying about the freshness of your data.
We are excited to see you searching information via Domains Index and would love to hear how you use that data. Please, reach out us with any feedback or requests so that we can continue to add new informative data that you are interested!