How We Collect Domain Names For Our Datasets

How We Collect Domain Names For Our Datasets

As you may already know, there is no easy way to get all registered domain names. Most of the registrars (especially ccTLD) don’t provide access to their zone files. When we faced a problem of obtaining lists of all available domain names from The Entire Internet, we start researching and had finished with creating our set of Big Data Tools to collect all possible¬†existing¬†domains.

Now we use a combination of reverse engineering, crawling and obtaining data from third party providers such as CommonCrawl.

Raw data flow from sources to domain names database

For crawling and reverse engineering we draw on data obtained from AWS Cloud with a plenty of EC2 instances.

Hundreds of instances working with data queue


Our internal database updates daily and we release fresh dataset each month.

With this data, you can stay in the flow of your analysis instead of worrying about the freshness of your data.

We are excited to see you searching information via Domains Index and would love to hear how you use that data. Please, reach out us with any feedback or requests so that we can continue to add new informative data that you are interested!

I definitely like to work with big data projects and SEO analytics.