Passive and Active Internet Measurements

This page contains datasets and tools that can aid in Internet Measurements research (esp. those focusing on security). Feel free to email me other links/datasets that you think would be useful.

Passive measurement datasets

• Twitter: Random tweets API (and relevant operators), Archive repo of tweets, Twitter Trends.
• Website reputation: URLVoid and URLhaus.
• Web and Virus databases: VirusTotal, Google Safe Browsing, and IBM X-Force.
• Top websites:
  • Lists: Alexa's top 1M sites, Tranco, Majestic, Open PageRank Initiative, Statvoo, Cisco's umbrella (see also: toplists).
    • Note: Up-to-date one-stop data for some of the above services is available here
  • Web scrapping tools: Snallygaster, Scrapy
• Network measurements:
  • Delay measurements: RIPE Atlas, Caida Ark, ICMP datasets.
  • IP geolocation: RIPE IPmap, ipinfo, Maxmind GeoLite2.
  • Network packet captures: Anonymized Internet Traces Dataset (April 2008 - January 2019) (from CAIDA).
  • Internet infrastructure:
    • AS costs.
    • BGP: route announcements (from Oregon Route Views).
    • From CAIDA: AS peering (monthly, since Jan 2016), prefix to AS mapping (daily, since Mar 2005).
    • Health and outages: DownDetector, Daily DDoS attacks, BGP outtages, CAIDA's Internet Outage Detection and Analysis (IODA) platform
• Device search engines: Shodan, Censys, ZoomEye, Project Sonar (Rapid7), thingful.
• Certificates:
  • CAs DB
  • CT logs: Google Transparency Report and crt.sh (monitored logs). A universal list of CT logs is here.
  • Intermediate certs databse.
• Domains, Registrars, and DNS:
  • OpenIntel: ALL second-level DNS records of .com, .net and .org zones.
  • Centralized Zone Data Service (CZDS) from ICANN: zone files of TLDs, and you can request all files here (free account creation required).
  • Verisign: Top-Level Domain Zone File Information.
  • Project Sonar: Forward/Reverse DNS.
  • RIPE Atlas: querying of DNS root servers.
  • DNSSEC: Stats (from Fraunhofer).
• Firefox Telemetry: Project, Data publishing.
• Stratosphere Lab datasets (IoT, Wifi, malware, botnet pcaps).
• Phishing: PhishTank (phishing domains), and per country reports from SecureList.
• Adblock lists: easylist.
• Web Censorship: Citizen Lab (censored websites), Open Observatory of Network Interference (OONI) (data and explore).
• Content measurements:
  • Hate: Hatebase, Weaponized Word.
  • Images: Pinterest images.
  • Memes.
  • Common crawl datasets (massive datasets of crawled websites).
• App reviews: Android
• MISC: Internetwache.

Active measurement tools

• Distributed proxy servers: Luminati (now Bright data), ProxyRack, Infatica, hola.
• Distributed testbeds: PlanetLab
• Selenium browser
• M-Lab tools
• Looking Glass

Research artifacts

• A Catalog of Research Artifacts for Computer Science: FindResearch.