Passive and Active Internet Measurements

This page contains datasets and tools that can aid in Internet Measurements research (esp. those focusing on security). Feel free to email me other links/datasets that you think would be useful.

Passive measurement datasets

• Twitter: Random tweets API (and relevant operators), Archive repo of tweets, Twitter Trends.
• Website reputation: URLVoid and URLhaus.
• Web and Virus databases: VirusTotal, Google Safe Browsing, and IBM X-Force.
• Top websites:
  • Lists: Alexa's top 1M sites, Tranco, Majestic, Open PageRank Initiative, Statvoo, Cisco's umbrella (see also: toplists).
    • Note: Up-to-date one-stop data for some of the above services is available here
  • Web scrapping tools: Snallygaster, Scrapy
• Network measurements:
  • Delay measurements: RIPE Atlas, Caida Ark, ICMP datasets.
  • Routing tables: Oregon Route Views.
  • IP geolocation: RIPE IPmap, ipinfo, Maxmind GeoLite2.
  • Network packet captures: Anonymized Internet Traces Dataset (April 2008 - January 2019) (from CAIDA).
  • Internet infrastructure (health and attacks): DownDetector, Daily DDoS attacks, BGP outtages, CAIDA's Internet Outage Detection and Analysis (IODA) platform
• Device search engines: Shodan, Censys, ZoomEye, Project Sonar (Rapid7), thingful.
• Certificates:
  • CAs DB
  • CT logs: Google Transparency Report and crt.sh (monitored logs). A universal list of CT logs is here.
  • Intermediate certs databse.
• DNS: OpenIntel, zone files of generic TLDs from ICANN, Verisign Top-Level Domain Zone File Information, Forward/Reverse DNS (from Sonar).
  • DNSSEC: Stats (from Fraunhofer).
• Firefox Telemetry: Project, Data publishing.
• Stratosphere Lab datasets (IoT, Wifi, malware, botnet pcaps).
• Phishing: PhishTank (phishing domains), and per country reports from SecureList.
• Adblock lists: easylist.
• Web Censorship: Citizen Lab (censored websites).
• Content measurements:
  • Hate: Hatebase, Weaponized Word.
  • Images: Pinterest images.
  • Memes.
  • Common crawl datasets (massive datasets of crawled websites).
• App reviews: Android
• MISC: Internetwache.

Active measurement tools

• Distributed proxy servers: Luminati (now Bright data), ProxyRack, Infatica, hola.
• Distributed testbeds: PlanetLab
• Selenium browser
• M-Lab tools
• Looking Glass

Research artifacts

• A Catalog of Research Artifacts for Computer Science: FindResearch.