just their smallest dataset looks to be 6 TB _compressed_. not a thing you can r... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		colinsane 3 days ago \| parent \| context \| favorite \| on: An update on residential proxies and the scraper s... just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
		help

binarymax 3 days ago | [–]

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

dylan604 3 days ago | [–]

If you scrape 6TB from across the web or grab it from one place, it's still 6TB

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact