Skip to content
← Rift // HomeUC—02 / SCRAPE

§ Use Case — Scraping

Web scraping at
industrial scale.

Datacenter SOCKS5 + HTTP/HTTPS CONNECT, unlimited concurrency, sub-second latency. Built for high-volume crawls of catalogs, public APIs and mid-defence targets — one gateway, one auth, every endpoint your stack needs, from $0.50/GB.

§ Pillars

Why operators run this stack on Rift.

01

Two SOCKS5 pools

Budget aggregated pool for cheap bulk crawls, or the curated datacenter pool for cleaner IPs and higher uptime — same endpoint, switch per job.

02

Drop-in protocols

SOCKS5, HTTP and HTTPS CONNECT endpoints. Works out of the box with Puppeteer, Playwright, Scrapy, Selenium, cURL.

03

Unmetered concurrency

No connection caps. Spin up thousands of parallel workers without throttling — pay only for the bandwidth you actually use.

Recommended product

SOCKS5 Premium

SOCKS5 Budget + Premium

For sustained crawls, the curated datacenter pool delivers cleaner IPs and higher uptime. Testing or running cost-sensitive bulk jobs on low-defence targets? Start on SOCKS5 Budget at $0.50/GB and your first 1 GB is free.

  • SOCKS5 + HTTP/HTTPS CONNECT
  • Sticky sessions via username token
  • Zero-log routing
  • Auto dead-IP cooldown
Configure SOCKS5 Premium

Drop-in snippet

PYTHON
# Scrapy settings.py — Rift SOCKS5/HTTP gateway
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 100,
}

import os
os.environ["http_proxy"]  = "http://user-session-job1:[email protected]:8080"
os.environ["https_proxy"] = "http://user-session-job1:[email protected]:8080"

Works with Puppeteer, Playwright, Selenium, Scrapy, cURL and every commercial scraping platform.

§ Decoded

Niche questions.

Yes. Standard SOCKS5, HTTP and HTTPS CONNECT endpoints. Set the proxy URL with your credentials and you're live — no SDK, no custom client.

High-volume crawls of sites without heavy bot defences — open catalogs, public APIs, sitemaps, price feeds, RSS, documentation and most mid-defence targets. For sites running aggressive bot management (Cloudflare, DataDome) a residential network is a better fit; SOCKS5 datacenter trades stealth for speed and cost.

Yes. Append a session token to your username and you'll hold the same exit IP for the session — ideal for pagination and multi-step flows. Drop the token for per-request rotation across the pool.

Fully. We expose standard SOCKS5 and HTTP proxy endpoints that any HTTP client or browser automation tool can consume. Examples in our docs for Puppeteer, Playwright, Selenium and Scrapy.

§ The scraping playbook

How teams scrape the open web on Rift

Most scraping work isn't a fight with sophisticated bot management — it's bulk collection from catalogs, public APIs, sitemaps, price feeds and mid-defence sites where throughput and cost matter more than stealth. Rift gives crawl engineers a single gateway with two SOCKS5 datacenter pools behind it: a budget aggregated pool for cheap, high-volume jobs and a curated datacenter pool for cleaner IPs and higher uptime — both on the same SOCKS5 + HTTP/HTTPS CONNECT endpoint, so the same code base switches pools by changing one credential.

Picking the right pool for the target

Reach for the budget pool when the target is a static catalog, an open API, a sitemap walk, an RSS or price feed, or any workload where speed and cost beat stealth. Move to the premium curated pool when you need cleaner IP reputation and steadier uptime for long-running production crawls. Both pools are datacenter SOCKS5 — they're a poor fit for sites running aggressive bot management (Cloudflare Bot Management, DataDome, PerimeterX), which fingerprint and rate-limit datacenter ASNs hard. For those targets a residential network is the right tool; Rift focuses on doing high-volume datacenter SOCKS5 well and cheaply rather than pretending datacenter beats DataDome.

Rotation, sticky sessions and concurrency

Two rotation modes cover almost every workload. Per-request rotation gives every outbound call a fresh IP from the pool — ideal for breadth-first crawls, price monitoring and any job where each request is independent. Sticky sessions lock an exit IP by appending a session token to the proxy username — required for paginated tracking, multi-step flows and anything that depends on a stable cookie jar. Concurrency is unmetered: spin up thousands of parallel workers in Scrapy, Playwright clusters or your own Go/Rust scrapers and only pay for the bandwidth you actually consume. Dead exits are cooled down automatically so a flaky IP never stalls your crawl.

Stack compatibility — Scrapy, Playwright, Puppeteer, Selenium

Rift exposes plain SOCKS5, HTTP and HTTPS CONNECT endpoints, so any language with an HTTP client works without an SDK. Scrapy users wire the gateway into HttpProxyMiddleware; Playwright and Puppeteer users pass it via the --proxy-server launch flag or per-context proxy option; Selenium users configure it through seleniumwire or chrome options; requests, httpx, aiohttp, node-fetch, got and axios all accept it as a standard proxy URL. Session stickiness is encoded in the username — for example user-session-job42 — so holding or dropping an IP never requires a config reload.

Pricing, free trial and acceptable use

SOCKS5 Budget runs at $0.50/GB and SOCKS5 Premium at $1.00/GB, pay-as-you-go with no subscription — and every new account gets 1 GB free to test the network before paying. Rift keeps zero request logs (only aggregate bandwidth counters for billing), supports IP whitelisting and revocable per-key credentials from the console, and operates under a clear acceptable-use policy that prohibits scraping personal data, CSAM, account takeover, and any activity that violates a target site's lawful terms. For committed-use volume, contact [email protected] for discounted per-TB pricing.

Open the rift. Tonight.

Spin up a key in 60 seconds. Pay-as-you-go, no commitment, no logs.