How to Use Proxies in Scrapy (Middleware Tutorial for Web Scraping)

Introduction

If you're using Scrapy for web scraping, adding proxies isn’t optional once you scale.

Without proxies:

• Requests come from a single IP

• Detection increases

• Your crawler gets blocked

Scrapy provides a built-in way to handle this through middleware.

In this tutorial, you’ll learn how to:

• Configure proxies in Scrapy

• Use middleware for request handling

• Rotate proxies

• Improve scraping reliability

How Scrapy Handles Requests

Before adding proxies, understand how Scrapy works.

Basic flow:

1. Spider generates requests

2. Requests pass through middleware

3. Downloader sends the request

4. Response comes back

👉 Middleware is where proxies are applied.

Basic Proxy Setup in Scrapy

You can assign a proxy directly to a request.

Example:

def start_requests(self):
    yield scrapy.Request(
        url="https://httpbin.org/ip",
        meta={"proxy": "http://user:pass@ip:port"}
    )

👉 This routes the request through a proxy.

Using Downloader Middleware

For scalable setups, use middleware instead of hardcoding proxies.

Step 1: Create middleware

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta['proxy'] = "http://user:pass@ip:port"

Step 2: Enable middleware

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ProxyMiddleware': 543,
}

👉 Now all requests go through your proxy.

Rotating Proxies in Scrapy

Using one proxy is not enough.

Example rotation:

import random

class RotatingProxyMiddleware:
    def __init__(self):
        self.proxies = [
            "http://user:pass@ip1:port",
            "http://user:pass@ip2:port",
            "http://user:pass@ip3:port"
        ]

    def process_request(self, request, spider):
        request.meta['proxy'] = random.choice(self.proxies)

👉 Each request uses a different proxy.

Handling Failed Proxies

Some proxies will fail. Handle it properly.

def process_exception(self, request, exception, spider):
    print("Proxy failed:", exception)

👉 You can remove bad proxies or retry requests.

Choosing a Proxy Provider

When working with Scrapy at scale, proxy quality affects request success rates.

Developers often test providers such as Bright Data, Oxylabs, NetNut, Decodo, and Squid Proxies to evaluate how proxies behave under load.

Different providers offer combinations of datacenter and residential proxies, which can impact request consistency depending on the target.

Integrating Proxies into Scrapy Workflows

In Scrapy-based systems, proxy integration should be handled centrally through middleware.

For example, Squid Proxies can be used within middleware configurations to assign proxies dynamically across requests.

This approach keeps proxy logic separate from spider logic, making it easier to maintain and scale scraping pipelines.

Best Practices

✅ Use middleware for proxy management

✅ Rotate proxies across requests

✅ Monitor failed requests

✅ Avoid hardcoding proxies in spiders

Common Mistakes

❌ Using a single proxy

❌ Not handling failed proxies

❌ Mixing proxy logic inside spiders

❌ Ignoring request errors