Python HTTP Request Libraries: From requests to httpx

Introduction

If any Python library has changed the barrier to entry for internet development, requests is definitely on the list.

With a minimalist API, it put an end to the chaotic era of urllib2.

To this day, requests.get(url) is still the first line of crawler code many programmers write instinctively.

But with the full explosion of Python's async ecosystem (Asyncio), and the widespread adoption of the HTTP/2 protocol, this veteran born in 2011 has begun to show its age.

PS: Why still study new technologies yourself in the AI era?

Because AI's underlying nature is the "past."

If you don't master advanced tools, AI will use its "experiential inertia" to keep you in the old era.

Only when you understand new things can you direct AI to write more modern code.

The essence of AI is a "probabilistic model"; it tends to give the most common, most conservative solutions from its training data (which are often the old solutions)

HttpX

httpx is not trying to overthrow requests, but to complete its modernization. Its core slogan is: "Almost fully compatible with the Requests API, but with native async support."

Reasons to switch to httpx for new projects:

• Native async support: When handling thousands upon thousands of API calls, synchronous requests will make your program "die" while waiting on network IO. With async/await, httpx can concurrently process extremely high-density requests within a single second, without consuming extra thread resources.
• HTTP/2 support: Compared to HTTP/1.1, HTTP/2 supports multiplexing. This means multiple requests can be sent in parallel on the same connection, significantly reducing latency.
• Strict timeout management: requests never times out by default, which is extremely dangerous in production. httpx enables timeout checks by default, better matching the robustness requirements of modern microservices.

Examples

If you're familiar with requests, the cost of migrating to httpx is almost zero.

Sync style comparison (almost identical)

# Requests
import requests
resp = requests.get("https://api.github.com/events")

# HTTPX
import httpx
resp = httpx.get("https://api.github.com/events")

Async style

import httpx
import asyncio

async def fetch_tech_news():
    async with httpx.AsyncClient() as client:
        # 同时发起多个请求，效率起飞
        tasks = [client.get(f"https://news.com/api/{i}") for i in range(10)]
        responses = await asyncio.gather(*tasks)
        return [r.json() for r in responses]

asyncio.run(fetch_tech_news())

Comparison

Summary

Async is great, but there are some pitfalls:

• httpx's AsyncClient is best used with an async with context manager, otherwise it's easy to cause connection leaks.
• Extra care is needed when using the async loop on Windows (e.g. the limitations of SelectorEventLoop)