What are the differences between the urllib, urllib2, urllib3 and requests module?

Python's HTTP Clients: urllib, urllib2, urllib3, and Requests Explained

Explore the evolution of Python's HTTP libraries, understanding the differences, capabilities, and best use cases for urllib, urllib2, urllib3, and the popular Requests module.

Making HTTP requests is a fundamental task in many Python applications, from web scraping to API interactions. Over the years, Python's standard library and third-party modules have offered various ways to achieve this. This article delves into the distinctions between urllib, urllib2, urllib3, and the widely-used requests library, helping you choose the right tool for your specific needs.

The Standard Library: urllib and urllib2 (Python 2)

In Python 2, the standard library provided urllib and urllib2 for handling URLs. While both were part of the standard distribution, they served slightly different purposes and had distinct functionalities. Understanding their roles is key to appreciating the subsequent developments.

flowchart TD
    A[Python 2] --> B{HTTP Libraries}
    B --> C[urllib]
    B --> D[urllib2]
    C -- "URL opening, basic auth" --> E[Functionality]
    D -- "More advanced: handlers, cookies, redirects" --> E
    E -- "Complex API, less intuitive" --> F[Developer Experience]

Relationship between urllib and urllib2 in Python 2

urllib (Python 2):

• Core Functionality: Primarily used for opening URLs (similar to open() for local files) and basic authentication.
• Features: URL encoding, parsing, and basic HTTP GET requests.
• Simplicity: Simpler API, but less powerful for complex HTTP operations.

urllib2 (Python 2):

• Core Functionality: Designed for more advanced HTTP requests, offering a richer set of features.
• Features: Supported HTTP PUT/POST, cookie handling, redirects, proxies, and custom handlers (e.g., for authentication, HTTPS).
• Complexity: More powerful but also more complex to use, often requiring the creation of OpenerDirector and Request objects.

Key Takeaway for Python 2: If you needed anything beyond a basic GET request, you'd likely turn to urllib2. However, its API was often considered cumbersome and not very Pythonic.

# Python 2 example using urllib
import urllib

response = urllib.urlopen('http://httpbin.org/get')
print response.read()

# Python 2 example using urllib2 for a POST request
import urllib2
import urllib

url = 'http://httpbin.org/post'
values = {'name': 'John Doe', 'language': 'Python'}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()

Basic HTTP GET with urllib and POST with urllib2 in Python 2

Python 3's Unified urllib and the Rise of urllib3

With Python 3, the urllib and urllib2 modules were refactored and merged into a single, more organized urllib package, which is now split into several sub-modules like urllib.request, urllib.error, urllib.parse, and urllib.robotparser. This brought some improvements but still retained a relatively low-level and verbose API.

Around the same time, urllib3 emerged as a powerful, user-friendly, and feature-rich HTTP client library, initially designed to address the shortcomings of the standard library modules. It's important to note that urllib3 is a third-party library, not part of Python's standard distribution, despite its name similarity to urllib.

urllib (Python 3):

• Core Functionality: The urllib.request module handles opening URLs, making HTTP requests, and basic authentication. urllib.parse handles URL parsing and encoding.
• Features: Supports various protocols (HTTP, HTTPS, FTP), cookie handling, redirects, and proxies. It's the standard way to make HTTP requests without external dependencies.
• Complexity: While improved from Python 2, it still requires more boilerplate code for common tasks compared to higher-level libraries.

urllib3 (Third-party):

• Core Functionality: A robust, production-ready HTTP client library.
• Features: Thread-safe connection pooling, client-side SSL/TLS verification, file uploads with multipart encoding, helpers for retrying failed requests, and support for gzip/deflate encodings.
• Performance & Reliability: Highly optimized for performance and reliability, making it a foundational component for many other HTTP libraries, including requests.
• Usage: Often used directly in applications requiring fine-grained control over HTTP connections or as a dependency for other libraries.

# Python 3 example using urllib.request
import urllib.request
import urllib.parse

url = 'http://httpbin.org/post'
values = {'name': 'Jane Doe', 'language': 'Python3'}
data = urllib.parse.urlencode(values).encode('utf-8') # Data must be bytes
req = urllib.request.Request(url, data=data, method='POST')
with urllib.request.urlopen(req) as response:
    print(response.read().decode('utf-8'))

# Python 3 example using urllib3
import urllib3

http = urllib3.PoolManager()
resp = http.request('GET', 'http://httpbin.org/get')
print(resp.data.decode('utf-8'))

resp = http.request('POST', 'http://httpbin.org/post', fields={'name': 'Alice', 'language': 'urllib3'})
print(resp.data.decode('utf-8'))

HTTP POST with urllib.request and GET/POST with urllib3 in Python 3

The Modern Standard: The Requests Library

The requests library, created by Kenneth Reitz, is a third-party library that has become the de facto standard for making HTTP requests in Python. It builds on top of urllib3 but provides a much more intuitive, human-friendly API, making common HTTP tasks incredibly simple and enjoyable.

requests (Third-party):

• Core Functionality: Simplifies HTTP requests for humans.
• Features: Automatic content decompression, JSON decoding, connection pooling, international domain names and URLs, persistent sessions, file uploads, authentication, and much more.
• Ease of Use: Its API is designed to be straightforward and expressive, reducing boilerplate code significantly.
• Popularity: Widely adopted in the Python community due to its excellent documentation, robust features, and ease of use.
• Underlying Engine: Uses urllib3 for its low-level HTTP functionality, handling connection pooling, retries, and other complexities behind the scenes.

flowchart TD
    A[Developer] --> B[Requests Library]
    B -- "Simple, Pythonic API" --> C[Common HTTP Tasks]
    B -- "Leverages" --> D[urllib3]
    D -- "Handles" --> E[Connection Pooling]
    D -- "Handles" --> F[Retries & Redirects]
    D -- "Handles" --> G[SSL Verification]
    C -- "GET, POST, PUT, DELETE" --> H[API Interactions]
    C -- "File Uploads" --> H
    C -- "JSON Data" --> H
    H -- "Productivity & Readability" --> I[Happy Developer]

How the Requests library simplifies HTTP interactions using urllib3

# Python 3 example using the Requests library
import requests

# GET request
response = requests.get('http://httpbin.org/get')
print(response.json()) # Automatically parses JSON response

# POST request with JSON data
headers = {'Content-Type': 'application/json'}
data = {'name': 'Charlie', 'language': 'Requests'}
response = requests.post('http://httpbin.org/post', json=data, headers=headers)
print(response.json())

# Handling errors
response = requests.get('http://httpbin.org/status/404')
try:
    response.raise_for_status() # Raises HTTPError for bad responses (4xx or 5xx)
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error: {e}")

Making GET and POST requests with the Requests library

Choosing the Right Tool

The choice of HTTP client depends heavily on your Python version, project requirements, and personal preference. Here's a summary to guide your decision:

Feature comparison of Python HTTP libraries

For Python 2 projects (Legacy):

• You'll primarily encounter urllib and urllib2. Use urllib2 for anything beyond basic GET requests.
• Recommendation: If possible, migrate to Python 3. If not, consider backporting requests if your project allows external dependencies.

For Python 3 projects:

• urllib.request: Use if you absolutely cannot add external dependencies, or for very simple, one-off requests where the verbose API is acceptable. It's part of the standard library, so it's always available.
• urllib3: Use if you need fine-grained control over HTTP connections, require high performance, or are building another library that needs a robust low-level HTTP client. It's also a good choice if you want to avoid the requests abstraction layer for specific reasons.
• requests: This is the recommended choice for most Python 3 applications. Its user-friendly API, comprehensive features, and excellent documentation make it the most productive and enjoyable library for making HTTP requests. It handles many complexities (like connection pooling and retries) transparently, thanks to urllib3.