Python requests - print entire http request (raw)?

Inspecting Raw HTTP Requests with Python's Requests Library

Learn how to capture and print the complete raw HTTP request sent by Python's requests library, including headers, body, and method, for debugging and analysis.

When working with HTTP requests in Python, especially for debugging or understanding how a web service interacts, it's often crucial to see the exact raw HTTP request being sent over the wire. The popular requests library simplifies HTTP interactions, but by default, it doesn't expose the raw request in an easily accessible format. This article will guide you through various methods to inspect and print the entire raw HTTP request generated by requests, from simple workarounds to more advanced techniques involving custom adapters and network sniffing.

Understanding the Challenge

The requests library abstracts away much of the underlying HTTP communication for convenience. While this is generally beneficial, it means that the raw HTTP message (e.g., GET /path HTTP/1.1\r\nHost: example.com\r\n...) isn't directly available as a property of the requests.Request or requests.Response objects. The requests.Request object holds the components of the request (method, URL, headers, body), but not the serialized raw string that gets sent.

sequenceDiagram
    participant User
    participant PythonScript
    participant RequestsLib
    participant HTTPAdapter
    participant Socket
    participant WebServer

    User->>PythonScript: `requests.get(url)`
    PythonScript->>RequestsLib: Create `Request` object
    RequestsLib->>HTTPAdapter: Prepare `Request` for sending
    Note over RequestsLib,HTTPAdapter: (Serialization happens here)
    HTTPAdapter->>Socket: Send raw HTTP bytes
    Socket->>WebServer: Raw HTTP Request
    WebServer-->>Socket: Raw HTTP Response
    Socket-->>HTTPAdapter: Receive raw HTTP bytes
    HTTPAdapter-->>RequestsLib: Parse raw HTTP bytes into `Response` object
    RequestsLib-->>PythonScript: Return `Response` object
    PythonScript-->>User: Process `Response`

Simplified sequence of an HTTP request with Python's requests library, highlighting where serialization occurs.

Method 1: Using requests.Request.prepare() and curl_cmd

The requests.Request object has a prepare() method that returns a PreparedRequest object. This PreparedRequest object is what requests actually sends. While it still doesn't give you the raw string directly, you can reconstruct a good approximation, especially for debugging. The curl_cmd property (available in some versions or via a custom helper) can also be very useful.

import requests

def print_raw_request(req):
    print('{}\n{}\n\n{}\n'.format(
        req.method + ' ' + req.url,
        '\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
        req.body.decode('utf-8') if req.body else ''
    ))

# Example GET request
req = requests.Request('GET', 'https://httpbin.org/get', headers={'User-Agent': 'MyCustomAgent/1.0'})
prepared_req = req.prepare()
print("--- Prepared GET Request ---")
print_raw_request(prepared_req)

# Example POST request with data
post_data = {'key': 'value', 'another': 'field'}
req_post = requests.Request('POST', 'https://httpbin.org/post', data=post_data)
prepared_req_post = req_post.prepare()
print("\n--- Prepared POST Request ---")
print_raw_request(prepared_req_post)

# Note: This doesn't include HTTP version or CRLF endings, but is close enough for many debugging scenarios.

Reconstructing a raw HTTP request from a PreparedRequest object.

Method 2: Using a Custom HTTPAdapter for Deeper Inspection

For a truly raw view, you need to intercept the request just before it's sent over the socket. This can be achieved by subclassing requests.adapters.HTTPAdapter and overriding its send method. This method receives the PreparedRequest object and is responsible for sending it. Here, you can access the raw bytes that requests is about to send.

import requests
from requests.adapters import HTTPAdapter
from requests.sessions import Session

class RawRequestAdapter(HTTPAdapter):
    def send(self, request, **kwargs):
        # Reconstruct the raw request string
        # This is a simplified reconstruction and might not be 100% identical
        # to what's sent over the wire due to internal socket handling,
        # but it's very close.
        raw_request_lines = []
        raw_request_lines.append(f"{request.method} {request.path_url} HTTP/1.1")
        for header, value in request.headers.items():
            raw_request_lines.append(f"{header}: {value}")
        raw_request_lines.append("") # Empty line separates headers from body
        if request.body:
            if isinstance(request.body, bytes):
                raw_request_lines.append(request.body.decode('utf-8', errors='ignore'))
            else:
                raw_request_lines.append(request.body)

        raw_request_string = '\r\n'.join(raw_request_lines)
        print("\n--- RAW HTTP Request (via Custom Adapter) ---")
        print(raw_request_string)
        print("---------------------------------------------")

        return super().send(request, **kwargs)

# Create a session and mount the custom adapter
s = Session()
s.mount('http://', RawRequestAdapter())
s.mount('https://', RawRequestAdapter())

# Now make requests using this session
print("Making a GET request...")
s.get('https://httpbin.org/get', headers={'X-Custom-Header': 'AdapterTest'})

print("\nMaking a POST request...")
s.post('https://httpbin.org/post', json={'data': 'hello world'})

Using a custom HTTPAdapter to intercept and print the raw HTTP request before sending.

Method 3: Leveraging Logging for Debugging

The requests library uses the standard Python logging module. You can configure logging to output debug information, which often includes details about the request being prepared and sent. This is less about the 'raw' string and more about the components, but it's a quick way to get verbose output without modifying your request logic.

import logging
import requests

# Configure logging to show HTTP debug information
logging.basicConfig(level=logging.DEBUG)

# Suppress urllib3 warnings if they are too noisy
logging.getLogger("urllib3").setLevel(logging.WARNING)

print("--- Logging Debug Output ---")
requests.get('https://httpbin.org/get', params={'param1': 'value1'})

print("\n--- Logging Debug Output (POST) ---")
requests.post('https://httpbin.org/post', data={'field': 'data'})

Enabling debug logging for requests to see request details.

Method 4: Network Sniffing (External Tool)

For the absolute most accurate raw HTTP request, a network sniffer like Wireshark or tcpdump is the definitive tool. These tools capture packets directly from your network interface, allowing you to see exactly what bytes are sent and received, including the full HTTP protocol details, TCP/IP headers, and SSL/TLS handshake (if applicable). This method is external to Python but provides the ground truth.

Network sniffing with tools like Wireshark captures the true raw HTTP request.

To use a network sniffer:

1. Install a sniffer: Wireshark (GUI) or tcpdump (command-line) are popular choices.
2. Start capturing: Begin capturing traffic on the network interface your Python script uses.
3. Run your Python script: Execute the requests call.
4. Stop capturing: Halt the sniffer.
5. Filter results: Apply filters (e.g., http or tcp port 80 / tcp port 443) to find your HTTP request.

This method is invaluable for diagnosing complex network issues or verifying exact protocol compliance.