Correct way to make a Python HTTPS request using requests module?

Mastering HTTPS Requests in Python with the Requests Library

Learn the correct and secure way to make HTTPS requests in Python using the popular requests library, covering certificate verification, custom headers, and error handling.

Making HTTP requests is a fundamental task in many Python applications, from web scraping to interacting with APIs. When dealing with sensitive data or ensuring data integrity, using HTTPS is paramount. The requests library is the de facto standard for making HTTP requests in Python, offering a user-friendly API and robust features for secure communication. This article will guide you through the best practices for making HTTPS requests, ensuring your applications are both functional and secure.

The Basics: Making a Secure GET Request

The requests library automatically handles many aspects of secure communication, including SSL/TLS certificate verification, by default. This means that for most standard HTTPS endpoints, a simple requests.get() call is sufficient to establish a secure connection and verify the server's identity. If the certificate verification fails, requests will raise an SSLError, preventing potential man-in-the-middle attacks.

import requests

try:
    response = requests.get('https://api.github.com/events')
    response.raise_for_status()  # Raise an exception for HTTP errors (4xx or 5xx)
    print(f"Status Code: {response.status_code}")
    print(response.json()[:1]) # Print first item of JSON response
except requests.exceptions.HTTPError as errh:
    print(f"Http Error: {errh}")
except requests.exceptions.ConnectionError as errc:
    print(f"Error Connecting: {errc}")
except requests.exceptions.Timeout as errt:
    print(f"Timeout Error: {errt}")
except requests.exceptions.RequestException as err:
    print(f"Something went wrong: {err}")

Basic secure GET request with error handling.

Understanding Certificate Verification

Certificate verification is a critical security feature. When you make an HTTPS request, the server presents a digital certificate. requests verifies this certificate against a bundle of trusted Certificate Authorities (CAs) that it ships with. If the certificate is invalid, expired, or issued by an untrusted CA, requests will refuse the connection. While it's possible to disable this verification, it's strongly discouraged in production environments as it opens your application to security vulnerabilities.

sequenceDiagram
    participant Client
    participant Server
    Client->>Server: HTTPS Request
    Server-->>Client: Server Certificate
    Client->>Client: Verify Certificate (against trusted CAs)
    alt Certificate Valid
        Client->>Server: Encrypted Request
        Server-->>Client: Encrypted Response
    else Certificate Invalid
        Client-->>Client: Raise SSLError
    end

Sequence diagram of HTTPS certificate verification process.

Handling Self-Signed or Custom Certificates

In some scenarios, such as internal networks or development environments, you might encounter servers using self-signed certificates or certificates issued by a private CA not included in requests' default trust store. In these cases, you have a few options:

1. Specify a custom CA bundle: Provide the path to a .pem file containing the trusted CA certificates.
2. Specify a client-side certificate: If the server requires client authentication, you can provide your client certificate and key.
3. Disable verification (use with extreme caution): Set verify=False. This should never be done in production unless you fully understand the risks and have alternative security measures in place.

import requests

# Option 1: Using a custom CA bundle
try:
    response = requests.get('https://internal.example.com/data', verify='/path/to/custom_ca_bundle.pem')
    response.raise_for_status()
    print("Request successful with custom CA bundle.")
except requests.exceptions.RequestException as e:
    print(f"Error with custom CA bundle: {e}")

# Option 2: Using a client-side certificate for mutual TLS
try:
    response = requests.get('https://secure.api.com/resource', cert=('/path/to/client.crt', '/path/to/client.key'))
    response.raise_for_status()
    print("Request successful with client certificate.")
except requests.exceptions.RequestException as e:
    print(f"Error with client certificate: {e}")

# Option 3: Disabling verification (DANGEROUS - AVOID IN PRODUCTION)
# This will suppress InsecureRequestWarning, but it's still insecure.
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

try:
    response = requests.get('https://insecure.example.com/data', verify=False)
    response.raise_for_status()
    print("Request successful with verification disabled (WARNING: INSECURE).")
except requests.exceptions.RequestException as e:
    print(f"Error with disabled verification: {e}")

Examples for handling custom certificates and disabling verification.