Under what conditions are HTTP request headers removed by proxies?

When Do Proxies Remove HTTP Request Headers?

Explore the conditions under which HTTP proxies modify or remove request headers, focusing on security, privacy, and protocol compliance according to RFCs.

HTTP proxies play a crucial role in web communication, acting as intermediaries between clients and servers. While they offer benefits like caching, security, and anonymity, they can also modify or remove HTTP request headers. Understanding these conditions is vital for developers and system administrators to ensure proper application behavior and security. This article delves into the common scenarios and reasons behind header removal by proxies, referencing relevant RFCs.

Hop-by-Hop Headers: The Primary Candidates for Removal

The most common reason for a proxy to remove headers is if they are classified as 'hop-by-hop' headers. These headers are intended only for a single transport-level connection and are not meant to be retransmitted by proxies to the next hop. RFC 2616, Section 13.5.1, explicitly lists several headers that must be removed by a proxy before forwarding a request. These headers manage the connection between the client and the proxy, or between the proxy and the origin server, and are not relevant to the end-to-end communication.

The standard hop-by-hop headers include:

Connection
Keep-Alive
Proxy-Authenticate
Proxy-Authorization
TE
Trailers
Transfer-Encoding
Upgrade

List of standard hop-by-hop HTTP headers.

Additionally, any header listed in a Connection header field value is also treated as hop-by-hop and must be removed. For example, if a client sends Connection: close, X-My-Custom-Header, both Connection and X-My-Custom-Header should be removed by the proxy.

flowchart TD
    Client["Client Request"] --> Proxy["HTTP Proxy"];
    Proxy --> |"Remove Hop-by-Hop Headers"| OriginServer["Origin Server"];
    subgraph Hop-by-Hop Headers
        H1["Connection"]
        H2["Keep-Alive"]
        H3["Proxy-Authenticate"]
        H4["Proxy-Authorization"]
        H5["TE"]
        H6["Trailers"]
        H7["Transfer-Encoding"]
        H8["Upgrade"]
    end
    Proxy -.-> H1;
    Proxy -.-> H2;
    Proxy -.-> H3;
    Proxy -.-> H4;
    Proxy -.-> H5;
    Proxy -.-> H6;
    Proxy -.-> H7;
    Proxy -.-> H8;
    OriginServer["Origin Server"]

Flow of HTTP request through a proxy, illustrating hop-by-hop header removal.

Security and Privacy Considerations

Beyond explicit hop-by-hop headers, proxies may remove or modify other headers for security, privacy, or operational reasons. This is often configurable by the proxy administrator. Common examples include:

1. Via Header: Proxies must add or append to the Via header to indicate the intermediate protocols and recipients between the client and the server. This is not removal, but modification, to trace the request path.
2. X-Forwarded-For / X-Real-IP: While not removed, proxies often add or modify these headers to pass the original client's IP address to the origin server, especially when operating as a reverse proxy or load balancer. This helps the origin server log the true client IP.
3. User-Agent: Some proxies might strip or modify the User-Agent header to anonymize clients or to present a consistent User-Agent string for all traffic passing through them.
4. Referer: For privacy reasons, proxies might remove or truncate the Referer header, preventing the origin server from knowing the previous page visited by the client.
5. Authorization Headers: In some highly secure environments, proxies might be configured to strip Authorization headers if they are not intended to be forwarded beyond a certain security boundary, though this is less common for general-purpose proxies.
6. Custom Headers: Any custom header (e.g., X-Custom-Header) can be removed or modified by a proxy if its configuration dictates so, often for security filtering, policy enforcement, or to prevent information leakage.

Proxy Configuration and Policy

The behavior of proxies regarding header removal is heavily dependent on their configuration and the policies enforced by their administrators. Enterprise proxies, for instance, often have strict rules for security and compliance. Web application firewalls (WAFs) and load balancers, which often act as reverse proxies, are particularly prone to modifying or adding headers to fulfill their specific functions (e.g., adding X-Forwarded-Proto, X-Forwarded-Host).

Developers should be aware that any header not explicitly defined as an end-to-end header in the HTTP specification could potentially be modified or removed by an intermediary. When designing APIs or web applications, it's best practice to assume that only end-to-end headers will reliably reach the origin server unmodified, unless a specific proxy chain is known and controlled.