Downloading files with curl

Learn downloading files with curl with practical examples, diagrams, and best practices. Covers linux, ubuntu, curl development techniques with visual explanations.
Tags:
Categories:

Mastering File Downloads with cURL

Hero image for Downloading files with curl

Learn how to efficiently download files from the command line using cURL, covering basic usage, advanced options, and common scenarios.

cURL is a powerful command-line tool and library for transferring data with URLs. It supports a wide range of protocols, including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP, LDAPS, FILE, POP3, POP3S, IMAP, IMAPS, SMTP, SMTPS, RTMP and RTSP. This article will focus on its capabilities for downloading files, a common task for developers and system administrators alike, especially in Linux and Ubuntu environments.

Basic File Download

The most straightforward way to download a file with cURL is to provide the URL of the file. By default, cURL will output the content of the file to standard output (your terminal). To save the file to a local file, you need to use specific options.

curl https://example.com/path/to/your/file.zip

Downloading a file and outputting to standard output.

To save the file to a local file, you can use the -o (output) or -O (remote file name) options. The -o option allows you to specify the local filename, while -O saves the file with its original name from the remote server.

# Save with a specified local filename
curl -o local_filename.zip https://example.com/path/to/your/file.zip

# Save with the remote filename
curl -O https://example.com/path/to/your/file.zip

Using -o and -O to save downloaded files.

Resuming Interrupted Downloads

One of cURL's most useful features is its ability to resume interrupted downloads. This is particularly helpful for large files or unreliable network connections. The -C - option tells cURL to automatically pick up where it left off.

curl -C - -O https://example.com/path/to/large_file.iso

Resuming an interrupted download using -C -.

When you run this command, cURL checks for an existing file with the same name. If found, it determines the size of the local file and sends a Range header to the server to request only the missing part of the file. This process is illustrated in the following sequence diagram.

sequenceDiagram
    participant User
    participant Client as cURL
    participant Server

    User->>Client: curl -C - -O URL
    Client->>Client: Check for existing local file
    alt Local file exists
        Client->>Client: Get local file size (N bytes)
        Client->>Server: GET /file HTTP/1.1\nRange: bytes=N-
        Server-->>Client: HTTP/1.1 206 Partial Content\nContent-Range: bytes N-M/Total\n[Remaining file data]
        Client->>Client: Append data to local file
    else No local file
        Client->>Server: GET /file HTTP/1.1
        Server-->>Client: HTTP/1.1 200 OK\n[Full file data]
        Client->>Client: Save full file
    end
    Client->>User: Download complete

Sequence diagram for resuming a cURL download.

Advanced Download Options

cURL offers many advanced options for more complex download scenarios. Here are a few commonly used ones:

Download with Progress Bar

To see a progress bar during download, which is very helpful for large files, use the # option with -# or --progress-bar.

curl -# -O https://example.com/path/to/large_file.zip

Follow Redirects

Many URLs redirect to another location. By default, cURL does not follow these redirects. Use the -L or --location option to enable this.

curl -L -O https://shorturl.at/abcde

Authentication

For files protected by HTTP basic authentication, use the -u or --user option.

curl -u "username:password" -O https://secure.example.com/private_file.pdf

Limit Download Speed

To prevent cURL from consuming all your bandwidth, you can limit the download speed using --limit-rate.

curl --limit-rate 100K -O https://example.com/path/to/large_file.zip