How can I split a large text file into smaller files with an equal number of lines?

Splitting Large Text Files into Equal-Sized Chunks

Learn how to efficiently divide large text files into smaller, manageable files, each containing a specified number of lines, using command-line tools in Unix-like environments.

Working with extremely large text files can often be cumbersome. Whether you're processing logs, analyzing data, or preparing files for parallel processing, splitting a single large file into multiple smaller ones, each with an equal number of lines, is a common and essential task. This article will guide you through various command-line methods available in Unix-like systems (Linux, macOS, WSL) to achieve this efficiently.

Understanding the 'split' Command

The split command is a powerful and versatile utility specifically designed for breaking files into smaller pieces. It offers several options to control how the file is split, including by number of lines, bytes, or even by the number of output files. For our purpose of splitting by an equal number of lines, split is the go-to tool.

flowchart TD
    A[Start] --> B{Input File (large.txt)}
    B --> C["Define Lines per File (e.g., 1000)"]
    C --> D["Execute 'split -l 1000 large.txt'"]
    D --> E["Output Files (xaa, xab, xac, ...)"]
    E --> F[End]

Basic workflow for splitting a file by lines using the split command.

split -l 1000 large.txt

Basic usage of split to divide large.txt into files of 1000 lines each.

By default, split will name the output files xaa, xab, xac, and so on. You can specify a prefix for the output files to make them more descriptive. For example, to use part_ as a prefix, the command would be:

split -l 1000 large.txt part_

Using a custom prefix for output files.

Advanced Splitting with Custom Naming and Header Handling

While the basic split command is effective, real-world scenarios often require more control over file naming and handling of headers. Let's explore how to achieve this using a combination of head, tail, and a for loop.

# Define variables
INPUT_FILE="data.csv"
LINES_PER_FILE=1000
OUTPUT_PREFIX="chunk_"
HEADER=$(head -n 1 "$INPUT_FILE") # Extract the first line as header

# Split the file, skipping the header
tail -n +2 "$INPUT_FILE" | split -l "$LINES_PER_FILE" - "$OUTPUT_PREFIX"

# Add header to each split file and rename
FILE_COUNT=0
for f in "${OUTPUT_PREFIX}"*; do
  FILE_COUNT=$((FILE_COUNT + 1))
  NEW_NAME="${OUTPUT_PREFIX}$(printf "%03d" "$FILE_COUNT").csv"
  echo "$HEADER" > "$NEW_NAME"
  cat "$f" >> "$NEW_NAME"
  rm "$f"
done

Script to split a CSV file, add a header to each chunk, and rename files sequentially.

This script first extracts the header from data.csv. Then, it pipes the rest of the file (from the second line onwards) to split, which creates temporary files with the specified prefix. Finally, it iterates through these temporary files, prepends the header, renames them with a sequential number (e.g., chunk_001.csv, chunk_002.csv), and removes the original temporary files.

Alternative: Using awk for More Control

For highly customized splitting logic, awk provides a powerful alternative. While split is generally faster for simple line-based splitting, awk gives you programmatic control over file content and naming within a single command. This can be particularly useful if you need to perform additional processing during the split.

awk -v lines_per_file=1000 -v prefix="part_" '{
    if (NR % lines_per_file == 1) {
        file_num = int((NR - 1) / lines_per_file) + 1;
        close(output_file);
        output_file = sprintf("%s%03d.txt", prefix, file_num);
    }
    print > output_file;
}' large.txt

Splitting a file into 1000-line chunks using awk with sequential numbering.

In this awk command:

• -v lines_per_file=1000 -v prefix="part_" sets variables for the number of lines and the output file prefix.
• NR is the current record (line) number.
• NR % lines_per_file == 1 checks if the current line is the first line of a new chunk.
• file_num calculates the sequential file number.
• close(output_file) ensures that the previous output file is properly closed before opening a new one.
• sprintf("%s%03d.txt", prefix, file_num) constructs the new filename with zero-padded numbers.
• print > output_file redirects the current line to the appropriate output file.