How to loop over files in directory and change path and add suffix to filename

Looping Over Files: Changing Paths and Adding Suffixes in Bash

Learn how to efficiently iterate through files in a directory, modify their paths, and append custom suffixes to filenames using Bash scripting and glob patterns.

Manipulating files in a directory is a common task in scripting. Whether you need to move files to a new location, rename them, or add specific identifiers, Bash provides powerful tools to achieve this. This article will guide you through the process of looping over files, changing their directory paths, and adding suffixes to their names, using practical examples and best practices.

Understanding File Iteration with Globbing

The most straightforward way to iterate over files in Bash is by using globbing. Glob patterns allow you to specify sets of filenames with wildcard characters. The for loop construct then processes each matching file. This method is generally preferred over find for simple file lists because it's often more readable and efficient for basic scenarios.

for file in *.txt; do
  echo "Processing file: $file"
done

Basic for loop iterating over all .txt files in the current directory.

Changing File Paths and Adding Suffixes

Once you're iterating through files, the next step is to manipulate their paths and names. Bash parameter expansion offers robust ways to extract parts of a filename (like the base name or extension) and construct new paths. We'll combine these techniques to move files to a new directory and append a suffix.

flowchart TD
    A[Start Loop] --> B{"File Found?"}
    B -- Yes --> C[Extract Filename & Extension]
    C --> D[Construct New Filename with Suffix]
    D --> E[Define Target Directory]
    E --> F[Construct New Full Path]
    F --> G[Move/Rename File]
    G --> H[Next Iteration]
    B -- No --> I[End Loop]

Workflow for processing files, changing paths, and adding suffixes.

#!/bin/bash

SOURCE_DIR="./input_files"
TARGET_DIR="./processed_files"
SUFFIX="_processed"

mkdir -p "$TARGET_DIR" # Ensure target directory exists

for filepath in "$SOURCE_DIR"/*.txt; do
  # Check if the glob found any files
  if [ -e "$filepath" ]; then
    filename=$(basename -- "$filepath") # Get just the filename (e.g., 'document.txt')
    extension="${filename##*.}"        # Get the extension (e.g., 'txt')
    basename_no_ext="${filename%.*}"  # Get filename without extension (e.g., 'document')

    new_filename="${basename_no_ext}${SUFFIX}.${extension}"
    new_filepath="$TARGET_DIR/$new_filename"

    echo "Moving '$filepath' to '$new_filepath'"
    mv "$filepath" "$new_filepath"
  fi
done

Bash script to move .txt files from a source directory to a target directory, adding a suffix before the extension.

Handling Edge Cases and Advanced Scenarios

While the basic for loop with globbing is powerful, you might encounter scenarios requiring more advanced handling, such as files with no extensions, or needing to process files recursively. For recursive operations, find combined with while read is often more robust.

#!/bin/bash

SOURCE_DIR="./input_files"
TARGET_DIR="./processed_files"
SUFFIX="_done"

mkdir -p "$TARGET_DIR"

# Using find for recursive processing
find "$SOURCE_DIR" -type f -name "*.log" -print0 | while IFS= read -r -d $'' filepath; do
  filename=$(basename -- "$filepath")
  extension="${filename##*.}"
  basename_no_ext="${filename%.*}"

  # Handle files without extensions gracefully
  if [ "$filename" = "$basename_no_ext" ]; then
    new_filename="${filename}${SUFFIX}"
  else
    new_filename="${basename_no_ext}${SUFFIX}.${extension}"
  fi

  # Create subdirectories in target if they existed in source
  relative_path="${filepath#$SOURCE_DIR/}"
  target_subdir=$(dirname -- "$relative_path")
  mkdir -p "$TARGET_DIR/$target_subdir"

  new_filepath="$TARGET_DIR/$target_subdir/$new_filename"

  echo "Processing '$filepath' to '$new_filepath'"
  mv "$filepath" "$new_filepath"
done

Advanced script using find for recursive processing, handling files without extensions and preserving directory structure.