Shell script not running via crontab, but runs fine manually

Troubleshooting Crontab: Why Your Shell Script Isn't Running Automatically

Discover common reasons why your shell scripts execute perfectly manually but fail when scheduled with crontab, and learn how to diagnose and fix these issues.

It's a common frustration for developers and system administrators: a shell script runs flawlessly when executed directly from the command line, but remains stubbornly silent when scheduled via crontab. This discrepancy often stems from differences in the execution environment between a manual shell session and a non-interactive cron job. This article will guide you through the most frequent culprits and provide systematic troubleshooting steps to get your cron jobs running reliably.

Understanding the Cron Environment

When you execute a script manually, it inherits the environment variables, PATH, and shell settings of your current interactive session. Cron, however, runs jobs in a minimal, non-interactive shell environment. This environment often lacks crucial variables like PATH, which specifies directories where executable commands are located, or other custom variables your script might rely on. This is the single most common reason for scripts failing in cron.

flowchart TD
    A[Manual Execution] --> B{Interactive Shell}
    B --> C[Full PATH & Environment]
    C --> D[Script Runs Successfully]

    E[Crontab Execution] --> F{Minimal Shell}
    F --> G[Limited PATH & Environment]
    G --> H{Script Fails/Behaves Unexpectedly}
    H --> I[Troubleshoot Environment Differences]

Comparison of Manual vs. Crontab Execution Environments

Common Causes and Solutions

Several factors can prevent a script from running correctly in crontab. Addressing these systematically will help pinpoint the problem.

1. Incomplete PATH or Environment Variables

As mentioned, cron's PATH is often very limited (e.g., /usr/bin:/bin). If your script uses commands not found in these directories (like python, node, java, or custom binaries), it will fail. Similarly, if your script relies on environment variables like HOME, USER, or custom variables, they might not be set.

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Or, for specific commands:
/usr/bin/python3 /path/to/your_script.py

# To see cron's default environment:
* * * * * env > /tmp/cron_env.log 2>&1

2. Relative Paths and Working Directory

When you run a script manually, you're usually in a specific directory. Cron jobs, however, often execute from the user's home directory (~) or /. If your script uses relative paths to access files or other scripts, these paths will likely be incorrect in the cron environment.

# In your script, use absolute paths:
#!/bin/bash
cd /path/to/your/script/directory
./my_utility.sh

# Or directly in crontab:
* * * * * cd /path/to/your/script/directory && ./your_script.sh

3. Permissions and Ownership

Ensure your script has execute permissions (chmod +x script.sh) and that the user under which the cron job runs (usually the user who owns the crontab, or root for system-wide crontabs) has read and execute permissions for the script and any files it accesses.

chmod +x /path/to/your_script.sh
ls -l /path/to/your_script.sh # Verify permissions

4. Shebang Line and Shell Differences

Always include a shebang line (e.g., #!/bin/bash or #!/usr/bin/env python) at the very beginning of your script. This tells cron which interpreter to use. If omitted, cron might default to /bin/sh, which has different behaviors than bash or other shells.

#!/bin/bash
# Your bash script content here

#!/usr/bin/env python3
# Your Python script content here

5. Output and Error Handling

Cron jobs typically email any output (stdout or stderr) to the user. If your script produces a lot of output, or if email is not configured, this can be problematic. Redirecting output to a log file is crucial for debugging.

* * * * * /path/to/your_script.sh > /var/log/my_script.log 2>&1
# To discard all output:
* * * * * /path/to/your_script.sh > /dev/null 2>&1

6. System-wide vs. User Crontab

Understand the difference between crontab -e (user-specific) and /etc/crontab or /etc/cron.d/* (system-wide). System-wide crontabs often require specifying the USER field. User crontabs run as the user who owns them.

# Example from /etc/crontab or /etc/cron.d/
# m h dom mon dow user  command
  0 * * * * root  /path/to/my_root_script.sh

1. Verify Script Execution Manually

First, ensure your script runs without errors from the command line. Use the exact command you plan to put in crontab, including absolute paths: /usr/bin/bash /path/to/your_script.sh.

2. Check Cron Logs

Look for cron-related messages in system logs. Common locations include /var/log/syslog, /var/log/cron, or /var/log/messages (depending on your Linux distribution).

3. Set Full PATH and Environment

At the top of your crontab file (using crontab -e), add a PATH variable that includes all necessary directories. You can also set other environment variables here. Alternatively, set them within the script itself.

4. Use Absolute Paths

Modify your script and crontab entry to use absolute paths for all commands, files, and directories. For example, instead of python, use /usr/bin/python.

5. Redirect Output to a Log File

Append >> /tmp/cron_debug.log 2>&1 to your crontab entry. This will capture both standard output and errors, providing valuable debugging information.

6. Test with a Simple Script

If all else fails, create a very simple script (e.g., echo "Hello from cron" > /tmp/cron_test.log) and schedule it. If this works, the issue is with your complex script; if not, the problem is with your cron setup itself.