What is "Signal 15 received"

Understanding 'Signal 15 Received': What It Means and How to Handle It

Explore the meaning of 'Signal 15 received' in Linux and C/MPI applications, why it occurs, and effective strategies for graceful process termination.

When working with C, MPI, or other applications on Linux systems, you might occasionally encounter the message "Signal 15 received" in your logs or console output. This message indicates that a process has received a SIGTERM signal. Understanding what SIGTERM is, why it's sent, and how your applications should respond to it is crucial for building robust and reliable software.

What is Signal 15 (SIGTERM)?

In Unix-like operating systems, signals are a limited form of inter-process communication (IPC) used to notify a process of an event. SIGTERM, or Signal 15, is the termination signal. Unlike SIGKILL (Signal 9), which forcefully terminates a process immediately without giving it a chance to clean up, SIGTERM is a polite request for a process to terminate. It allows the receiving process to perform necessary cleanup operations before exiting.

When a process receives SIGTERM, it can:

1. Catch the signal: The process can install a signal handler to execute specific code when SIGTERM is received. This is the ideal scenario for graceful shutdown.
2. Ignore the signal: While possible, ignoring SIGTERM is generally not recommended as it can lead to resource leaks or data corruption if the process is eventually killed by SIGKILL.
3. Perform default action: If no signal handler is installed, the default action for SIGTERM is to terminate the process.

Flowchart of SIGTERM handling in a process

Common Scenarios for Receiving SIGTERM

Several situations can lead to a process receiving SIGTERM:

• System Shutdown/Reboot: When a Linux system shuts down or reboots, the init system (e.g., systemd) sends SIGTERM to all running processes to allow them to shut down cleanly before resorting to SIGKILL.
• Manual Termination: A user or administrator can manually send SIGTERM using commands like kill, pkill, or killall.
  • kill <PID> (sends SIGTERM by default)
  • kill -15 <PID> (explicitly sends SIGTERM)
• Orchestration Systems: Container orchestrators (like Kubernetes) or process managers (like supervisord) often send SIGTERM to containers or managed processes when they need to be stopped or restarted.
• Resource Managers (e.g., MPI, HPC): In High-Performance Computing (HPC) environments, job schedulers and MPI runtimes might send SIGTERM to processes when a job is cancelled, time-limited, or needs to be reallocated.

# Find the PID of a process named 'my_application'
PID=$(pgrep my_application)

# Send SIGTERM to the process
kill $PID

# Alternatively, explicitly send SIGTERM
kill -15 $PID

# Send SIGTERM to all processes named 'my_application'
killall my_application

Using kill and killall to send SIGTERM

Implementing Graceful Shutdown in C and MPI

For applications, especially long-running services or MPI programs, it's critical to implement signal handlers for SIGTERM to ensure a graceful shutdown. This prevents data loss, resource leaks, and allows for proper state saving.

C Example: Basic Signal Handler

In C, you use the signal() function (or the more robust sigaction()) to register a function that will be called when a specific signal is received. Inside this handler, you should set a flag that the main loop checks, rather than performing complex operations directly in the handler.

MPI Example: Coordinated Shutdown

For MPI applications, a SIGTERM received by one process should ideally trigger a coordinated shutdown across all participating processes. This often involves broadcasting a shutdown message to all ranks and then calling MPI_Finalize().

C Signal Handler

#include <stdio.h> #include <stdlib.h> #include <signal.h> #include <unistd.h>

volatile sig_atomic_t shutdown_requested = 0;

void sigterm_handler(int signum) { printf("\nSignal %d (SIGTERM) received. Initiating graceful shutdown...\n", signum); shutdown_requested = 1; }

int main() { // Register signal handler for SIGTERM if (signal(SIGTERM, sigterm_handler) == SIG_ERR) { perror("Error registering SIGTERM handler"); return 1; }

printf("Application running. PID: %d\n", getpid());
printf("Send 'kill %d' to terminate gracefully.\n", getpid());

while (!shutdown_requested) {
    printf("Working...\n");
    sleep(2); // Simulate work
}

printf("Performing cleanup operations...\n");
// Add your cleanup code here (e.g., close files, save state, free memory)
printf("Cleanup complete. Exiting.\n");

return 0;

}

MPI Graceful Shutdown

#include <mpi.h> #include <stdio.h> #include <stdlib.h> #include <signal.h> #include <unistd.h>

volatile sig_atomic_t shutdown_requested = 0;

void sigterm_handler(int signum) { printf("\nRank %d: Signal %d (SIGTERM) received. Setting shutdown flag.\n", MPI_COMM_WORLD, signum); shutdown_requested = 1; }

int main(int argc, char** argv) { MPI_Init(&argc, &argv);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);

// Register signal handler for SIGTERM on all ranks
if (signal(SIGTERM, sigterm_handler) == SIG_ERR) {
    perror("Error registering SIGTERM handler");
    MPI_Abort(MPI_COMM_WORLD, 1);
}

printf("Rank %d: Application running. PID: %d\n", world_rank, getpid());
if (world_rank == 0) {
    printf("Send 'kill %d' to rank 0 to initiate coordinated shutdown.\n", getpid());
}

int global_shutdown = 0;
while (!global_shutdown) {
    // Simulate work
    printf("Rank %d: Working...\n", world_rank);
    sleep(1);

    // Check local shutdown request and broadcast to all ranks
    MPI_Allreduce(&shutdown_requested, &global_shutdown, 1, MPI_INT, MPI_LOR, MPI_COMM_WORLD);
}

printf("Rank %d: Performing cleanup operations...\n", world_rank);
// Add rank-specific cleanup code here
printf("Rank %d: Cleanup complete. Finalizing MPI.\n", world_rank);

MPI_Finalize();
return 0;

}

Distinguishing SIGTERM from SIGKILL

It's important to understand the difference between SIGTERM (Signal 15) and SIGKILL (Signal 9):

• SIGTERM (Signal 15): A polite request to terminate. The process can catch it, clean up, and exit gracefully. If a process ignores SIGTERM or takes too long to shut down, the system or orchestrator might escalate to SIGKILL.
• SIGKILL (Signal 9): An immediate, unconditional termination. This signal cannot be caught, ignored, or blocked by the process. The operating system forcefully stops the process, potentially leading to data corruption or resource leaks if the process was in the middle of a critical operation. SIGKILL is typically used as a last resort.