Fully understanding how .exe file is executed

Fully Understanding How an .exe File is Executed on Windows

Dive deep into the intricate process of how a Windows executable (.exe) file comes to life, from user double-click to process creation, covering loader operations, memory management, and security mechanisms.

When you double-click an .exe file on Windows, it triggers a sophisticated sequence of operations orchestrated by the operating system. This article breaks down the journey of an executable, from disk to active process, highlighting the critical stages and components involved in its execution.

The Initial Spark: User Interaction and OS Intervention

The execution process begins with a user interaction—typically a double-click on the .exe file in File Explorer. This action is intercepted by the Windows shell, which then invokes the CreateProcess API call. CreateProcess is the fundamental function responsible for creating a new process, and it handles much of the initial setup.

The CreateProcess function performs several key tasks:

1. Step 1

Validating the executable file's format (e.g., PE format for Windows).

2. Step 2

Creating a new process object in the kernel, which includes a virtual address space and other resources.

3. Step 3

Creating the initial thread within this new process.

4. Step 4

Loading the executable image (the .exe file itself) into the process's virtual address space.

5. Step 5

Loading necessary dynamic-link libraries (DLLs) that the executable depends on.

The Role of the PE Loader and Virtual Memory

Once the CreateProcess function is initiated, the Windows loader (part of ntdll.dll and the kernel) takes over. The loader's primary responsibility is to map the executable's image from disk into the process's virtual address space. This isn't a simple copy operation; it involves complex memory management.

Flowchart of the Windows PE File Loading Process

The Portable Executable (PE) format is the standard executable file format for Windows. It contains essential information such as code sections, data sections, import tables (listing required DLL functions), and export tables (listing functions provided by the DLL).

#include <windows.h>
#include <stdio.h>

int main()
{
    STARTUPINFO si;
    PROCESS_INFORMATION pi;

    ZeroMemory( &si, sizeof(si) );
    si.cb = sizeof(si);
    ZeroMemory( &pi, sizeof(pi) );

    // Start the child process.
    if( !CreateProcess( "C:\\Windows\\System32\\notepad.exe",   // No module name (use command line)
                        NULL,        // Command line
                        NULL,           // Process handle not inheritable
                        NULL,           // Thread handle not inheritable
                        FALSE,          // Set handle inheritance to FALSE
                        0,              // No creation flags
                        NULL,           // Use parent's environment block
                        NULL,           // Use parent's starting directory 
                        &si,            // Pointer to STARTUPINFO structure
                        &pi )           // Pointer to PROCESS_INFORMATION structure
    )
    {
        printf( "CreateProcess failed (%d).\n", GetLastError() );
        return 1;
    }

    // Wait until child process exits.
    WaitForSingleObject( pi.hProcess, INFINITE );

    // Close process and thread handles.
    CloseHandle( pi.hProcess );
    CloseHandle( pi.hThread );

    return 0;
}

A simple C program demonstrating how to use CreateProcess to launch Notepad.

Dynamic Linking and Entry Point Execution

After the main executable image is loaded, the loader proceeds to resolve its dependencies. Most Windows applications rely heavily on DLLs for functionality (e.g., kernel32.dll, user32.dll). The loader scans the executable's Import Address Table (IAT) to find the addresses of functions imported from these DLLs.

For each imported function, the loader locates the corresponding DLL, loads it into the process's address space if not already present, and then patches the IAT entries with the actual memory addresses of those functions. This process is called dynamic linking.

Diagram illustrating dynamic linking and the Import Address Table (IAT).

Once all necessary DLLs are loaded and the IAT is resolved, the loader performs any required relocations. Relocations are necessary if the preferred base address of an executable or DLL is already occupied, requiring the module to be loaded at a different address and all internal pointers to be adjusted accordingly.

Finally, the loader transfers control to the executable's entry point. For C/C++ applications, this is typically the CRT (C Runtime) startup code (e.g., mainCRTStartup or WinMainCRTStartup), which then calls your application's main or WinMain function. At this point, your application's code begins to execute.