x86 function call types
Categories:
Understanding x86 Function Call Types
Explore the different calling conventions used in x86 assembly, including cdecl, stdcall, fastcall, and thiscall, and their impact on stack management and register usage.
Function calls are fundamental to structured programming, allowing code reuse and modularity. In x86 assembly, the way functions are called and how parameters are passed and return values are handled is governed by calling conventions. These conventions dictate aspects like parameter order, stack cleanup responsibilities, and register preservation. Understanding these differences is crucial for low-level programming, reverse engineering, and interfacing with libraries written in different languages.
Common x86 Calling Conventions
Several calling conventions have evolved for x86 architectures, each with its own set of rules. The most prevalent ones include cdecl, stdcall, fastcall, and thiscall. While they all achieve the goal of transferring control and data to a subroutine, their specific implementations lead to different performance characteristics and compatibility requirements.
flowchart TD
A[Caller] --> B{Push Parameters};
B --> C[CALL Instruction];
C --> D[Callee Entry];
D --> E{Function Body Execution};
E --> F{Return Value Handling};
F --> G[RET Instruction];
G --> H[Caller Return];
H --> I{Stack Cleanup (Who?)}General flow of an x86 function call, highlighting key stages.
1. cdecl (C Declaration)
The cdecl calling convention is the default for C and C++ programs on x86. It is characterized by the caller being responsible for cleaning up the stack after the function returns. Parameters are pushed onto the stack from right to left. This convention supports variadic functions (functions with a variable number of arguments) because the caller knows exactly how many arguments it pushed and can clean them up accordingly.
; Example of cdecl call
; Caller side
push arg3
push arg2
push arg1
call my_cdecl_function
add esp, 12 ; Caller cleans up 3 arguments (3 * 4 bytes)
; Callee side (my_cdecl_function)
; ... function body ...
ret ; Callee does not clean up stack
Illustrative x86 assembly for a cdecl function call.
cdecl convention is flexible due to caller cleanup, making it suitable for functions like printf that accept a variable number of arguments. However, it can lead to slightly larger code size as each call site includes stack cleanup instructions.2. stdcall (Standard Call)
The stdcall convention is commonly used for Windows API functions. Unlike cdecl, the callee is responsible for cleaning up the stack. Parameters are pushed onto the stack from right to left, similar to cdecl. The ret instruction in stdcall functions often includes an immediate operand to pop the arguments off the stack, e.g., ret 12.
; Example of stdcall call
; Caller side
push arg2
push arg1
call my_stdcall_function
; No stack cleanup by caller
; Callee side (my_stdcall_function)
; ... function body ...
ret 8 ; Callee cleans up 2 arguments (2 * 4 bytes)
Illustrative x86 assembly for a stdcall function call.
stdcall results in smaller code size compared to cdecl because the stack cleanup instruction is only present once in the callee, rather than at every call site. However, it does not support variadic functions.3. fastcall
The fastcall convention attempts to improve performance by passing some arguments in CPU registers instead of the stack. The specific registers used can vary between compilers and architectures (e.g., ECX and EDX are common on x86 for the first two arguments). Remaining arguments are pushed onto the stack from right to left. Stack cleanup is typically handled by the callee, similar to stdcall.
; Example of fastcall call (Microsoft Visual C++ convention)
; Caller side
mov ecx, arg1 ; First argument in ECX
mov edx, arg2 ; Second argument in EDX
push arg3 ; Third argument on stack
call my_fastcall_function
; Callee side (my_fastcall_function)
; ... function body using ECX, EDX, and [ESP+4] for arguments ...
ret 4 ; Callee cleans up stack arguments (arg3)
Illustrative x86 assembly for a fastcall function call, showing register usage.
fastcall is compiler-dependent. This means code compiled with one compiler using fastcall might not be directly compatible with code compiled with another, or even different versions of the same compiler.4. thiscall
The thiscall convention is specifically used for non-static member functions of C++ classes. It's essentially a variation of stdcall or fastcall where the this pointer (a pointer to the object instance) is passed to the member function. Typically, the this pointer is passed in the ECX register (on MSVC) or pushed onto the stack as the first argument (on GCC/Clang). Other arguments follow the stdcall or cdecl rules, depending on the compiler and platform.
; Example of thiscall (Microsoft Visual C++ convention)
; Caller side
mov ecx, OFFSET my_object ; 'this' pointer in ECX
push arg1
call MyClass::my_member_function
; Callee side (MyClass::my_member_function)
; ... function body using ECX for 'this' and [ESP+4] for arg1 ...
ret 4 ; Callee cleans up stack arguments
Illustrative x86 assembly for a thiscall member function call.
thiscall convention is implicitly handled by C++ compilers when you call a member function. You typically don't explicitly specify it in your C++ code, but it's important to understand its mechanics when debugging or analyzing compiled code.