Comparing the value pointed by a pointer

Comparing Values Pointed To by Pointers in x86-64 Assembly

Understand how to dereference and compare values stored at memory addresses using pointers in x86-64 assembly language, covering common scenarios and instructions.

In x86-64 assembly, pointers are essentially memory addresses. Comparing the value pointed to by a pointer involves dereferencing that pointer to access the data it holds, and then performing a comparison operation on that data. This is a fundamental operation in low-level programming, crucial for control flow, array processing, and data manipulation. This article will guide you through the process, illustrating with practical examples.

Understanding Pointers and Dereferencing

A pointer in assembly is typically stored in a general-purpose register (e.g., RAX, RBX, RCX, RDX, RSI, RDI, RBP, RSP, or R8-R15). The value in such a register represents a memory address. To access the data at that address, you must dereference the pointer. In x86-64 assembly, dereferencing is implicitly done by using square brackets [] around the register containing the address. For example, [RAX] means 'the value at the memory address stored in RAX'.

flowchart TD
    A[Register holds Address] --> B{Dereference [Register]}
    B --> C[Access Value at Address]
    C --> D[Perform Comparison]

Conceptual flow of dereferencing and comparing values

Comparing a Dereferenced Value to a Constant

The most straightforward comparison involves checking if the value pointed to by a pointer is equal to a specific constant. This is typically done using the CMP (compare) instruction. CMP performs a subtraction internally and sets the EFLAGS register based on the result, but it does not store the result of the subtraction. Subsequent conditional jump instructions (e.g., JE, JNE, JG, JL) then use these flags to alter program flow.

section .data
    my_value dq 1234h

section .text
    global _start

_start:
    mov rax, my_value      ; Load the address of my_value into RAX
    cmp qword [rax], 1234h ; Compare the QWORD at [RAX] with 1234h
    je  .equal_label       ; Jump if equal
    ; ... code if not equal

.equal_label:
    ; ... code if equal
    mov rax, 60            ; sys_exit
    xor rdi, rdi           ; exit code 0
    syscall

Comparing a QWORD pointed to by RAX with a constant value.

Comparing Two Dereferenced Values

Comparing the values pointed to by two different pointers requires dereferencing both. Since CMP can only take one memory operand, you'll typically need to load one of the dereferenced values into a general-purpose register first, and then compare that register's content with the value dereferenced from the second pointer.

section .data
    ptr1_data dq 100
    ptr2_data dq 200

section .text
    global _start

_start:
    ; Assume RDI holds address of ptr1_data, RSI holds address of ptr2_data
    mov rdi, ptr1_data
    mov rsi, ptr2_data

    mov rax, qword [rdi]   ; Load value pointed to by RDI into RAX
    cmp rax, qword [rsi]   ; Compare RAX with value pointed to by RSI
    je  .values_equal      ; Jump if values are equal
    ; ... code if not equal

.values_equal:
    ; ... code if values are equal
    mov rax, 60            ; sys_exit
    xor rdi, rdi           ; exit code 0
    syscall

Comparing two QWORD values pointed to by RDI and RSI.

Using TEST for Bitwise Comparisons

While CMP is used for arithmetic comparisons (equality, greater than, less than), the TEST instruction is used for bitwise comparisons. TEST performs a bitwise AND operation between its operands and sets the EFLAGS register (specifically the Zero Flag, ZF) based on the result, without modifying either operand. It's commonly used to check if specific bits are set or if a value is zero.

section .data
    flags_byte db 01010101b

section .text
    global _start

_start:
    mov rax, flags_byte    ; Load the address of flags_byte into RAX
    test byte [rax], 00000001b ; Check if the least significant bit is set
    jnz .lsb_is_set        ; Jump if not zero (i.e., bit is set)
    ; ... code if LSB is not set

.lsb_is_set:
    ; ... code if LSB is set
    mov rax, 60            ; sys_exit
    xor rdi, rdi           ; exit code 0
    syscall

Using TEST to check a specific bit in a byte pointed to by RAX.