Hardware Knowledge You Have to Know Before Assembly

Summary

After leaving the abstract world of high-level programming languages, you plan to start with assembly.

In assembly, you need to manipulate registers and memory directly. Not only that, but you also have to deal with stack alignment, System V, call stack, and so on.

Prerequisite knowledge is crucial, otherwise you will miss out on truely on valuable ideas.

The least important thing is the language itself

What if you plan to start Reverse Engineering of Pwn Challenge, here is a guide for beginners like you

Registers

The most intuitive concept is registers.

Registers are the most fast, smallest storage unit located directly inside the CPU. They hold memory address, data and instructions.

Depending on the architecture, registers can be classified into several functions.

General-Purpose Registers

These registers can be used by developers to store return value, temporary data, function arguments or operands for arithmetic operations.

General-Purpose Registers you can operate on them.

Such like these instructions:

push rbp
mov rbp, rsp
pop rdi

You can easily modify the value these registers store.

Control Registers

Control Registers in the x86 architecture determine the global operating mode of the processor and control system-level features such as virtual memory management (paging), processor protection rings, caching strategies, and task switching.

PC

The most basic control register is PC (Program Counter). Its name has historical reasons. Learn more: Wiki

RIP is the register which was implemented in x86_64. It store the current instruction address.

Also, there are different name of PC register in different architectures.

ARM: R15

RISC-V: PC

PowerPC: IAR/NIA

CR0-CR8 (x86)

CR0 (System Status & Mode Switch): Controls the foundational operating modes and memory policies of the CPU. It manages the transition from Real Mode to Protected Mode (via the PE bit), enables Paging (via the PG bit), and enforces Write Protection (WP) to prevent Ring 0 from modifying read-only pages.

CR3 (Page Directory Base Register - PDBR): Holds the physical address of the root page table for the current process. It is the anchor for the Memory Management Unit (MMU) during address translation; loading a new value into CR3 effectively switches the virtual memory context during a process context switch.

CR2 (Page Fault Linear Address): Serves as a diagnostic register that automatically stores the exact linear (virtual) address that triggered a Page Fault (#PF). The kernel’s exception handler reads this register to determine which memory address needs to be resolved or swapped in.

CR4 (Architectural Extensions & Security): Toggles advanced hardware capabilities and critical kernel-level security mitigations. This includes enabling Physical Address Extension (PAE) and activating hardware defenses like SMEP (Supervisor Mode Execution Prevention) and SMAP (Supervisor Mode Access Prevention) to block local privilege escalation exploits.

CR8 (Task Priority Register): Available exclusively in 64-bit long mode to manage interrupt prioritization via the Local APIC, allowing the kernel to mask external interrupts below a specific priority threshold.

Register File

In computer architecture, the Register File is a small, ultra-fast array of internal storage cells built directly into the CPU processor core. It holds the immediate operands and results that the Execution Units (like the ALU) need to read and write during every clock cycle.

Actually, Register File Implement Registers, when you operate on rbp or rax by asm, the register file implements the underlying circuit.

You don’t need to figure out the Implementation principle of register file at first.

ELF

ELF(Executable and Linkable File)，which is the default executable format in GNU/Linux.

There are different executable format in different operating system.

GNU/Linux: ELF

Windows: PE

Darwin(Mac OS): Mach-O

ELF is the most popular format, so I just discuss ELF.

Section (ELF)

const char *str = "Hello World";

This line of C code declares a pointer which point to a string. This string was stored in .rodata section.

Don’t be obsessed with note the name and functions of sections.

The core idea of this is “everything is in the memory, sections are a kind of format to organize data”

objdump is a tool to easily show each section

objdump -s -j .rodata a.out

-s dumps the raw bytes, -j selects which section to dump.

Sections are the linker’s view of the file. When gcc a.c b.c -o a.out runs, the linker merges .text from both object files into a single .text in the output. Every section carries a name, a type, an address, and a flag telling what kind of data it holds.

But the linker does not run your program. The kernel does, and the kernel doesn’t care about section names. It only needs to know which range of the file to map into memory, at which virtual address, with which permission. That information lives in the program header table.

Segment (ELF)

readelf -l a.out

Program Headers:
  Type   Offset   VirtAddr           FileSiz  MemSiz   Flg Align
  LOAD   0x000000 0x0000000000400000 0x0001e4 0x0001e4 R   0x1000
  LOAD   0x001000 0x0000000000401000 0x000135 0x000135 R E 0x1000
  LOAD   0x002000 0x0000000000402000 0x000088 0x000130 RW  0x1000

Section to Segment mapping:
  Segment Sections...
   00     .interp .dynsym .dynstr .gnu.hash .rela.dyn
   01     .init .plt .text .fini
   02     .dynamic .got .got.plt .data .bss

.init, .plt, .text, .fini are four sections, but they all fall inside one segment, and that segment is marked R E. This is where “the text segment is executable” actually comes from.

So the same pile of bytes gets indexed twice:

The third LOAD line has FileSiz smaller than MemSiz. Those missing bytes are .bss: declared in the file with a size but with no content, so the loader just zeroes that range after mapping.

The advice from before still applies. Don’t try to memorize every section and segment name. ELF is not magic, it is two indexes over the same bytes, and there are two because there are two consumers.

Process Memory Layout

When you execute ./a.out, the kernel creates a new address space, walks the program headers, and maps each PT_LOAD segment into it. Then it builds the stack, loads the interpreter (/lib64/ld-linux-x86-64.so.2 for a dynamically linked binary), and jumps to the entry point.

What you end up with, from high address to low:

+--------------------+
| stack              |  argv, envp, auxv on top, grows downward
+--------------------+
| mmap region        |  libc.so, ld.so, large malloc, grows downward
+--------------------+
|       ...          |
+--------------------+
| heap (brk)         |  grows upward
+--------------------+
| RW LOAD segment    |  .data .bss .got .dynamic
+--------------------+
| RE LOAD segment    |  .text .plt
+--------------------+
| R  LOAD segment    |  .interp .dynsym .rodata
+--------------------+
low address

Two growth directions: the heap moves up, the stack moves down, and mmap sits in between. They only meet when you run out of address space.

Run cat /proc/self/maps and you will see exactly this layout for a live process, with the address of every region. In practice each region gets a random base at exec time, which is why the addresses you leak in a pwn challenge change on every run.

Endianness

Memory is addressed in bytes, but registers hold 8-byte values. Endianness decides which byte of that value goes to the lowest address. x86-64 is little-endian: least significant byte first.

the qword 0x0000000000401000 stored at 0x7ffd0000:

0x7ffd0000: 00
0x7ffd0001: 10
0x7ffd0002: 40
0x7ffd0003: 00
0x7ffd0004: 00
0x7ffd0005: 00
0x7ffd0006: 00
0x7ffd0007: 00

Why this matters when you read a hexdump: bytes appear in address order, so reconstructing a number means reading backwards. The first time you compare a register against a raw memory dump, the value will look scrambled.

Strings are unaffected, a byte sequence is a byte sequence. The confusion only starts when the same bytes are interpreted as an integer.

ARM and RISC-V can be configured either way by the boot firmware, so the byte order is something you have to confirm per platform. x86 is always little-endian.

Stack and Call Stack

Registers give you fast storage, memory gives you a lot of storage, but neither one remembers where to go back. When A calls B, B needs to know the address it was called from. That is the job of the call stack.

The stack is a region of memory managed by rsp, following LIFO:

push rax   ; rsp -= 8, then [rsp] = rax
pop  rax   ; rax = [rsp], then rsp += 8

Two instructions define everything about function calls:

call foo   ; push the address of the next instruction, then jump to foo
ret        ; pop that address into rip

call writes the return address on the stack, ret reads it back. The CPU has no idea what a function is.

Stack frame

Each function owns a slice of the stack called its stack frame. The standard prologue:

push rbp          ; save the caller's frame pointer
mov  rbp, rsp     ; the current stack top becomes the frame base
sub  rsp, 0x20    ; allocate room for local variables
...
mov  rsp, rbp     ; these two lines are usually written as a single
pop  rbp          ; leave instruction
ret

After mov rbp, rsp, everything in this frame is addressed as a fixed offset from rbp:

mov  [rbp-0x8], rdi    ; spill the first argument
mov  rax, [rbp-0x8]

That is why rbp is called the frame pointer. It does not move while the function runs, even though rsp moves every time something gets pushed. gdb’s info frame and bt depend on this: each frame stores the caller’s rbp, so walking backwards is just following rbp -> [rbp] -> [[rbp]].

This chain is not guaranteed. gcc compiles with -fomit-frame-pointer at -O2, rbp becomes an ordinary callee-saved register, and gdb has to recover frames by scanning for return addresses. Debugging an optimized binary is harder for exactly this reason.

Where the return address sits:

        [rbp+0x10]  first stack argument (the 7th parameter)
        [rbp+0x08]  return address        <- pushed by call
rbp ->  [rbp+0x00]  saved rbp             <- pushed by push rbp
        [rbp-0x08]  local variable
        ...
rsp ->

The 8 bytes between the saved rbp and the return address are the most important layout in pwn. Overwriting it means deciding where ret goes.

Calling Convention (System V AMD64 ABI)

The hardware only gives you call and ret. It says nothing about where arguments go. Two functions compiled separately must agree on something, and that agreement is the calling convention.

For GNU/Linux x86-64 it is the System V AMD64 ABI.

Integer and pointer arguments

The first six go in rdi, rsi, rdx, rcx, r8, r9. The rest go on the stack, pushed right to left, in 8-byte slots. The return value comes back in rax, using rdx:rax when the result is 128 bits.

long add(long a, long b, long c, long d, long e, long f, long g);

mov  rdi, a
mov  rsi, b
mov  rdx, c
mov  rcx, d
mov  r8,  e
mov  r9,  f
push g            ; 7th argument, on the stack
call add
; result in rax

Floating point arguments use xmm0 through xmm7, counted separately from the integer registers. f(1, 2.0, 3) ends up as rdi=1, xmm0=2.0, rsi=3.

Caller-saved and callee-saved

After a call, can you still trust rax? The ABI answers with two groups:

Caller-saved (volatile): rax, rcx, rdx, rsi, rdi, r8-r11, xmm0-xmm15

The caller must assume any call destroyed them.

Callee-saved (non-volatile): rbx, rbp, r12-r15, rsp

The callee must restore them before ret.

This is exactly why the prologue starts with push rbp. rbp belongs to the caller, so if you want it as your own frame base you have to save the old one and put it back before returning. The same applies to rbx and r12-r15, which is why you see them pushed at the top of functions all over libc.

It is a convention, nothing in the hardware enforces it. A function that clobbers rbx does not crash on the spot, it just breaks the contract, and the caller fails somewhere unrelated later.

Windows x64 does it differently: four integer arguments in rcx, rdx, r8, r9, plus 32 bytes of shadow space the caller reserves. Since the ABI is per operating system and not per architecture, the same CPU can have two different rules.

Stack Alignment

The ABI adds one more rule: rsp must be 16-byte aligned right before a call.

Trace it through:

before the call:  rsp = 0 (mod 16)
call pushes 8 bytes
on function entry: rsp = 8 (mod 16)
push rbp:         rsp = 0 (mod 16)
sub rsp, N:       N must keep it at 0 (mod 16)

So inside a normal function body, rsp stays 16-byte aligned. Compilers assume this silently. SSE instructions such as movaps (move aligned packed single) fault immediately when the operand address is not 16-byte aligned.

Hand-write an assembly routine that calls printf with a misaligned stack, and you get a SIGSEGV somewhere inside libc with no obvious clue about what went wrong. The bug is in your prologue.

A local fix:

push rbx             ; save a callee-saved register
mov  rbx, rsp        ; remember the original stack pointer
and  rsp, -16        ; force alignment
call puts
mov  rsp, rbx        ; restore it, nothing below was valid anyway
pop  rbx

There is one exception worth knowing: the red zone, the 128 bytes below rsp. Leaf functions may store data there without adjusting rsp at all, because signal handlers are required to leave it alone. This is why an optimized leaf function often never subtracts anything from rsp.

System Call

Everything so far stays inside your own address space. The moment your assembly wants to print something or exit, it has to leave.

On x86-64 Linux the instruction is syscall:

mov  rax, 1        ; syscall number: write
mov  rdi, 1        ; fd: stdout
lea  rsi, [rel msg]
mov  rdx, 3        ; count
syscall

The number table lives in the kernel header unistd_64.h.

The register layout is close to, but not the same as, the function calling convention:

syscall uses rcx to stash the return address and r11 to stash rflags, so the fourth argument has to move out of rcx into r10. Forgetting that is the classic beginner bug: the first three arguments work, the fourth arrives as garbage.

Wrap up

A few ideas carry the whole post:

• ELF is two indexes over the same bytes. Sections serve the linker, segments serve the kernel.
• The call stack is just memory, and call/ret are the only bookkeeping the hardware does for you.
• The calling convention is a contract between compilers, not something the CPU enforces.
• Stack alignment is a rule the compiler assumes silently, and breaks loudly.

When you open a pwn challenge in gdb or a disassembler, these are the things the disassembly is telling you. The push rbp at the top is the ABI. The odd sub rsp, 0x18 is alignment accounting. The address you are asked to leak lives in the mmap region of the layout above.

The syntax of the assembly is still the least important part.