The syscall: crossing the boundary into the kernel

Your process cannot open a file, read a byte from disk, or send a packet. It can only compute over memory it already owns. Everything else belongs to the kernel, and the syscall is the single doorway: your code loads a syscall number and arguments into agreed registers, executes one special instruction, and the CPU switches to kernel mode at an entry point the kernel installed at boot. read(), malloc growing the heap, printf flushing a buffer — every effect that leaves your process bottoms out here.

The reset: a syscall is not a function call into the kernel. It is a controlled trap — a CPU mode switch to one fixed kernel entry point, with arguments passed in registers according to a contract the kernel publishes.

The boundary: privilege rings

x86-64 CPUs run code at privilege levels; in practice only two matter: ring 0 (kernel) and ring 3 (user). Ring 3 code cannot execute privileged instructions, talk to devices, or touch kernel memory — the page tables map kernel pages as inaccessible to user mode. This is what makes an OS an OS: a buggy or hostile process can be contained because the hardware, not convention, enforces the boundary.

The consequence: user code cannot "call" kernel code, because a plain call to a kernel address would fault. Instead the CPU provides trap instructions (syscall on x86-64, svc on ARM64) that atomically raise the privilege level and transfer control to one kernel-chosen address. The kernel decides what to do by looking at the syscall number you left in a register — you never choose the destination, only the request.

How it really works: from read() to syscall

When you call read(fd, buf, len), you're calling a small libc wrapper function. On x86-64 Linux it does approximately this:

• Move the syscall number for read (0) into rax.
• Arguments are already almost in place: the System V function-call convention passes them in rdi, rsi, rdx, rcx, r8, r9 — the syscall convention is the same except the 4th argument goes in r10, because the syscall instruction itself clobbers rcx (it saves the return rip there) and r11 (saved rflags).
• Execute syscall. The CPU jumps to the kernel's entry point in ring 0.
• The kernel dispatches on rax, does the work, puts the result back in rax, and returns to user mode with sysret.

The register contract on x86-64 Linux:

The -errno convention. The kernel does not know about the errno variable — that is a libc fiction. On Linux, a failing syscall returns a small negative number in rax: read on a bad file descriptor returns -9, which is -EBADF. The libc wrapper checks whether the result is in [-4095, -1]; if so, it negates it, stores it in the thread-local errno, and returns -1 to you. That is the entire mechanism behind "returns -1 and sets errno". XNU (macOS) signals errors differently: the kernel returns the positive errno value and sets the carry flag, and libSystem's wrapper does the translation — same fiction, different contract.

Wrappers, syscall(2), and the vDSO

There are three depths at which you can make a syscall, plus one case where you don't cross at all:

• The libc wrapper (read, write, mmap…) — what you should use. It knows the numbers, does the errno translation, and sometimes adds real logic (e.g. wrappers that call a newer syscall and fall back to an older one).
• syscall(2), the generic wrapper — you supply the number (SYS_gettid, …) and raw arguments; libc still performs the trap and errno translation. Useful for Linux syscalls that have no wrapper yet.
• The raw instruction — inline asm loading registers yourself. Legitimate inside libc implementations, freestanding code, and teaching demos like the one below; everywhere else it's a portability bug waiting to happen.
• The vDSO — Linux maps a small shared object (the virtual dynamic shared object) into every process, containing user-space implementations of hot, read-mostly syscalls: gettimeofday, clock_gettime, time, getcpu on x86-64. The kernel keeps a data page updated; the vDSO function just reads it. That's why a tight gettimeofday loop shows nothing in strace — no crossing happens. macOS does the analogous trick with the commpage that libSystem reads.

Why avoid crossing? A syscall is far more expensive than a function call: mode switch, speculation barriers (post-Spectre/Meltdown mitigations made this worse), cache and TLB pollution, and on return the kernel may reschedule you. Tens to hundreds of nanoseconds versus ~1 ns for a call — which is exactly why buffered I/O in libc exists: fwrite batches many small writes into one write crossing.

Executable artifact: one write(), three ways down

The example lives in examples/systems-programming/the-syscall-crossing-into-the-kernel/. It performs the same write through the libc wrapper, through syscall(2), and through the raw trap instruction in inline asm (x86-64 and ARM64, Linux and macOS variants), then makes read fail on purpose to expose what the kernel really returns before libc dresses it up as errno. The heart of it:

// Raw 3-argument syscall, no libc: load the registers the kernel contract
// names, execute the trap instruction.
static long raw_syscall3(long number, long arg1, long arg2, long arg3,
                         int *is_error) {
#if defined(__x86_64__)
    long result;
    char carry;
    __asm__ volatile(
        "syscall\n\t"
        "setc %1" /* XNU reports errors in the carry flag */
        : "=a"(result), "=r"(carry)
        : "a"(number), "D"(arg1), "S"(arg2), "d"(arg3)
        : "rcx", "r11", "memory");
#if defined(__APPLE__)
    *is_error = carry;
#else
    (void)carry; /* Linux: error iff result is a small negative (-errno) */
    *is_error = (unsigned long)result >= (unsigned long)-4095L;
#endif
    return result;
#endif
}

Compile and run:

gcc -Wall -Wextra demo.c -o demo && ./demo

Real output from this machine (an x86-64 Mac, so the numbers are XNU's):

1: libc wrapper write()
   write() returned 24
2: generic syscall(2) wrapper
   syscall(SYS_write=4, ...) returned 30
3: raw trap instruction, no libc
   raw syscall 0x2000004 returned 33 (is_error=0)
errno: libc read(-1,..) -> ret=-1, errno=9 (Bad file descriptor)
errno: raw  read(-1,..) -> ret=9, is_error=1 (EBADF is 9)

Read the last two lines together: libc reported the polite fiction (-1, errno = 9), while the raw syscall shows what XNU actually handed back — a positive 9 plus the carry flag. On Linux the same raw call returns -9 and the first line's machinery (negate, store, return -1) is the whole story of errno. Also note SYS_write is 4 here and the raw number is 0x2000004 (XNU's BSD syscall class); on x86-64 Linux it is 1. Same source, different kernel contract.

Watching the crossing: strace and friends

The boundary is observable, because the kernel can log every trap a process makes:

• Linux: strace ./demo prints every syscall with decoded arguments and return values — read(3, "abc", 128) = 3. strace -c aggregates counts and time per syscall; ltrace shows library calls instead, which is exactly the wrapper-vs-syscall distinction of this note, live.
• macOS: dtruss ./demo (a DTrace script) is the rough equivalent, but SIP blocks it for most binaries unless partially disabled; dtrace/Instruments are the supported paths.
• A syscall trace is often the fastest way to answer "what is this program actually doing?" — no source needed. If a process is slow, strace -c telling you it made two million 1-byte write calls is a complete diagnosis.

macOS/BSD vs Linux: same idea, different contract

• Numbers differ per kernel. write is 1 on x86-64 Linux, 4 in XNU's BSD class, 64 on ARM64 Linux. A syscall number is meaningful only relative to one kernel — there is no portable numbering.
• Linux's syscall table is a stable ABI. Numbers are never reused or renumbered; static binaries making raw syscalls keep working across kernel upgrades. This is a deliberate, load-bearing kernel promise.
• macOS makes the opposite promise. The stable interface is libSystem, not the syscall table; Apple renumbers and changes kernel interfaces between releases, and syscall(2) is officially deprecated. Go famously had to switch from raw syscalls to libSystem calls on macOS for exactly this reason. Ship raw syscalls on macOS and a routine OS update can break you.
• Error signaling differs (-errno in the return value vs positive errno + carry flag), as the demo shows.

Failure modes & trade-offs

• Every syscall can fail; unchecked returns are latent bugs. write can return a short count on success — checking only != -1 is not enough.
• errno is only meaningful after a failure. Successful calls may leave stale values; test the return value first, then read errno.
• Raw syscalls skip libc's bookkeeping. Bypassing wrappers can desynchronize libc state (buffered stdio, pthread bookkeeping, the runtime's view of brk); fork via raw syscall famously skips pthread_atfork handlers.
• Syscall cost is a design constraint. Chatty crossings (byte-at-a-time I/O) are a classic performance failure; batching (buffered stdio, writev, larger buffers) is the standard fix.
• The boundary is also the security perimeter. Seccomp filters, sandboxes, and container runtimes reason about processes in terms of which syscalls they may make — another reason "what syscalls does this program make?" is worth knowing how to answer.

In practice

• Use the libc wrappers. Reach for syscall(2) only for Linux syscalls without wrappers, and for raw asm only in freestanding code — where you are the libc.
• strace first, debugger second when a program misbehaves at the OS boundary: wrong path, missing permission, surprising ENOENT — the trace shows it immediately.
• Learn to read man 2 pages. Section 2 is the syscall contract: arguments, return value, and the exact errno values each call can produce.
• Count crossings in hot paths. If profiling shows kernel time, ask "which syscalls, how often, how small?" before optimizing user-space code.
• This boundary is what you will build in the OS project. Installing the entry point, dispatching on the number, returning to user mode — the other side of this note.

ARM64 appendix

Same shape, different names. On ARM64 Linux the trap instruction is svc #0 (supervisor call): syscall number in x8, arguments in x0–x5, return value in x0, with the same -errno convention. Numbers differ from x86-64 (write is 64, not 1) because the ARM64 port started from a clean, unified table. On Apple Silicon, XNU takes the number in x16 and the trap is svc #0x80, with the carry-flag error convention — the demo's __aarch64__ branches show both side by side:

#if defined(__APPLE__)
    register long x16 __asm__("x16") = number; /* XNU: number in x16 */
    __asm__ volatile("svc #0x80\n\t"
                     "cset %1, cs" ...);
#else
    register long x8 __asm__("x8") = number; /* Linux: number in x8 */
    __asm__ volatile("svc #0" ...);
#endif

Connects to: Systems Programming · The process address space & virtual memory · Registers & the ISA · System V AMD64 calling convention · The minimal standard library · OS from Scratch

Sources

• Michael Kerrisk — The Linux Programming Interface, ch. 3 "System Programming Concepts" — the syscall bible; the trap mechanism, wrapper functions, and error handling in depth. https://man7.org/tlpi/
• syscall(2) — man7.org — the generic wrapper, plus the authoritative per-architecture table of trap instructions and register conventions (number, args, return). https://man7.org/linux/man-pages/man2/syscall.2.html
• syscalls(2) — man7.org — the catalogue of Linux syscalls and the kernel's ABI-stability promise. https://man7.org/linux/man-pages/man2/syscalls.2.html
• vdso(7) — man7.org — why some "syscalls" never enter the kernel, and which ones per architecture. https://man7.org/linux/man-pages/man7/vdso.7.html
• System V AMD64 ABI — appendix A documents the Linux x86-64 syscall calling convention alongside the function-call convention it mirrors. https://gitlab.com/x86-psABIs/x86-64-ABI
• Stevens & Rago — Advanced Programming in the UNIX Environment (APUE) — the classic treatment of the syscall/library-function distinction and errno discipline. http://www.apuebook.com/
• XNU syscalls.master — Apple's actual BSD syscall table, where the macOS numbers in the demo come from. https://github.com/apple-oss-distributions/xnu/blob/main/bsd/kern/syscalls.master