File descriptors and low-level I/O (open/read/write)

A file descriptor is not a file. It is a small integer index into a per-process table maintained by the kernel. That table entry points at an open file description: the kernel object that carries the current file offset, status flags, access mode, and a reference to the underlying file, pipe, socket, terminal, or device. open creates a new descriptor, read and write move bytes through it, and close drops your reference.

The reset: fd = 3 means "slot 3 in this process's descriptor table." The slot is not the file; it points at kernel state. dup creates another slot pointing at the same state, while a separate open creates a separate state object.

The Unix I/O model

Unix tries to make many I/O objects look like byte streams behind the same tiny API:

int fd = open("data.bin", O_RDONLY);
ssize_t n = read(fd, buf, sizeof buf);
ssize_t m = write(STDOUT_FILENO, buf, (size_t)n);
close(fd);

Regular files, pipes, sockets, terminals, /dev/null, and many device nodes all fit through the descriptor interface. The object type still matters: regular files support lseek; pipes and sockets do not. A terminal can block on human input. A socket can return EAGAIN in nonblocking mode. But the shape is stable enough that shell redirection works: a program writes to fd 1, and the parent process decides whether fd 1 points at a terminal, a file, or a pipe before exec.

Every process normally starts with three descriptors:

They are conventions, not magic. A shell can close fd 1 and duplicate a file onto slot 1 before running your program. Your code still calls write(1, ...); the kernel sends the bytes wherever slot 1 points.

Descriptor table vs open file description

The distinction is the whole note:

Descriptor flags live on the descriptor table entry. FD_CLOEXEC is the important one: close this descriptor during successful exec. File status flags live on the open file description. O_APPEND, O_NONBLOCK, and the current file offset are shared by descriptors that point at the same open file description.

That is why dup(fd) is not the same as open(path) again. dup returns a new descriptor that points at the same open file description, so both descriptors share the same offset. A second open returns a descriptor pointing at a new open file description, so its offset starts independently.

This also explains fork. After fork, parent and child have separate descriptor tables, but the entries refer to the same open file descriptions. If both processes write to the same inherited regular-file description, they share the offset. If they inherited the two ends of a pipe, they share pipe kernel state. This is the plumbing behind shells, pipelines, supervisors, and test harnesses.

open: path lookup plus flags

open(path, flags, mode) asks the kernel to resolve a pathname and create an open file description. If it succeeds, the kernel installs a descriptor table entry and returns the lowest unused descriptor number in the process.

Common flags:

The mode argument is not the final permission bits. It is filtered by the process umask. open("x", O_CREAT | O_WRONLY, 0666) might create 0644 if the umask removes group/other write permission.

Prefer O_CLOEXEC when creating descriptors. Setting FD_CLOEXEC later with fcntl works in single-threaded code, but in multi-threaded programs there is a race: another thread could fork and exec between open and fcntl, leaking the fd into the new program. Atomic close-on-exec is small hygiene that prevents strange bugs.

read and write: bytes, counts, and interruption

read(fd, buf, count) asks for up to count bytes. It can return:

• a positive count: bytes actually read, possibly less than requested;
• 0: end-of-file for regular files, or peer closed for some stream objects;
• -1: error, with errno explaining why.

write(fd, buf, count) asks the kernel to consume up to count bytes. It can return:

• a positive count: bytes actually written, possibly less than requested;
• -1: error, with errno explaining why.

The dangerous word is up to. Regular-file reads often fill the requested buffer until EOF, and regular-file writes often complete fully, but the API does not promise that in general. Pipes, sockets, terminals, signals, quotas, nonblocking mode, and resource limits all make short I/O part of the contract. Robust code writes loops like this:

static void write_all(int fd, const char *buf, size_t len) {
    while (len > 0) {
        ssize_t n = write(fd, buf, len);
        if (n < 0) {
            if (errno == EINTR) {
                continue;
            }
            die("write");
        }
        buf += (size_t)n;
        len -= (size_t)n;
    }
}

Signals matter. A blocking read, write, or close can return -1 with errno == EINTR if a signal handler ran. Some platforms and flags restart some calls automatically; do not build correctness on that. Decide whether to retry, abort, or propagate interruption.

Offsets, lseek, and random access

Regular files have a current offset in the open file description. read starts there and advances it by the number of bytes read. write starts there and advances it by the number of bytes written, unless O_APPEND makes each write seek to end-of-file first. lseek(fd, off, whence) changes the offset without transferring bytes.

pread and pwrite are the useful escape hatch: they read or write at an explicit offset without changing the open file description's current offset. That makes them friendlier for multi-threaded code and for code where shared descriptor offsets would be a bug.

Pipes, sockets, and terminals are not seekable. lseek on them fails with ESPIPE because there is no random-access offset to move.

Lifetime: close, reuse, and leaks

close(fd) releases one descriptor table entry. If that was the last reference to the open file description, the kernel can release the underlying open-file state. Descriptor numbers are reused aggressively: after you close fd 4, the next open or dup may return 4 again. That is why stale fd bugs are nasty. The integer can look valid while now pointing at a completely different object.

close can report errors, especially for network filesystems or delayed writeback. The hard part is that retrying close(fd) is usually wrong on modern Unix systems: the fd number may already have been released and reused, so a retry could close someone else's new descriptor. Check the error for diagnostics, but design writes and fsync/fdatasync so important data errors surface before the final close when possible.

Descriptors also leak across exec unless close-on-exec is set. This links directly to the previous note: a child can deliberately inherit fd 4 as a pipe, but a server should not accidentally hand a private listening socket or secret file to an executed helper.

Stdio is a layer above descriptors

FILE * streams from <stdio.h> wrap file descriptors with user-space buffering, formatting, and convenience. printf may not call write immediately; it may copy bytes into a libc buffer and flush later. fread may read more bytes from the kernel than your code asked for and keep the extra bytes in user space.

This is why mixing descriptor I/O and stdio on the same underlying fd is dangerous unless you flush and coordinate carefully. If you call read(fd, ...) behind a FILE * that has already buffered data, the kernel offset and libc's buffered view can surprise you.

The bridge functions exist: fileno(FILE *) gets the underlying fd, and fdopen(fd, "r") wraps an existing descriptor in a stream. Use them deliberately, with one buffering owner at a time.

Executable artifact: the offset is shared

The example lives in examples/systems-programming/file-descriptors-and-low-level-io/. It creates a small file, writes bytes with write_all, duplicates the fd, reads through both descriptors, opens the same path separately, and finally shows that descriptor numbers are reused after close.

The heart of the demo:

int fd = open(path, O_CREAT | O_TRUNC | O_RDWR | O_CLOEXEC, 0600);
write_all(fd, "abcdef\n", 7);
lseek(fd, 0, SEEK_SET);

int dupfd = dup(fd);
read_and_print("original fd", fd, 2);  /* reads "ab", offset becomes 2 */
read_and_print("dup fd     ", dupfd, 2); /* reads "cd", shared offset */

int separate = open(path, O_RDONLY | O_CLOEXEC);
read_and_print("separate fd", separate, 2); /* reads "ab", independent */

Compile and run:

cd examples/systems-programming/file-descriptors-and-low-level-io
./run.sh

Real output from this machine:

== build ==
== run ==
open("fd-demo-data.txt") -> fd=3, close-on-exec=yes
write_all wrote 7 bytes
dup(fd) -> fd=4 (same open file description)
original fd read 2 bytes: "ab"; offset now 2
dup fd      read 2 bytes: "cd"; offset now 4
open("fd-demo-data.txt") again -> fd=5 (new open file description)
separate fd read 2 bytes: "ab"; offset now 2
after close(4), open("/dev/null") -> fd=4

Read it carefully: dupfd did not read "ab" because it shared the original open file description and therefore shared the offset. The separate open did read "ab" because it created a new open file description. The /dev/null open reused fd number 4 because descriptor numbers are just table slots.

macOS/BSD vs Linux: POSIX core, platform extensions

The core descriptor model is portable Unix: descriptors, open, read, write, lseek, dup, fcntl, close, and close-on-exec exist across POSIX systems. The portable contract is strong enough for shells, C programs, and most systems tools.

The extension layer differs:

• Linux has Linux-only fd-producing APIs. eventfd, timerfd, signalfd, pidfd, memfd_create, epoll, openat2, and O_PATH are powerful but not portable.
• BSD/macOS use different readiness mechanisms. kqueue is the native BSD/macOS readiness interface; Linux uses epoll. Both operate on descriptors, but their APIs and event models differ.
• Atomic close-on-exec support varies by API. Prefer O_CLOEXEC and fd-creating functions with close-on-exec variants where available. If a platform lacks a variant, repair with fcntl and understand the multi-threaded race.
• Some flags have different details. O_DIRECT, O_SYNC, F_FULLFSYNC, advisory locks, and file cloning APIs differ enough that serious portable code isolates them.

The good rule: write the common path against POSIX, isolate platform acceleration, and keep the descriptor lifetime model the same everywhere.

Failure modes & trade-offs

• Treating fd numbers as stable identities. They are reusable slots. After close, never keep using the integer.
• Forgetting close-on-exec. Accidental fd inheritance can keep pipes from reaching EOF, leak secrets, or keep sockets open long after the parent thought it closed them.
• Assuming write writes everything. Short writes are legal. Use a loop for buffers that must be fully written.
• Assuming read fills the buffer. read returns what is available, what fits, or what the object can provide before EOF/interruption.
• Ignoring EINTR and EAGAIN. Blocking and nonblocking descriptors have different retry rules. Design them explicitly.
• Mixing stdio and raw descriptor I/O casually. Buffered user-space state and kernel offsets can diverge in ways that look like missing or duplicated bytes.
• Sharing offsets accidentally. dup, fork, and descriptor passing can make two code paths share one open file description. Use separate open, pread, or pwrite when independent offsets matter.
• Retrying close blindly. The fd number may already be reused. Report close errors, but do not write a loop that can close an unrelated descriptor.

In practice

• Check every syscall return. open, read, write, lseek, dup, fcntl, and close all fail in real programs.
• Make ownership visible. If a function takes an fd, state whether it borrows it or consumes it and closes it.
• Set close-on-exec at creation. Use O_CLOEXEC or equivalent APIs by default.
• Use dup2 or dup3 for deliberate redirection. This is how shells put a file or pipe onto fd 0, 1, or 2 before exec.
• Use pread/pwrite for random access in shared code. They avoid hidden offset coupling.
• Use stdio for formatted buffered text, descriptors for OS boundaries. Both are good tools; confusion starts when one fd has two owners.
• Think like the kernel you will build. A descriptor table, an open-file table, reference counts, offsets, flags, and close-on-exec bits are the minimum shape of Unix-like I/O.

ARM64 appendix

Nothing about the descriptor model is x86-64-specific. The C API and kernel objects are the same idea on ARM64 Linux and Apple Silicon. What changes is the syscall entry layer: ARM64 Linux uses svc #0 with ARM64 syscall numbers, while x86-64 Linux uses syscall with the x86-64 table. macOS keeps the stable boundary at libSystem rather than raw syscall numbers.

For the OS project, the architecture-specific part is how user mode enters the kernel and how arguments arrive. The descriptor machinery after dispatch is mostly architecture neutral: validate the fd, find the open file description, perform the object-specific operation, update offsets, and return a count or error.

Connects to: Systems Programming · The syscall: crossing the boundary into the kernel · Processes: fork, exec, and wait · The minimal standard library · What a pointer really is · OS from Scratch

Sources

• Michael Kerrisk - The Linux Programming Interface, ch. 4-5 - the universal I/O model, file descriptors, open file descriptions, and descriptor duplication. https://man7.org/tlpi/
• open(2) - man7.org - flags, modes, O_CLOEXEC, open file descriptions, and Linux-specific extensions. https://man7.org/linux/man-pages/man2/open.2.html
• read(2) - man7.org - return values, short reads, errors, interruption, and object-specific behavior. https://man7.org/linux/man-pages/man2/read.2.html
• write(2) - man7.org - partial writes, errors, interruption, and atomicity notes. https://man7.org/linux/man-pages/man2/write.2.html
• lseek(2) - man7.org - file offsets, seekability, sparse files, and ESPIPE. https://man7.org/linux/man-pages/man2/lseek.2.html
• dup(2) - man7.org - duplicated descriptors and shared open file descriptions. https://man7.org/linux/man-pages/man2/dup.2.html
• fcntl(2) - man7.org - descriptor flags such as FD_CLOEXEC and file status flag manipulation. https://man7.org/linux/man-pages/man2/fcntl.2.html
• Stevens & Rago - Advanced Programming in the UNIX Environment (APUE) - classic treatment of Unix file I/O, descriptor lifetime, and stdio interactions. http://www.apuebook.com/