A calling convention is the set of rules that two pieces of compiled code must agree on so that one can call the other as a function. When code calls a function, both sides need to know exactly where the arguments go, where the return value comes back, how the stack is arranged, and which registers the called function is allowed to clobber versus which it must leave unchanged. None of this is implied by the source language; it is a low-level contract that the compiler, the operating system, and any hand-written assembly must all follow identically, or the call will corrupt data and crash.
The convention answers a few concrete questions. Where do arguments live, in specific registers, on the stack, or a mix? Who cleans up the stack space used for arguments, the caller or the callee? Where is the return value? And critically, which registers are caller-saved (volatile, so the caller must preserve them itself if it still needs their values) versus callee-saved (the called function must restore them before returning)? The classic cdecl convention on 32-bit x86, for instance, passes arguments on the stack and has the caller clean up, while other conventions pass some arguments in registers.
The dominant convention on modern Unix-like x86-64 systems is defined by the “System V Application Binary Interface, AMD64 Architecture Processor Supplement.” It specifies that the first six integer or pointer arguments are passed in the registers rdi, rsi, rdx, rcx, r8, and r9, with further arguments placed on the stack, and that the return value comes back in rax. It also fixes which registers are volatile, including rax, rcx, rdx, rsi, rdi, and r8 through r11, and which are callee-saved, including rbx, rbp, and r12 through r15. Because every compiler and library on the platform follows the same document, code built by different toolchains interoperates cleanly.
A calling convention is one component of a larger agreement called the Application Binary Interface, or ABI. The ABI covers everything two compiled modules must share to work together at the binary level, including data type sizes and alignment, struct layout, the object-file format, and the calling convention itself. Where a programming language’s API describes how to use a library in source code, the ABI describes how the compiled artifacts fit together in memory and across function-call boundaries at runtime. A calling convention is the part of the ABI that governs the function call specifically.
Conventions matter precisely because they are invisible when correct and catastrophic when violated. A program that links a library compiled for a different convention may pass arguments in the wrong place or fail to preserve a register the caller depended on, producing bugs that look like memory corruption rather than a mismatch of rules. Standardizing the convention in a published ABI is what lets a single compiled library serve programs written in many languages, what makes separate compilation possible, and what allows system calls and dynamic linking to function across an entire operating system.