Reptilian

This page lists a few important crumbs for modifying the Linux kernel.

There are three major steps to adding a system call to the Linux kernel:

1. Entry to the system call table for the appropriate architecture
2. Prototype of system call & marking so parameters do not reside in registers (only the CPU stack)
3. Definition of system call (implementation)

Every system call must have an associated system call number which corresponds to its position in the system call table. This table is transformed / imported into the source as part of the build process. The correct table varies by architecture, but is in the following format/location in the source tree:

<kernelsourcedir>/arch/<archname>/entry/syscalls/syscall_<variant>.tbl

For example, in the x64 version of Linux, <archname> is x86 and <variant> is 64.

The table has four columns: <call_number> <abi> <call_name> <entry_point>

• When the syscall machine instruction executes, it loads a unique call number to identify the function.
• The ABI (application binary interface) for ABI-specific calls. Most calls use “common” (32- and 64-bit).
• A unique call name is the programmer-readable version of the system call used to define it later.
• Finally, the entry point is the assembly signature for the call (which uses the arch-specific prefix).

This example uses the “common” ABI and appends the x64 prefix (__x64_sys_) for the system call entry point:

435	common	sample_syscall	__x64_sys_sample_syscall

The kernel has no “safety” checks like user-space programs; registers can easily (and often are) corrupted. Instead, we should tell the compiler to limit the function to the CPU stack only and avoid dependence on registers. This can be done using the asmlinkage flag in GCC (the compiler we use) as part of the system call's prototype (which is used in C to declare a function which is defined elsewhere). The declaration must be in a place that makes it accessible to the definition. If you explore the kernel, you'll find some examples like this:

asmlinkage int sys_sample_syscall(int sample_param);

Finally, we need to add the system call definition. Thise will need to be compiled into the kernel as well. The kernel has lots of examples along these lines, which is for a system call with one parameter (int sample_param):

SYSCALL_DEFINE1(sample_syscall, int, sample_param)
{
    return sample_param + 1; // Adds 1 to the parameter and returns it
}

The SYSCALL_DEFINE<#> pattern supports between 0 (SYSCALL_DEFINE0) and 6 (SYSCALL_DEFINE6) parameters.