What is the difference between system call and library call?

Looking at the comment, let me try this. A system service is a procedure that executes with elevated privilege (usually kernel mode). Everything else is a library call.

The underlying hardware provides a gate for user applications to enter kernel mode. The operating system guards this gate for basic system security.

Doing a little bit of simplification here --- a common method used by processors is for the process to explicitly trigger an exception (Intel has an alternate SYSCALL method).

A system will have a set of interrupt/exception vectors (pointer to handler procedures) for responding to exception or interrupts (e.g. page fault, divide by zero). The system will define some set of vectors (usually low numbered ones) for hardware exceptions and interrupts. However, they usually leave slots for the operating system to use.

An instruction something like:

   INT #12

Will explicitly trigger an exception and invoke the 12th procedure in the vector. A system might allow you to simulate a divide by zero exception doing this

Let us assume that the operating system uses vector 123 for system services.

   INT #123

would call a system service. A system might reserve a separate vector for each system service or it could use one and dispatch.

So you would do something like this:

  MOVL  #23, R0
  INT  #123

The value 23 in Register 0 tells the interrupt handler to invoke system service #23.

So you can see that this all takes assembly language. What every operating system does is create wrapper that can be called like functions from high level languages.

This is then the sequence of what happens:

1. A user calls the named wrapper with normal parameters. The wrapper sets up the registers and stack for the system service.

2. The wrapper triggers the exception that dispatches to the system service.

3. The system service then has to check ALL the parameters. This is one of the reasons system services have high overhead. Exceptions in kernel mode cause blue screens of death. If the system service needs to write to a buffer supplied by the user, it needs to ensure that every byte it writes to is writeable memory.

4. The system service does whatever it needs to do.

5. The system service executes a hardware instruction to return from an exception or interrupt. That returns to user mode and back to the wrapper function.

6. The wrapper may unpack parameters returned in registers.

7. The wrapper returns to the caller.

Low level kernel calls which are handled by the kernel are the system calls.

The man page says:

2 System calls (functions provided by the kernel)

3 Library calls (functions within program libraries)

A pictorial image can make it clear:

and