What stops an assembly program from crashing the operating system?
In the end, all programs are machine code, regardless of whether the source language was assembler or a high-level language.
The important thing is that there are hardware mechanisms that limit what a given process can do, including "messing with" registers that could affect other programs or the operating system itself.
This started with a simple distinction between "user" and "supervisor" modes of operation, and has since evolved into a security architecture with multiple "rings" of privilege.
Here's a very generic example to make it a little more concrete:
In "user mode", a process cannot access memory that hasn't been assigned to its process ID. Memory assigned to other processes and the operating system itself is blocked. This includes the values of registers used by those other processes. This is enforced by the MMU hardware.
Therefore, in "user mode", a process cannot access the MMU control registers.
And obviously, in "user mode", a process cannot change the mode to "supervisor mode" except through a very well-defined mechanism that involves calling an operating system function.
The operating system gets control if the process tries to break any of these rules. Once that happens, the OS can simply halt the offending process, never executing any more of its instructions.
Many multitasking operating systems use a data structure called a Process Control Block (PCB) to take care of the register overwrite problem. When you run your code, the OS creates a new process to keep track of it. The PCB contains information about your process and space allocated to hold register contents. Let's say process A is currently running on the processor and your code is in process B. What happens when you run your code goes something lke this:
Process A's state data (register contents, program counter, etc.) are copied into its PCB.
Process B's state data are copied from its PCB into the CPU registers
Process B runs on the processor until it finishes or is preempted
Process B's state data are copied back into its PCB
Process A's state data are copied back into the CPU and it continues running
If the operating system is running as process A, then you can see how saving its registers before your program runs then copying them back into the CPU once your program ends prevents user programs from messing with what's going in other processes.
To avoid user processes writing over OS data in memory, most platforms use memory segmentation. Basically, using virtual memory, the address space a process sees can be mapped to any arbitrary range of physical addresses. As long as the processes' physical memory spaces don't overlap, it's impossible for one process to overwrite another's data.
Of course, different OSes do things differently, so this won't apply in every case. Also, on most platforms, OS processes run in a different mode from user processes and there are some intricacies there.
Depends on what platform you're talking about.
On a more basic processor, the processor just executes whatever instructions the program tells it to execute.
On a more sophisticated processor, there's (at least) two modes. In one mode, the processor does whatever the program tells it to do. In the other mode, the processor itself refuses to execute certain instructions.
What stops a program crashing the whole system? With the first type of processor, the answer is "nothing". With a basic processor, a single rogue program can indeed crash the entire system. All the early 8-bit home computers, and many of the 16-bit ones, fall into this category.
On a modern PC, the processor has "protection" hardware. Basically the OS runs in a special mode that lets it do anything, whereas normal programs run in a mode where the processor will only allow certain actions (based on what settings the OS has configured on the processor). Stuff like only accessing certain registers, or only accessing particular memory ranges.
Obviously, allowing a single rogue program to crash the entire system is bad. (There are also severe security implications in letting a rogue program access whatever data it wants.) To avoid this, you need two things:
A processor that actually has protection hardware (i.e., it can be configured to refuse to execute certain instructions).
An OS that actually uses these facilities to protect itself. (It's no good having a chip with protection circuitry if the OS never uses it!)
Just about any modern desktop OS you can name (Windows, Linux, Mac OS, BSD...) it a protected-mode OS running on a processor with protection hardware. If you're doing embedded development on some 8-bit microcontroller, that probably doesn't have any protection hardware. (Or any OS, for that matter...)