Why did the command ":(){ :|: & };:" make my system lag so badly I had to reboot?

This is called a fork bomb.

:() means you are defining a function called :

{:|: &} means run the function : and send its output to the : function again and run that in the background.

The ; is a command separator.

: runs the function the first time.

Essentially you are creating a function that calls itself twice every call and doesn't have any way to terminate itself. It will keep doubling up until you run out of system resources.

Running in Virtualbox was quite sensible really otherwise you would have had to restart your pc.

This is a so called fork bomb implemented in shell.

from wikipedia:

:(){ :|:& };:
\_/| |||| ||\- ... the function ':', initiating a chain-reaction: each ':' will start    two more.
 | | |||| |\- Definition ends now, to be able to run ...
 | | |||| \- End of function-block
 | | |||\- disown the functions (make them a background process), so that the children    of a parent
 | | |||   will not be killed when the parent gets auto-killed
 | | ||\- ... another copy of the ':'-function, which has to be loaded into memory.
 | | ||   So, ':|:' simply loads two copies of the function, whenever ':' is called
 | | |\- ... and pipe its output to ...
 | | \- Load a copy of the function ':' into memory ...
 | \- Begin of function-definition
 \- Define the function ':' without any parameters '()' as follows:

That command is a well known version of the fork bomb

It causes your computer to run out of memory by forking a process infinitely. There exist some safeguards you can use against it as well:

Unix-type systems typically have a process-limit, controlled by a ulimit shell command or its successor, setrlimit. Linux kernels set and enforce the RLIMIT_NPROC rlimit ("resource limit") of a process. If a process tries to perform a fork and the user that owns that process already owns RLIMIT_NPROC processes, then the fork fails. Additionally, on Linux or *BSD, one can edit the pam_limits config file /etc/security/limits.conf to the same effect. However, not all distributions of Linux have the pam_limits module installed by default.