How to delete a non-working kernel after update?

If the latest kernel update break things (Original question), the best option for most is to fall back to the previous working kernel (provided that you keep at least 1 fallback option).

Otherwise, use @ZAB 's solution to install a specific version of the kernel version provided via the apt package manager.

In generic, to remove the unwanted out-dated kernels, open a terminal session and run the command:

IMPORTANT: If /boot is NOT on its tiny partition (running out of space), avoid explicitly purging old kernels. Consider them backup / fallback options when upgrade introduces problems (hardware driver modules, etc.).

Good read on kernel upgrade and preservation mechanism: How does apt on Ubuntu decide how many old kernels to keep

TL;DR: Take a look at /etc/apt/apt.conf.d/01autoremove-kernels file generated by postinstall hook (script).

Snippet from Ubuntu 20.04 LTS running 5.4.0 for a better understanding:

// DO NOT EDIT! File autogenerated by /etc/kernel/postinst.d/apt-auto-removal
APT::NeverAutoRemove
{
   "^linux-.*-5\.4\.0-28-generic$";
   "^linux-.*-5\.4\.0-29-generic$";
   "^linux-.*-5\.4\.0-31-generic$";
   "^kfreebsd-.*-5\.4\.0-28-generic$";
   "^kfreebsd-.*-5\.4\.0-29-generic$";
   "^kfreebsd-.*-5\.4\.0-31-generic$";
   "^gnumach-.*-5\.4\.0-28-generic$";
   "^gnumach-.*-5\.4\.0-29-generic$";
   "^gnumach-.*-5\.4\.0-31-generic$";
   "^.*-modules-5\.4\.0-28-generic$";
   "^.*-modules-5\.4\.0-29-generic$";
   "^.*-modules-5\.4\.0-31-generic$";
   "^.*-kernel-5\.4\.0-28-generic$";
   "^.*-kernel-5\.4\.0-29-generic$";
   "^.*-kernel-5\.4\.0-31-generic$";
};

Find the kernel package name

dpkg -l | grep linux-image

At the time of writing, on amd64/x86_64 architecture the image name was: linux-image-$(uname -r)-generic

For unsigned kernel images, like those created by nvidia drivers, the image could be: linux-image-unsigned-4.20.17-042017-generic

Remove (purge - remove all configuration files as well)

sudo apt-get purge linux-image-3.5.0-40-generic

If you want to do a deep clean (leftover package configs...), USE WITH CAUTION:

dpkg -l | awk '/^rc/ { print $2 }' | xargs apt-get purge -y
dpkg -l | awk '/^rc/ { print $2 }' | xargs dpkg -P

NOTE: For Ubuntu 18.04 LTS or later, you may also want to remove any linux-modules or linux-headers packages of the same version to save disk spaces / keep you system lean.

Update - 2020-05-21

Explicitly purging old kernels may cause unexpected results. Not recommended unless /boot is on its own tiny partition (out of space).

Refer to @ZAB 's approach to get rid of the latest kernel update which introduces bug (break things).

Ubuntu wiki on Removing old kernels

There is no (I personally haven't found) easy equivalent to Fedora/RHEL/CentOS approach by setting installonly_limit=2 to tell DNF or YUM to only keep latest 2 kernels.

sudo apt-get remove linux-image-3.5.0-40*.

I will suggest you always keep at least one older kernel available, just in case you need to boot into it for reasons you may never expect right now.

Don't blindly do what was suggested in the accepted answer. It will break your system. Read what apt utility tells you before pressing y.

The following steps was done to remove the buggy kernel 5.3.0-53 for example. The previous working kernel was 5.3.0-51.

First run sudo apt remove linux-image-5.3.0-53-generic --verbose-versions and read the output:

...
The following NEW packages will be installed:
  linux-image-unsigned-5.3.0-53-generic (5.3.0.53.109)
...

We don't need this package, press n and ask to remove it as well sudo remove linux-image-5.3.0-53-generic linux-image-unsigned-5.3.0-53-generic --verbose-versions now prints this:

....
The following packages will be REMOVED:
 linux-generic-hwe-18.04 (5.3.0.53.109)
 ....

This package is important, this is the main distributive package that provides us with kernel updates. We need to reinstall the previous version of it. So press y and find the previous good kernel version. In my case it is 5.3.0.51.104. Try to reinstall it sudo apt install linux-generic-hwe-18.04=5.3.0.51.104 --verbose-versions, prints this:

The following packages have unmet dependencies:
 linux-generic-hwe-18.04 : Depends: linux-image-generic-hwe-18.04 (= 5.3.0.51.104)  but 5.3.0.53.109 is to be installed
                           Depends: linux-headers-generic-hwe-18.04 (= 5.3.0.51.104) but 5.3.0.53.109 is to be installed

Let blacklist this newer version we don't like. Edit file /etc/apt/preferences and add this:

Package: linux-generic-hwe-18.04 linux-image-generic-hwe-18.04 linux-headers-generic-hwe-18.04
Pin: version 5.3.0.53.109
Pin-Priority: -1

Repeat sudo apt install linux-generic-hwe-18.04=5.3.0.51.104 --verbose-versions check the output, in my case everything seems fine, so I agree and reboot.