How to check "apt-get upgrade" status after losing ssh connection?
The following logs are related to apt upgrades:
/var/log/apt/history.log
/var/log/apt/term.log
/var/log/dpkg.log
If the command was dist-upgrade, there are additional logs in:
/var/log/dist-upgrade
FYI, it is usually safe to just re-run the upgrade and apt will continue where it left off when the process died due to disconnection. However...
A GNU Screen Primer:
When ssh'ing into a remote server and starting a long-running process in the foreground, it is best practice to use GNU Screen. Screen provides a virtual terminal that continues running even if your ssh connection is lost.
Install screen:
sudo apt-get install screen
Run screen:
screen
After running screen you will get a command line prompt as with a normal terminal. You can then run the upgrade from inside screen:
sudo apt-get upgrade
To understand how this works, "detach" screen by pressing Ctrl+a, d. This will return you to the non-screen terminal. You can see the list of running screens with
screen -list
If you only have one screen running, you can reattach it with:
screen -raAd
(This detaches screen in case it is attached elsewhere, and reattaches it to the terminal you are currently running.)
Typically you cannot scroll 'normally' from within screen without some extra setup. To scroll within screen, press Ctrl-Esc to enter cursor mode. You can then scroll down and up with j and k. Press Esc again to exit cursor mode.
There are many more resources on the net available for additional screen functions. It is an invaluable standard tool for system administration.
See also:
- How to keep processes running after ending ssh session?
In addition to doublerebel's answer, I noticed an alternative today.
I went to bed last night after starting an upgrade over SSH. I stupidly forgot to start it in screen and lost my SSH session overnight.
I was just about to start researching retty when I noticed that root had started a screen session.
me@GAMMA:~$ ps aux | grep -E 'release|upgrade|apt'
root 6208 0.0 0.0 29140 1628 ? Ss 01:57 0:05 SCREEN -e \0\0 -L -c screenrc -S ubuntu-release-upgrade-screen-window /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root 6209 0.2 5.6 287428 93144 pts/2 Ss+ 01:57 3:13 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
root 6239 0.0 0.0 50052 1184 ? Ss 01:58 0:00 /usr/sbin/sshd -o PidFile=/var/run/release-upgrader-sshd.pid -p 1022
root 7306 0.0 4.6 287432 77284 pts/2 S+ 02:43 0:08 /usr/bin/python /tmp/ubuntu-release-upgrader-1h6_g4/raring --mode=server --frontend=DistUpgradeViewText
me 26829 0.0 0.0 9440 956 pts/5 S+ 22:18 0:00 grep --color=auto -E release|upgrade|apt
So I listed root's screens and attached to it:
me@GAMMA:~$ sudo screen -list
There is a screen on:
6208.ubuntu-release-upgrade-screen-window (12/11/2013 01:57:58 AM) (Detached)
1 Socket in /var/run/screen/S-root.
me@GAMMA:~$ sudo screen -x -r
And Bam! I was back in the game.
To see real-time output from background apt job, use:
sudo tail -f /var/log/apt/term.log