How to interpret the output of netstat -o / netstat --timers
The timer column has two fields (from your o/p above):
keepalive (6176.47/0/0)
<1st field> <2nd field>
The 1st field can have values:
keepalive - when the keepalive timer is ON for the socket
on - when the retransmission timer is ON for the socket
off - none of the above is ON
The 2nd field has THREE subfields:
(6176.47/0/0) -> (a/b/c)
a=timer value (a=keepalive timer, when 1st field="keepalive"; a=retransmission timer, when 1st field="on")
b=number of retransmissions that have occurred
c=number of keepalive probes that have been sent
For example, I had two sockets opened between a client & a server (not loopback). The keepalive setting are:
KEEPALIVE_IDLETIME 30
KEEPALIVE_NUMPROBES 4
KEEPALIVE_INTVL 10
And I did a shutdown of the client machine, so at the server side I ran a netstat & the output was:
Port1:
netstat -c --timer | grep "192.0.0.1:43245 192.0.68.1:49742"
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.92/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.71/0/0)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.46/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.30/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.14/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.98/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.82/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.66/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.50/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.33/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.17/0/1)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.01/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.75/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (6.47/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.29/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.08/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.89/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.73/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.54/0/2)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (9.38/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.23/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.08/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.93/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.76/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.62/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (2.48/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.32/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.13/0/3)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (8.98/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (7.78/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (6.62/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (5.45/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (4.29/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (3.14/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (1.99/0/4)
tcp 0 0 192.0.0.1:43245 192.0.68.1:49742 ESTABLISHED keepalive (0.85/0/4)
You can see above that the server sent FOUR keepalive probes, each after 10 sec and since it is not getting any response, the number-of-probes-sent count increases, and after 4, it disconnects from the client.
Port 2:
For the 2nd connection, the socket was identical, except for, my application at server side was trying to send some message after the client went down & before the keepalive expired:
netstat -c --timer | grep "192.0.0.1:36483 192.0.68.1:43881"
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (8.18/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (7.00/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (5.86/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (4.71/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (3.55/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (2.40/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (1.21/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (0.05/0/1)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (8.91/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (7.75/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (6.56/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (5.39/0/2)
tcp 0 0 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED keepalive (4.14/0/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.21/2/2) // <---- retransmission timer kicks in
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.68/3/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.74/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.59/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.43/4/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.28/5/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.11/5/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.95/6/2)
. . . . .
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.65/249/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.58/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.48/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.36/250/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.26/251/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.15/251/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (3.01/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.92/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.84/252/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.72/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.64/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.55/253/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.47/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.39/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (0.31/254/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (2.19/255/2)
tcp 0 210 192.0.0.1:36483 192.0.68.1:43881 ESTABLISHED on (1.12/255/2)
As you can see, in this case things are a little different. When the client went down, my server started sending keepalive messages, but while it was still sending those keepalives, my server tried to send a message to the client. Since the client had gone down, the server couldn't get any ACK from the client, so the TCP retransmission started and the server tried to send the data again, each time incrementing the retransmit count (2nd field) when the retransmission timer (1st field) expired.
Hope this explains the netstat --timer option well.
The first number is clearly the countdown timer. Depending on status type, once the timer times out, it will retry and send out another FIN or whatever needed packet to try again. So the second number keeps track of the number of retries. Notice that the timer increases because TCP has backoff timers in case there is a traffic condition. The backoff timer avoids excessively frequent retries. Each failure results in a larger and larger backoff. The backoff could be exponential growth or linear, depending on TCP stack. I have not figured out what the 3 figure is for. Always zero in my displays.