The TCP keep-alive is a great way to detect TCP connection lost; if a connection list lost, "select()" will show readable data, yet the data length returned by "recv()" is less or equal to zero. Yet, if you've already sent a data during the connection lost, the TCP retransmit is triggered, stopping the mechanism of TCP keep-alive, the bad news is that the retransmit time can go on to 15-20 mins, and keep-alive will not be invoked during this time period.
In other words, you will not notice the connection lost unless you full the Tx buffer of the net device driver(the default size is 4096 bytes).
If you have little or no data to send, detecting the connection loss will take forever, and become a pain in the ass.
To solve this issue,you can reduce the tcp retransmit times by doing:
system("sysctl -w net.ipv4.tcp_retries1=2");
system("sysctl -w net.ipv4.tcp_retries2=3");
yet the effect is system wide.... not a very elegant solution.
P.S.
It was interesting to find out that linux socket always tries to send every packet, even if "netif_queue_stopped(netdev *)" (used for flow control on the Tx side of the driver) or "netif_carrier_off(netdev *)"(used when carrier is down ex. unplugged cable) is called by the net device driver, Linux queues the unsent packets and sends it the instance when carrier or queue is available. This mechanism guarantees the message you sent is delivered to the destination if possible.
