[lwip-users] Questions about LwIP KEEPALIVE...

[Frédéric BERNON]

Hi group,

Can you give me some informations about he LwIP KEEPALIVE features? I use Socket Layer API, and I would like to know if it's possible to enable it "per socket" and to fix for each one a different timeout? I found in several files these informations :

In sockets.h/.c, I see :

#define SO_KEEPALIVE 0x0008 /* keep connections alive */

For this one, except seeing it is set/unset in the sock->conn->pcb.tcp->so_options.

In ip.h "These are the same like SO_XXX." :

#define SOF_KEEPALIVE (u16_t)0x0008U /* keep connections alive */


In tcp.h, I see :

#define TCP_KEEPALIVE 0x02 /* send KEEPALIVE probes when idle for pcb->keepalive miliseconds */

/* Keepalive values */
#define TCP_KEEPDEFAULT 7200000 /* KEEPALIVE timer in miliseconds */
#define TCP_KEEPINTVL 75000 /* Time between KEEPALIVE probes in miliseconds */
#define TCP_KEEPCNT 9 /* Counter for KEEPALIVE probes */
#define TCP_MAXIDLE TCP_KEEPCNT * TCP_KEEPINTVL /* Maximum KEEPALIVE probe time */

So, TCP_MAXIDLE is 675 seconds (11.25min), and TCP_KEEPDEFAULT is 7200 seconds (~120min : 2 hours) !?!?!!


In my lwipopts.h, I have set :

/* The TCP timer interval in milliseconds. */
#define TCP_TMR_INTERVAL 100

/* the fine grained timeout in milliseconds */
#define TCP_FAST_INTERVAL TCP_TMR_INTERVAL

/* the coarse grained timeout in milliseconds */
#define TCP_SLOW_INTERVAL (2*TCP_TMR_INTERVAL)


In tcp.c, I see :

In tcp.c :

/* Check if KEEPALIVE should be sent */
if((pcb->so_options & SOF_KEEPALIVE) && ((pcb->state == ESTABLISHED) || (pcb->state == CLOSE_WAIT))) {
if((u32_t)(tcp_ticks - pcb->tmr) > (pcb->keepalive + TCP_MAXIDLE) / TCP_SLOW_INTERVAL) {
LWIP_DEBUGF(TCP_DEBUG, ("tcp_slowtmr: KEEPALIVE timeout. Aborting connection to %"U16_F".%"U16_F".%"U16_F".%"U16_F".\n",
ip4_addr1(&pcb->remote_ip), ip4_addr2(&pcb->remote_ip),
ip4_addr3(&pcb->remote_ip), ip4_addr4(&pcb->remote_ip)));

tcp_abort(pcb);
}
else if((u32_t)(tcp_ticks - pcb->tmr) > (pcb->keepalive + pcb->keep_cnt * TCP_KEEPINTVL) / TCP_SLOW_INTERVAL) {
tcp_keepalive(pcb);
pcb->keep_cnt++;
}
}

If I unserstand all these informations, at socket layer, I can set with setsockopt on a TCP socket the TCP_KEEPALIVE with my own timeout (in the setsockopt value), and set the SO_KEEPALIVE option to 1 to enable KEEPALIVE checking.

(pcb->keepalive + TCP_MAXIDLE) / TCP_SLOW_INTERVAL) give with default values (TCP_KEEPDEFAULT + TCP_MAXIDLE) / TCP_SLOW_INTERVAL) : (7200000+675000)/(2*100) = 7875000/200 = 39375 ticks.

After that, if my connection is in ESTABLISHED or CLOSE_WAIT state, if (tcp_ticks - pcb->tmr) > 39375 , I suppose the tcp connection will be abort. Right?

So, if I dont do any mistake, if a router between my lwip devices crashs, the connection will detect it in... 39375*200 ms = 7875 sec = 131.25 min !?!?!

I suppose I do a mistake somewhere, or I don't understand (because more than 2 hours - by default - to detect a connection failure seems to be very very long)...

Can you help me ?

Thank you


====================================
Frédéric BERNON
HYMATOM SA
Chef de projet informatique
Microsoft Certified Professional
Tél. : [phone]
Fax. : [phone]
Email : ***@hymatom.fr
Web Site : http://www.hymatom.fr
====================================
P Avant d'imprimer, penser à l'environnement

[Kieran Mansley]

Post by Frédéric BERNON
So, if I dont do any mistake, if a router between my lwip devices
crashs, the connection will detect it in... 39375*200 ms = 7875 sec =
131.25 min !?!?!
I suppose I do a mistake somewhere, or I don't understand (because
more than 2 hours - by default - to detect a connection failure seems
to be very very long)...

I think you've read it right. To comply with the TCP RFCs a stack MUST
default to a TCP keepalive timeout of no less that 2 hours. See
http://www.freesoft.org/CIE/RFC/1122/114.htm

If you were to send any data you'd find out sooner of course - TCP
keepalive is really only of use for connections that are idle (or I
suppose uni-directional).

The algorithm is quite conservative on the side of not closing
connections until it's really sure they are dead.

Kieran

[Frédéric BERNON]

Ok, I didn't think that was to be RFC compliant...

But don't you think these values would have to be settable in lwipopts.h ? Because, even with TCP_KEEPALIVE to 0, the minimum delay to detect a loss of stream connection is TCP_MAXIDLE (75000*9 = 675000ms =~ 11.25 min).


/* Keepalive values */
#define TCP_KEEPINTVL 75000 /* Time between KEEPALIVE probes in miliseconds */
#define TCP_KEEPCNT 9 /* Counter for KEEPALIVE probes */
#define TCP_MAXIDLE TCP_KEEPCNT * TCP_KEEPINTVL /* Maximum KEEPALIVE probe time */


On a basic XP pc (yes, I know, this is not the "best" reference), registry values (in "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters") seems to be TCP_KEEPCNT=5 ("TcpMaxDataRetransmissions"), TCP_KEEPINTVL=1000 ("KeepAliveInterval"), and TCP_KEEPDEFAULT=7200000 ("KeepAliveTime"). http://support.microsoft.com/kb/314053/en-us

What do you think about this idea?

====================================
Frédéric BERNON
HYMATOM SA
Chef de projet informatique
Microsoft Certified Professional
Tél. : [phone]
Fax. : [phone]
Email : ***@hymatom.fr
Web Site : http://www.hymatom.fr
====================================
P Avant d'imprimer, penser à l'environnement



-----Message d'origine-----
De : lwip-users-bounces+frederic.bernon=***@nongnu.org [mailto:lwip-users-bounces+frederic.bernon=***@nongnu.org] De la part de Kieran Mansley
Envoyé : jeudi 15 février 2007 16:35
À : Mailing list for lwIP users
Objet : Re: [lwip-users] Questions about LwIP KEEPALIVE...

Post by Frédéric BERNON
So, if I dont do any mistake, if a router between my lwip devices
crashs, the connection will detect it in... 39375*200 ms = 7875 sec =
131.25 min !?!?!
I suppose I do a mistake somewhere, or I don't understand (because
more than 2 hours - by default - to detect a connection failure seems
to be very very long)...

I think you've read it right. To comply with the TCP RFCs a stack MUST default to a TCP keepalive timeout of no less that 2 hours. See http://www.freesoft.org/CIE/RFC/1122/114.htm

If you were to send any data you'd find out sooner of course - TCP keepalive is really only of use for connections that are idle (or I suppose uni-directional).

The algorithm is quite conservative on the side of not closing connections until it's really sure they are dead.

Kieran

[Kieran Mansley]

Post by Frédéric BERNON
Ok, I didn't think that was to be RFC compliant...
But don't you think these values would have to be settable in lwipopts.h ? Because, even with TCP_KEEPALIVE to 0, the minimum delay to detect a loss of stream connection is TCP_MAXIDLE (75000*9 = 675000ms =~ 11.25 min).
/* Keepalive values */
#define TCP_KEEPINTVL 75000 /* Time between KEEPALIVE probes in miliseconds */
#define TCP_KEEPCNT 9 /* Counter for KEEPALIVE probes */
#define TCP_MAXIDLE TCP_KEEPCNT * TCP_KEEPINTVL /* Maximum KEEPALIVE probe time */
On a basic XP pc (yes, I know, this is not the "best" reference), registry values (in "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters") seems to be TCP_KEEPCNT=5 ("TcpMaxDataRetransmissions"), TCP_KEEPINTVL=1000 ("KeepAliveInterval"), and TCP_KEEPDEFAULT=7200000 ("KeepAliveTime"). http://support.microsoft.com/kb/314053/en-us
What do you think about this idea?

It's not an option that most people would want to change, and as it's
still configurable by those who want to and who understand the
implications (by editing tcp.h) I'm happy with the way it is.

Out of interest, can I ask what problem you want the KEEPALIVE option to
solve? In most cases people see TCP connections surviving short term
connectivity breakage as a good thing, but sounds like you want to know
(and close connections) if something has gone wrong in the network.

Kieran

[Frédéric BERNON]

Hi Kieran,

I just change in tcp.h this :

/*
* User-settable options (used with setsockopt).
*/
#define TCP_NODELAY 0x01 /* don't delay send to coalesce packets */
#define TCP_KEEPALIVE 0x02 /* send KEEPALIVE probes when idle for pcb->keepalive miliseconds */

/* Keepalive values */
#define TCP_KEEPDEFAULT 7200000 /* KEEPALIVE timer in miliseconds */

#ifndef TCP_KEEPINTVL
#define TCP_KEEPINTVL 75000 /* Time between KEEPALIVE probes in miliseconds */
#endif

#ifndef TCP_KEEPCNT
#define TCP_KEEPCNT 9 /* Counter for KEEPALIVE probes */
#endif

#define TCP_MAXIDLE TCP_KEEPCNT * TCP_KEEPINTVL /* Maximum KEEPALIVE probe time */


So, default values are respected, and can be change by lwipopts.h.

In changing that, I want to be able to close my tcp connections when the virtual circuit between my client and my server is broken, to avoid waiting a too long time before the client detect the problem (in this protocol, it can have a long time between exchanges, but if the link is "broken", we have to change to another server...).

====================================
Frédéric BERNON
HYMATOM SA
Chef de projet informatique
Microsoft Certified Professional
Tél. : [phone]
Fax. : [phone]
Email : ***@hymatom.fr
Web Site : http://www.hymatom.fr
====================================
P Avant d'imprimer, penser à l'environnement



-----Message d'origine-----
De : lwip-users-bounces+frederic.bernon=***@nongnu.org [mailto:lwip-users-bounces+frederic.bernon=***@nongnu.org] De la part de Kieran Mansley
Envoyé : vendredi 16 février 2007 12:10
À : Mailing list for lwIP users
Objet : Re: RE : [lwip-users] Questions about LwIP KEEPALIVE...

Post by Frédéric BERNON
Ok, I didn't think that was to be RFC compliant...
But don't you think these values would have to be settable in
lwipopts.h ? Because, even with TCP_KEEPALIVE to 0, the minimum delay
to detect a loss of stream connection is TCP_MAXIDLE (75000*9 =
675000ms =~ 11.25 min).
/* Keepalive values */
#define TCP_KEEPINTVL 75000 /* Time between KEEPALIVE probes in miliseconds */
#define TCP_KEEPCNT 9 /* Counter for KEEPALIVE probes */
#define TCP_MAXIDLE TCP_KEEPCNT * TCP_KEEPINTVL /* Maximum KEEPALIVE probe time */
On a basic XP pc (yes, I know, this is not the "best" reference),
registry values (in "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters") seems to be TCP_KEEPCNT=5 ("TcpMaxDataRetransmissions"), TCP_KEEPINTVL=1000 ("KeepAliveInterval"), and TCP_KEEPDEFAULT=7200000 ("KeepAliveTime"). http://support.microsoft.com/kb/314053/en-us
What do you think about this idea?

It's not an option that most people would want to change, and as it's still configurable by those who want to and who understand the implications (by editing tcp.h) I'm happy with the way it is.

Out of interest, can I ask what problem you want the KEEPALIVE option to solve? In most cases people see TCP connections surviving short term connectivity breakage as a good thing, but sounds like you want to know (and close connections) if something has gone wrong in the network.

Kieran

[David Empson]

On Saturday, February 17, 2007 2:21 AM, "Fr=E9d=E9ric BERNON"=20

In changing that, I want to be able to close my tcp connections whe=

n the=20

virtual circuit between my client
and my server is broken, to avoid waiting a too long time before th=

e=20

client detect the problem (in this
protocol, it can have a long time between exchanges, but if the lin=

k is=20

"broken", we have to change to
another server...).

I'm in the process of implementing changes to our DNP 3.0 protocol dr=
iver to=20
use TCP/IP (LWIP). The DNP 3.0 documentation specifically mentions t=
he=20
keep-alive problem, and it suggests that rather than modifying or rel=
ying on=20
the TCP keep-alive timer (which might not be possible in some TCP/IP=
=20
stacks), it is better to implement a higher level method for the clie=
nt to=20
poll the server regularly, so it can rapidly detect a broken connecti=
on and=20
reconnect.

In DNP 3.0's case, the recommended method is for the client to send a=
DNP=20
3.0 data link layer "request link status" every ten seconds (if no ot=
her=20
messages have been sent). The server is supposed to respond with a si=
mple=20
ack.

The server doesn't need a similar mechanism unless it is also able to=
=20
initiate a connection to the client (i.e. both ends can listen for an=
d=20
initiate connections).

A little background: DNP 3.0 is a SCADA protocol used to transfer cur=
rent=20
state and event information for digital and analog inputs and outputs=
=20
between a "slave" device (e.g. a device operating in a power substati=
on) and=20
a "master" device (e.g. a display terminal in a central office). The =
slave=20
typically acts as the TCP/IP server (listening) and the client typica=
lly=20
acts as the TCP/IP client (connecting) but it is possible for a slave=
to=20
initiate a connection if it restarts and wants to send data.

The typical data flow for DNP 3.0 over TCP/IP will be for the server =
(DNP=20
3.0 slave) to send data to the client (DNP 3.0 master) when data chan=
ges=20
occur. The client (master) can also poll the server (slave) to get cu=
rrent=20
state information and to issue control operations, but these are rand=
om and=20
likely to be rarely sent, or regularly sent at a slow rate.

[Frédéric BERNON]

Hi David,

You're right, if the protocol got a specific message to poll the peer, this is not a problem. But some protocols don't get them. Sometimes, we can found some workarounds. By example, with Telnet : you can send a "<SPACE><BACKSPACE>" to peer to force a communication checking, but, it can only work with my own telnet clients/servers. More, when you port some sources, you need to adapt them, and it's not very always simple.

To my point of view, porting applications on LwIP at socket layer would need to do only very limited modifications on applications. I thought that enable to customize default values with lwipopts.h seems to be in the "lwip spirit". If doing this is "too specific", ok, I will do it only in my own lwip copy. It's like IGMP, some modifications don't seem to be very "popular" (even if with good #ifdef, there is no more code), so, I just set them in my copy... :)

I just look the DNP protocol. Is it a very used protocol? Do you an overview about it (send me a private email if you can)...

====================================
Frédéric BERNON
HYMATOM SA
Chef de projet informatique
Microsoft Certified Professional
Tél. : [phone]
Fax. : [phone]
Email : ***@hymatom.fr
Web Site : http://www.hymatom.fr
====================================
P Avant d'imprimer, penser à l'environnement



-----Message d'origine-----
De : lwip-users-bounces+frederic.bernon=***@nongnu.org [mailto:lwip-users-bounces+frederic.bernon=***@nongnu.org] De la part de David Empson
Envoyé : dimanche 18 février 2007 23:56
À : Mailing list for lwIP users
Objet : Re: RE : [lwip-users] Questions about LwIP KEEPALIVE...


On Saturday, February 17, 2007 2:21 AM, "Frédéric BERNON"

Post by Frédéric BERNON
In changing that, I want to be able to close my tcp connections when
the
virtual circuit between my client
and my server is broken, to avoid waiting a too long time before the
client detect the problem (in this
protocol, it can have a long time between exchanges, but if the link is
"broken", we have to change to
another server...).

I'm in the process of implementing changes to our DNP 3.0 protocol driver to
use TCP/IP (LWIP). The DNP 3.0 documentation specifically mentions the
keep-alive problem, and it suggests that rather than modifying or relying on
the TCP keep-alive timer (which might not be possible in some TCP/IP
stacks), it is better to implement a higher level method for the client to
poll the server regularly, so it can rapidly detect a broken connection and
reconnect.

In DNP 3.0's case, the recommended method is for the client to send a DNP
3.0 data link layer "request link status" every ten seconds (if no other
messages have been sent). The server is supposed to respond with a simple
ack.

The server doesn't need a similar mechanism unless it is also able to
initiate a connection to the client (i.e. both ends can listen for and
initiate connections).

A little background: DNP 3.0 is a SCADA protocol used to transfer current
state and event information for digital and analog inputs and outputs
between a "slave" device (e.g. a device operating in a power substation) and
a "master" device (e.g. a display terminal in a central office). The slave
typically acts as the TCP/IP server (listening) and the client typically
acts as the TCP/IP client (connecting) but it is possible for a slave to
initiate a connection if it restarts and wants to send data.

The typical data flow for DNP 3.0 over TCP/IP will be for the server (DNP
3.0 slave) to send data to the client (DNP 3.0 master) when data changes
occur. The client (master) can also poll the server (slave) to get current
state information and to issue control operations, but these are random and
likely to be rarely sent, or regularly sent at a slow rate.

[David Empson]

Hi Fr=E9d=E9ric

On Monday, February 19, 2007 10:19 PM, "Fr=E9d=E9ric BERNON"=20

You're right, if the protocol got a specific message to poll the pee=

r, this=20

is not a problem. But some protocols
don't get them. Sometimes, we can found some workarounds. By example=

, with=20

Telnet : you can send a
"<SPACE><BACKSPACE>" to peer to force a communication checking, but,=

it can=20

only work with my
own telnet clients/servers. More, when you port some sources, you ne=

ed to=20

adapt them, and it's not very
always simple.
To my point of view, porting applications on LwIP at socket layer wo=

uld=20

need to do only very limited
modifications on applications. I thought that enable to customize de=

fault=20

values with lwipopts.h seems to be
in the "lwip spirit". If doing this is "too specific", ok, I will do=

it=20

only in my own lwip copy.

Even if you only do it in your copy of LWIP, I can see two problems w=
ith=20
modifying the LWIP keep-alive timer:

1. It only affects one direction. The other direction will still be s=
ending=20
slow keep-alive packets (unless you have sufficient control over the =
other=20
end as well).

2. If you are running multiple protocols or applications over TCP/IP,=
a=20
global change in the keep-alive timer may not be suitable for all=
=20
applications. Even if you are running the same protocol over multiple=
=20
connections, you might need different keep-alive settings for each=
=20
connection (e.g. a LAN connection vs a cellular data connection).

I just look the DNP protocol. Is it a very used protocol? Do you an=

=20

overview about it (send me a private email if you can)...

(replied privately to this part)
=20

[Frédéric BERNON]

Even if you only do it in your copy of LWIP, I can see two problems with
1. It only affects one direction. The other direction will still be sending
slow keep-alive packets (unless you have sufficient control over the other
end as well).

Sure

2. If you are running multiple protocols or applications over TCP/IP, a
global change in the keep-alive timer may not be suitable for all
applications. Even if you are running the same protocol over multiple
connections, you might need different keep-alive settings for each
connection (e.g. a LAN connection vs a cellular data connection).

Yes, that's why the TCP_KEEPALIVE can change the keepalive timeout for each socket. But the default interval seems to long for most of applications. So, I think that changing it enable a "faster" reaction for some sockets, and allow to set a longer keepalive timeout for others ones...

I just look the DNP protocol. Is it a very used protocol? Do you an
overview about it (send me a private email if you can)...
(replied privately to this part)

Yes, I got your email, thank you...