Interesting things with openSSL programing.

    It's not hard to find a OpenSSL programming tutorial or some example code on the internet, but there are some interesting detail that is not mentioned in most of the articles online.
    A while ago, I was involved in a new project that gave me the chance to play with libSSL for a while, I did read some tutorials and studied some examples on the Internet; but when I was testing the actual source code, I found out that It was quite unstable; the software that I wrote acts unexpectedly as if It has its own mind and wound crash due to segmentation faults or memory violations unpredictably.
    After tracing the actual source code of the OpenSSL library, I figured out a few interesting things that I would like to share.

1.  OpenSSL library could be build without multi-thread support; since most commercial software products are designed with multi-thread, please be sure to check if the library that you're using is build with multi-thread enabled.
2. Even if you have enabled the multi-thread support, OpenSSL library is not thread safe right outside of the box, you have to implement two operations:
1. first is a function that will return the current thread ID.
2. the second is a function that will handle the mutex lock and unlock actuion.
3. the two operation functions mentioned in the previous section should be introduced  to the OpenSSL library by using some provided APIs like:
• CRYPTO_set_id_callback
• CRYPTO_set_locking_callback

if you are looking for a example code you can download the openssl source code and checkout the demo code (crypto/threads/mttest.c), which comes with the  source.

Speeding up TCP connection lost detection in a TCP client application

When we're writing a TCP server, it is very easy to detect connection lost of a remote TCP peer by enabling "TCP keep alive";with this feature enabled, when no data is actually transmitting on the present TCP connection, a "tcp-keep-alive" is sent and a "ack" return from the remote peer is required.
If there is no "ack" after several retries, the connection is  determined as "lost" by the OS.
When connection lost is detected, the system(Linux) will signal you by returning "read data ready" on the "select()" function, but returning a zero length data on "read()".
But, when we're writing a TCP client, the tricky part is: as a client, 90% of the time we always have data to send, so when the connection is lost, the TCP stack automatically retransmits the data; as a result, before "tcp-retransmit" actually timeouts, the "tcp-kee-alive" mechanism will not kick in.
Thus, if you are looking for a way to speed up "TCP connection lost" detection efficiency  in a client application, this "keep alive" mechanism  will do you no good.
A possible approach is to use "ioctl(socket , SIOCOUTQ, &BufferUsedSize)" to return the actual data size that is still waiting in the send queue, if no data has been successfully sent for a certain period of time, than the connection is possibly lost.

reference:
man 7 tcp

getting both IPv4 and IPv6 addresses of all local network interfaces via "getifaddrs"

In the old days, we often use ioctl to get the IPv4 address  of our local interfaces via :
ioctl(socketfd, SIOCGIFCONF, (struct ifconf)&buffer);
if we are interested in the IPv6 local addresses, we can always check them out via "/proc", for example, this is what we might do:

[sauron@icomserver ~]$ cat /proc/net/if_inet6
fe8000000000000002d0c9fffebb030b      03 40 20 80       eth1
00000000000000000000000000000001 01 80 10 80       lo

1. IPv6 address displayed in 32 hexadecimal chars without colons as separator
2. Netlink device number (interface index) in hexadecimal (see “ip addr” , too)
3. Prefix length in hexadecimal
4. Scope value (see kernel source “ include/net/ipv6.h” and “net/ipv6/addrconf.c” for more)
5. Interface flags (see “include/linux/rtnetlink.h” and “net/ipv6/addrconf.c” for more)
6. Device name

Quite easy, right? But what should we do to get both IPv4 and IPv6 addresses via the same API?
if you are seeking for a IPv4 and IPv6 total solution, then "getifaddrs"  is the answer that satisfies all your needs.

reference:
man 3 getifaddrs, man  3 getaddrinfo, man 3 getnameinfo

#include <arpa/inet.h>
#include <sys/socket.h>

#include <netdb.h>
#include <ifaddrs.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int

main(int argc, char *argv[])
{
   struct ifaddrs *ifaddr, *ifa;

   int family, s;
   char host[NI_MAXHOST];

   if (getifaddrs(&ifaddr) == -1) {
       perror("getifaddrs");

       exit(EXIT_FAILURE);
   }

/* Walk through linked list, maintaining head pointer so we
*               can free list later */

   for (ifa = ifaddr; ifa != NULL; ifa = ifa->ifa_next) {

       family = ifa->ifa_addr->sa_family;

/*   Display interface name and family (including symbolic
*               form of the latter for the common families) */

       printf("%s  address family: %d%s\n",

        ifa->ifa_name, family,
        (family == AF_PACKET) ? " (AF_PACKET)" :
        (family == AF_INET) ?   " (AF_INET)" :
        (family == AF_INET6) ?  " (AF_INET6)" : "");

       /* For an AF_INET* interface address, display the address */

       if (family == AF_INET || family == AF_INET6) {

    s = getnameinfo(ifa->ifa_addr,
     (family == AF_INET) ? sizeof(struct sockaddr_in) :

      sizeof(struct sockaddr_in6),
     host, NI_MAXHOST, NULL, 0, NI_NUMERICHOST);

    if (s != 0) {
        printf("getnameinfo() failed: %s\n", gai_strerror(s));

        exit(EXIT_FAILURE);
    }
    printf("\taddress: <%s>\n", host);
       }
   }

   freeifaddrs(ifaddr);
   exit(EXIT_SUCCESS);
}

how to get default gateway via rtnetlink (c code example)

this is a example code showing you how to get the ip of the default gateway via netlink socket. I've also attached some useful information if you want to study netlink

further reading:

Sample Code To Learn Netlink Infrastructure

 understanding and programming with netlink sockets(PDF)

reference:
man 3 rtnetlink, man 7 rtnetlink, man 3 netlink, man 7 netlink


example code:

#include <asm/types.h>
#include <netinet/ether.h>

#include <netinet/in.h>
#include <net/if.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <linux/netlink.h>

#include <linux/rtnetlink.h>
#include <sys/types.h>

#define BUFSIZE 8192

 char gateway[255];

 struct route_info{
  u_int dstAddr;
  u_int srcAddr;
  u_int gateWay;

  char ifName[IF_NAMESIZE];
 };

 int readNlSock(int sockFd, char *bufPtr, int seqNum, int pId){

  struct nlmsghdr *nlHdr;
  int readLen = 0, msgLen = 0;

  do{
   /* Recieve response from the kernel */
   if((readLen = recv(sockFd, bufPtr, BUFSIZE - msgLen, 0)) < 0){

    perror("SOCK READ: ");
    return -1;
   }

   nlHdr = (struct nlmsghdr *)bufPtr;

   /* Check if the header is valid */
   if((NLMSG_OK(nlHdr, readLen) == 0) || (nlHdr->nlmsg_type == NLMSG_ERROR))
   {

    perror("Error in recieved packet");
    return -1;
   }

   /* Check if the its the last message */

   if(nlHdr->nlmsg_type == NLMSG_DONE) {
   break;
   }
   else{

   /* Else move the pointer to buffer appropriately */
   bufPtr += readLen;
   msgLen += readLen;
   }

   /* Check if its a multi part message */
   if((nlHdr->nlmsg_flags & NLM_F_MULTI) == 0) {

   /* return if its not */
   break;
   }
  } while((nlHdr->nlmsg_seq != seqNum) || (nlHdr->nlmsg_pid != pId));

  return msgLen;
 }

 /* For printing the routes. */
 void printRoute(struct route_info *rtInfo)
 {

  char tempBuf[512];

  /* Print Destination address */
  if(rtInfo->dstAddr != 0)

   strcpy(tempBuf, (char *)inet_ntoa(rtInfo->dstAddr));
  else

   sprintf(tempBuf,"*.*.*.*\t");
  fprintf(stdout,"%s\t", tempBuf);

  /* Print Gateway address */
  if(rtInfo->gateWay != 0)
   strcpy(tempBuf, (char *)inet_ntoa(rtInfo->gateWay));

  else
   sprintf(tempBuf,"*.*.*.*\t");
  fprintf(stdout,"%s\t", tempBuf);

  /* Print Interface Name*/
  fprintf(stdout,"%s\t", rtInfo->ifName);

  /* Print Source address */
  if(rtInfo->srcAddr != 0)
   strcpy(tempBuf, (char *)inet_ntoa(rtInfo->srcAddr));

  else
   sprintf(tempBuf,"*.*.*.*\t");
  fprintf(stdout,"%s\n", tempBuf);
 }

 void printGateway()
 {
  printf("%s\n", gateway);
 }

 /* For parsing the route info returned */
 void parseRoutes(struct nlmsghdr *nlHdr, struct route_info *rtInfo)
 {

  struct rtmsg *rtMsg;
  struct rtattr *rtAttr;

  int rtLen;
  char *tempBuf = NULL;

  tempBuf = (char *)malloc(100);
  rtMsg = (struct rtmsg *)NLMSG_DATA(nlHdr);

  /* If the route is not for AF_INET or does not belong to main routing table
  then return. */
  if((rtMsg->rtm_family != AF_INET) || (rtMsg->rtm_table != RT_TABLE_MAIN))

   return;

  /* get the rtattr field */
  rtAttr = (struct rtattr *)RTM_RTA(rtMsg);

  rtLen = RTM_PAYLOAD(nlHdr);
  for(;RTA_OK(rtAttr,rtLen);rtAttr = RTA_NEXT(rtAttr,rtLen)){

   switch(rtAttr->rta_type) {
   case RTA_OIF:
     if_indextoname(*(int *)RTA_DATA(rtAttr), rtInfo->ifName);

     break;
   case RTA_GATEWAY:
     rtInfo->gateWay.s_addr = *(u_int *)RTA_DATA(rtAttr);

     break;
   case RTA_PREFSRC:
     rtInfo->srcAddr.s_addr = *(u_int *)RTA_DATA(rtAttr);

     break;
   case RTA_DST:
     rtInfo->dstAddr.s_addr = *(u_int *)RTA_DATA(rtAttr);

     break;
   }
  }
  //printf("%s\n", (char *)inet_ntoa(rtInfo->dstAddr));
  if (strstr((char *)inet_ntoa(rtInfo->dstAddr), "0.0.0.0"))

   sprintf(gateway, (char *)inet_ntoa(rtInfo->gateWay));
  //printRoute(rtInfo);

  free(tempBuf);
  return;
 }

 int main()
 {

  struct nlmsghdr *nlMsg;
  struct rtmsg *rtMsg;

  struct route_info *rtInfo;
  char msgBuf[BUFSIZE];

  int sock, len, msgSeq = 0;

  char buff[1024];

  /* Create Socket */
  if((sock = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE)) < 0)

   perror("Socket Creation: ");

  /* Initialize the buffer */
  memset(msgBuf, 0, BUFSIZE);

  /* point the header and the msg structure pointers into the buffer */
  nlMsg = (struct nlmsghdr *)msgBuf;
  rtMsg = (struct rtmsg *)NLMSG_DATA(nlMsg);

  /* Fill in the nlmsg header*/
  nlMsg->nlmsg_len = NLMSG_LENGTH(sizeof(struct rtmsg)); // Length of message.

  nlMsg->nlmsg_type = RTM_GETROUTE; // Get the routes from kernel routing table .

  nlMsg->nlmsg_flags = NLM_F_DUMP | NLM_F_REQUEST; // The message is a request for dump.

  nlMsg->nlmsg_seq = msgSeq++; // Sequence of the message packet.
  nlMsg->nlmsg_pid = getpid(); // PID of process sending the request.


  /* Send the request */
  if(send(sock, nlMsg, nlMsg->nlmsg_len, 0) < 0){

   printf("Write To Socket Failed...\n");
   return -1;
  }

  /* Read the response */

  if((len = readNlSock(sock, msgBuf, msgSeq, getpid())) < 0) {

   printf("Read From Socket Failed...\n");
   return -1;
  }
  /* Parse and print the response */
  rtInfo = (struct route_info *)malloc(sizeof(struct route_info));

  /* THIS IS THE NETTSTAT -RL code I commented out the printing here and in parse routes */
  //fprintf(stdout, "Destination\tGateway\tInterface\tSource\n");
  for(;NLMSG_OK(nlMsg,len);nlMsg = NLMSG_NEXT(nlMsg,len)){

   memset(rtInfo, 0, sizeof(struct route_info));
   parseRoutes(nlMsg, rtInfo);
  }

  free(rtInfo);
  close(sock);

  printGateway();

  return 0;
 }
   

Using socket RAW to send an ICMP protocal

this is a simple program that i found on the internet showing how to send ICMP using RAW socket.

example code:

/*
 *    pinger.c 
 *    This is a ping imitation program 
 *    It will send an ICMP ECHO packet to the server of 
 *    your choice and listen for an ICMP REPLY packet
 *    Have fun!
 */
/*
 *    pinger.c 
 *    This is a ping imitation program 
 *    It will send an ICMP ECHO packet to the server of 
 *    your choice and listen for an ICMP REPLY packet
 *    Have fun!
 */
#include <stdio.h>
#include <stdlib.h>

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <linux/ip.h>

#include <linux/icmp.h>
#include <string.h>
#include <unistd.h>


char dst_addr[15];

char src_addr[15];

unsigned short in_cksum(unsigned short *, int);

void parse_argvs(char**, char*, char* );

void usage();
char* getip();

int main(int argc, char* argv[])

{
    struct iphdr* ip;
    struct iphdr* ip_reply;

    struct icmphdr* icmp;
    struct sockaddr_in connection;

    char* packet;
    char* buffer;
    int sockfd;

    int optval;
    int addrlen;
    
    if (getuid() != 0)

    {
 fprintf(stderr, "%s: root privelidges needed\n", *(argv + 0));

 exit(EXIT_FAILURE);
    }

    parse_argvs(argv, dst_addr, src_addr);

    printf("Source address: %s\n", src_addr);
    printf("Destination address: %s\n", dst_addr);

    
    /*
     * allocate all necessary memory
    */
    ip = malloc(sizeof(struct iphdr));

    ip_reply = malloc(sizeof(struct iphdr));
    icmp = malloc(sizeof(struct icmphdr));

    packet = malloc(sizeof(struct iphdr) + sizeof(struct icmphdr));

    buffer = malloc(sizeof(struct iphdr) + sizeof(struct icmphdr));

    /****************************************************************/
    
    ip = (struct iphdr*) packet;

    icmp = (struct icmphdr*) (packet + sizeof(struct iphdr));

    
    /*  
     * here the ip packet is set up except checksum
     */
    ip->ihl   = 5;
    ip->version   = 4;

    ip->tos   = 0;
    ip->tot_len   = sizeof(struct iphdr) + sizeof(struct icmphdr);

    ip->id   = htons(random());
    ip->ttl   = 255;

    ip->protocol  = IPPROTO_ICMP;
    ip->saddr   = inet_addr(src_addr);

    ip->daddr   = inet_addr(dst_addr);

    
    if ((sockfd = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP)) == -1)

    {
 perror("socket");
 exit(EXIT_FAILURE);

    }
    
    /* 
     * IP_HDRINCL must be set on the socket so that
     * the kernel does not attempt to automatically add
     * a default ip header to the packet
     */
    
    setsockopt(sockfd, IPPROTO_IP, IP_HDRINCL, &optval, sizeof(int));

    
    /*
     * here the icmp packet is created
     * also the ip checksum is generated
     */
    icmp->type   = ICMP_ECHO;
    icmp->code   = 0;

    icmp->un.echo.id  = 0;
    icmp->un.echo.sequence = 0;

    icmp->checksum   = 0;
    icmp-> checksum  = in_cksum((unsigned short *)icmp, sizeof(struct icmphdr));

    
    ip->check   = in_cksum((unsigned short *)ip, sizeof(struct iphdr));

    
    connection.sin_family = AF_INET;
    connection.sin_addr.s_addr = inet_addr(dst_addr);

    
    /*
     * now the packet is sent
     */
    
    sendto(sockfd, packet, ip->tot_len, 0, (struct sockaddr *)&connection, sizeof(struct sockaddr));

    printf("Sent %d byte packet to %s\n", sizeof(packet), dst_addr);
    
    /*
     * now we listen for responses
     */

    addrlen = sizeof(connection);
    if (recvfrom(sockfd, buffer, sizeof(struct iphdr) + sizeof(struct icmphdr), 0, (struct sockaddr *)&connection, &addrlen) == -1)

    {
 perror("recv");
    }
    else
    {

 printf("Received %d byte reply from %s:\n", sizeof(buffer), dst_addr);
        ip_reply = (struct iphdr*) buffer;

 printf("ID: %d\n", ntohs(ip_reply->id));
 printf("TTL: %d\n", ip_reply->ttl);

    }
    close(sockfd);
    return 0;
}

void parse_argvs(char** argv, char* dst, char* src)

{
    int i;
    if(!(*(argv + 1))) 
    {

 /* there are no options on the command line */
 usage();
 exit(EXIT_FAILURE); 
    }

    if (*(argv + 1) && (!(*(argv + 2)))) 
    {

 /* 
  *   only one argument provided
  *   assume it is the destination server
  *   source address is local host
  */
 strncpy(dst, *(argv + 1), 15);

 strncpy(src, getip(), 15);
 return;

    }
    else if ((*(argv + 1) && (*(argv + 2))))

    {
 /* 
  *    both the destination and source address are defined
  *    for now only implemented is a source address and 
  *    destination address
  */
 strncpy(dst, *(argv + 1), 15);

 i = 2;
 while(*(argv + i + 1))

 {
     if (strncmp(*(argv + i), "-s", 2) == 0)

     {
  strncpy(src, *(argv + i + 1), 15);

  break;
     }
     i++;
 }

    }
}

void usage()
{
    fprintf(stderr, "\nUsage: pinger [destination] <-s [source]>\n");

    fprintf(stderr, "Destination must be provided\n");
    fprintf(stderr, "Source is optional\n\n");

}

char* getip()
{
    char buffer[256];

    struct hostent* h;
    
    gethostname(buffer, 256);

    h = gethostbyname(buffer);
    
    return inet_ntoa(*(struct in_addr *)h->h_addr);

    
}
/*
 * in_cksum --
 * Checksum routine for Internet Protocol
 * family headers (C Version)
 */
unsigned short in_cksum(unsigned short *addr, int len)

{
    register int sum = 0;
    u_short answer = 0;

    register u_short *w = addr;
    register int nleft = len;

    /*
     * Our algorithm is simple, using a 32 bit accumulator (sum), we add
     * sequential 16 bit words to it, and at the end, fold back all the
     * carry bits from the top 16 bits into the lower 16 bits.
     */
    while (nleft > 1)
    {
   sum += *w++;

   nleft -= 2;
    }
    /* mop up an odd byte, if necessary */
    if (nleft == 1)

    {
   *(u_char *) (&answer) = *(u_char *) w;

   sum += answer;
    }
    /* add back carry outs from top 16 bits to low 16 bits */
    sum = (sum >> 16) + (sum & 0xffff);  /* add hi 16 to low 16 */

    sum += (sum >> 16);    /* add carry */
    answer = ~sum;    /* truncate to 16 bits */

    return (answer);
}

Stopping TCP retransmit

The TCP keep-alive is a great way to detect TCP connection lost; if a connection list lost, "select()" will show readable data, yet the data length returned by "recv()" is less or equal to zero. Yet, if you've already sent a data during the connection lost, the TCP retransmit is triggered, stopping the  mechanism of TCP keep-alive, the bad news is that the retransmit time can go on to 15-20 mins, and keep-alive will not be invoked during this time period.
In other words, you will not notice the connection lost unless you full the Tx buffer of the net device driver(the default size is 4096 bytes).
If you have little or no data to send, detecting the connection loss will take forever, and become a pain in the ass.
To solve this issue,you can reduce the tcp retransmit times by doing:
    system("sysctl -w net.ipv4.tcp_retries1=2");
    system("sysctl -w net.ipv4.tcp_retries2=3");

yet the effect is system wide.... not a very elegant solution.

P.S.
It was interesting to find out that linux socket always tries to send every packet, even if "netif_queue_stopped(netdev *)" (used for flow control on the Tx side of the driver) or "netif_carrier_off(netdev *)"(used when carrier is down ex. unplugged cable) is called by the net device driver, Linux queues the unsent packets and sends it the instance when carrier or queue is available. This mechanism guarantees the message you sent is delivered to the destination if possible.

socket keep alive time

You might know that,while setting up a TCP connection you can enable SO_KEEPALIVE, forcing the system to check on the connection routinely to see if the connection is still there, not broken by the any connection issues or so.
Yet, have you ever wondered, how often does the system go through this connection check?
This is how we change "keep alive time" for individual socket on linux systems, otherwise the system checks for the connection every 30 mins or so, by default, which is usually too long.
Notice: this is not a standard socket operation, which may not be used on other operating systems. (for example: windows)
Code:

int keepAlive = 1;

// set timeout params for SO_KEEPALIVE
keepTime = 5; //the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe
keepProbe = 1; //number of retry
keepInvl = 1; //the interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
if( setsockopt(cliSocket, SOL_TCP, TCP_KEEPIDLE, &keepTime, sizeof(keepTime)) == -1
||setsockopt(cliSocket, SOL_TCP, TCP_KEEPCNT, &keepProbe, sizeof(keepProbe)) == -1
||setsockopt(cliSocket, SOL_TCP, TCP_KEEPINTVL, &keepInvl, sizeof(keepInvl)) == -1
||setsockopt(cliSocket,SOL_SOCKET,SO_KEEPALIVE, &keepAlive,sizeof(keepAlive)) == -1)
{
printf("Socket %d setKeepAlive timeout failed.\n", cliSocket);
close(cliSocket);
}