At work I was doing some stuff with the libmodbus library: https://github.com/stephane/libmodbus/. This particular part of our codebase is c++. We needed the modbus server to listen for incoming connections in one thread and have other work happening in another thread. At some point we need to cleanup the server (when everything is shutting down, or in tests or what have you).

The typical pattern for the server is:

listen(socket, OUTSTANDING_REQUESTS);
while (running) {
  int client_socket = accept(socket, ...);

  // do something with the connection
  send(client_socket ...);
  recv(client_socket ...);

  close(client_socket);
}

However, by default sockets will block at the accept call until there is a connect call from a client. Similarly, it will block at the recv call until there is a send from a connected client. This can be a problem when shutting down because there isn't really a way to break out. In java, for instance you can close the socket, or do a thread.interrupt() to break out, but once these calls happen it isn't possible from what I could tell.

The solution

Instead of blocking on the accept and receive calls, we can block on a select call instead. Select takes a group of file descriptors and unblocks when one or more are ready for IO (ie: an accept, receive, etc.). A timeout can be used, or you can use a self-pipe. The read side of the pipe is added to the group of file descriptors and when the loop should be broken out of, the pipe is written to (via a signal or something).

For instance, the above example:

listen(socket, OUTSTANDING_REQUESTS);
FD_SET(socket, readfds);
int client_socket = -1;
while (running) {
  select(readfds ...);

  if (FD_ISSET(socket_readfds)) {
    client_socket = accept(socket, ...);
    FD_SET(client_socket, readfds);
  }

  if (client_socket != -1 && FD_ISSET(client_socket, readfds)) {
    // do something with the connection
    send(client_socket ...);
    recv(client_socket ...);
    close(client_socket);
  }
}

A complete example is available here: https://github.com/compscidr/non-blocking-sockets