Threaded Asynchronous Tcp Server with a blocking Client

I don't know now how many prototypes I have written in the past few weeks to verify which approach would fit best to scale and be most performance for indeXus.Net Shared Cache. To write a Client / Server Architecture which mostly will be N:M connectivity was not that easy as I thought in beginning.

Honestly I believe I have study every single C# sample I found on the common search engines and arrived to the point where I started to adapt the samples. Unfortunately 99% of founded samples are showing how to pass a string from the client to the server beside one blog which has written down a lot of theory and some very nice diagrams without to come up with code. I only can recommend you to read this blog up and down before you write even one single line of code: http://www.coversant.net/Coversant/Blogs/tabid/88/EntryID/10/Default.aspx. There are some very smart people around they have done great work.

After a while of research, I get to koders.com where I found a strip-down version of their product.

Another very useful page with a lot of sample code around .net is Mike Woodrings's .net Sample Page which contains a bunch of great examples for different domains.

I think I have tried any way to use Sockets now:

• Blocked Sockets
• Unblocked Sockets
• Poll Sockets
• Select

Every prototype I have done so far had his advantages and disadvantages. In the end of this post I will provide 2 downloads:

• prototype with poll sockets on client and server
• protptype with block sockets on client side and async server handling

Once I get comfortable with all the different Issues which come across the write the server part I decided to use Asad Aziz - Asynchronous Server Socket using C# Sample as my base for the server. Its very easy written but my knowledge on this issue was limited until I started to work on it quite intensive and I think I learned every day at least 10 new things :-)

Lets dive into different key parts of the client and the server. An additional issue I would like to mention here is that the whole code is 100% managed.

We gone start first with the server. As already mention the server is working Asynchronous which means we have to handle with IAsyncResult. Upon Server start we begin run the server within a different thread and the client cleanup will be handled by a TimerCallback to purge disconnected clients in case they not removed before upon disconnection.

ThreadStart serverListener = new ThreadStart(this.StartListening);
serverThread[0] = new Thread(serverListener);
serverThread[0].IsBackground = true;
serverThread[0].Start();
TimerCallback timerDelegate = new TimerCallback(this.CheckSockets);
this.lostTimer = new Timer(timerDelegate, null, SharedCacheTcpServer.timerTimeout, SharedCacheTcpServer.timeoutMinutes);

Since we have started now the serverListener we Bind the server IPEndPoint to the requested ip and port and start to listen for connections from clients. Once a connection is received it only will be destroyed in one of the following 2 cases:

1. The Socket has not been used for a certain amount of time
2. Client Socket get disconnected.

With this we avoid to much Server resources. As started we will use a ManualResetEvent to accept only one by one client. I have commented this part for testing purposes and the server started to throw memory exceptions. So keep it simple in the meaning, only start a new BeginAccept once you finished to move the client into AccecptCallback.

private void AccecptCallback(IAsyncResult ar)
{
// signal main thread to continue
this.allDone.Set();
Socket clientListener = ar.AsyncState as Socket;
if (clientListener != null)
{
Socket handler = clientListener.EndAccept(ar);
Console.WriteLine(@"Connected by client: {0}", handler.RemoteEndPoint);
StateObject state = new StateObject();
state.WorkSocket = handler;
state.AliveTimeStamp = DateTime.Now;

... and some more code ..

once you have done all setup for you StateObject (this is the object which contains all different data between the calls) we able to call BeginReceive. SharedCache Protocol between client and server keep simple: [messageLength][message] so if we are not able to get the whole message at once we have to call several times BeginReceive until we have received everything from the client and we able to run the custom stuff on the server.

Once we done with our reading we can start to process received package and process with it. To keep server resources (amount of threads) under control I decided to use Mike's Threadpool.


I think the 2 most important parts are within this message is to know that we need to read the header which indicates to message length and if we need to read more data for this message or do we can proceed it arrived data.

// check for header
if (state.ReadHeader) { .....


// check for message length -> TODO: how to check longer values as int count in list????
if (state.DataBuffer.ToArray().LongLength == state.MessageLength) { ...


as you can see there is even one open point since I use in my state object a List. I had some discussions with people on asp.net forums but I believe i can life right now with this
int.MaxValue as maximal message length ;-)


the next key position in above code print screen is that we use at this place again handler.BeginReceive() since we want to client connection keep opened as long as possible. Some previous tests had shown me while i open and close 1000 Sockets the time reduction to use always the same is more then 50%. Before we can post the request into ThreadPool we need to remove the Range of our message size [8 bytes] that's why we do: state.DataBuffer.RemoveRange(0,8); once we have posted it to the threadpool to pool handles the message sending the Echo to the waiting client.
HandleClientMessage is the place which is called after you get the free Thread from the ThreadPool - here you can manage your Server side stuff and once you done you can send it back to the Client. If you work within Winform environment a more event driven design would be correct but for sharedcache this would not be correct since the client is waiting for the server response.



in the above example we do nothing else then to set an Attribute of the object IndexusMessage to Successful.
The response from the server needs to be prepared, then also the client waits for the same struct [message length][message data]. Therefore we call UtilByte.CreateMessageHeader(with the msg.GetBytes() which returns a byte array) and we combine both values to 1 single byte array. since this is done we can free up resources and set the actual messageLength on the state so we know how much data we need to send to the client which is not less important.



Within the Send() method we start the async call: socket.BeginSend() or if we do not have anything to send back (this case does not happen here) we reset all state data and start to receive again without to destroy the state and the socket which is included in the state. The
same idea happens within the SendCallback() we call BeginSend() as long we have data to send and once we sent everything to the client we reset the state and start to receive again with socket.BeginReceive().

These are all important parts on the server side, I will wrap up the client in a different post within the next few days.

The 2 best code-solution I have found on the web are the following 2 links:

http://www.codeproject.com/KB/IP/Generic_TCP_IP_server.aspx
http://www.codeproject.com/KB/IP/AsyncSocketServerandClien.aspx

Both of them could not handle the case I needed so I hope people can use the Prototypes in this or another way. It would be great if you let me know if you are using them. So here are the downloads:

• Polling (Client and Server)
• Async Server and Block client with manager to connect to several servers

I really would like to say "thank you very much" to all contributors of the following articles and coding samples:

• Atif Aziz for all his good suggestions about how to handle the different issues
• Gil Y. - Generic TCP/IP Client / Server
• Andre Azevedo - An Asynchronous Socket Server and Client
• Mike Woodring - Custom Thread Pool
• Chris Mullins - Windows Sockets and Threading: How well does it scale?
• Mark Strawmyer - Communication over Sockets: Blocking vs Unblocking