Coding Lifestyle

Multi-threaded WebService: “Unable to connect to remote server” April 24, 2007

You know you’ve made it in to hackerdom when 4000 ports just isn’t enough.  I need more power!

I have a webservice which spawns, potentially thousands, of threads.  These in turn are calling a webservice in SharePoint to perform a real-time query (or else we could make a single call to the search service which relies on the index being up-to-date).  I did think to include a throttle which would restrict the total number of threads spawned across all calls to the webmethod.  However, even with this number safely at 100 threads I didn’t account for TCP/IP’s default settings keeping the port alive 4 minutes.  It didn’t take long to put around 4000 ports in a TIME_WAIT state.  When my webservice would make a call to the SharePoint webservice this would result in a System.Net.WebException:

“Unable to connect to remote server” with an inner exception of:

“Only one usage of each socket address (protocol/network address/port) is normally permitted”

The solution was simple enough and is common across a number of greedy applications.  TCP/IP allows us to up the ante by changing a few parameters in the registry.  This allows you to adjust the delay before a port is available again.  In addition you may increase the number of ports at your disposal by nearly 60,000.  If this isn’t enough, maybe a design change is in order!

Here’s how you do it:

The product handles heavy query volumes more efficiently if the following TCP/IP parameters are set in the Windows® registry:

1.       Start the registry editor:

a.       Click the Windows Start button.

b.       Click Run.

c.       Type regedit in field provided.

d.       Click OK

2.       Use the following directory path to navigate to the registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

3.       In the right pane of the registry editor, look for the TcpTimedWaitDelay value name. If it is not there, add it by selecting Edit > New > DWORD Value from the menu bar. Type the value name TcpTimedWaitDelay in the name box that appears with the flashing cursor.

Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.

4.       Double-click inside the right pane again to set the value of TcpTimedWaitDelay. Select Decimal as the Base, and enter 30 in the Value data field.

5.       In the right pane of the registry editor, look for the MaxUserPort value name. If it is not there, add it by selecting Edit > New > DWORD Value from the menu bar. Type the value name MaxUserPort in the name box that appears with the flashing cursor.

Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.

6.       Double-click inside the right pane again to set the value of MaxUserPort. Select Decimal as the Base, and enter 65534 in the Value data field.

7.       You must restart Windows for these settings to take effect

Reference: http://blogs.msdn.com/dgorti/archive/2005/09/18/470766.aspx

Update! 

After issuing the above changes were made and the customer finally rebooted the server I ran in to an new, exciting System.Net.WebException:

“The underlying connection was closed: An unexpected error occurred on a send.” with a status of: SendFailure

Searching for this exception lead me to exactly the information I was looking for.  Let’s close down those ports as soon as we are done with them.  I have no desire to use up thousands of ports and wear the hacker crown.  We can do this by changing the keep alive state of our web request to false.  Funny that this isn’t by default false for a webservice nor is it a public member to set like the timeout.  We have two choices, the high road and the low road.  First the low road:

We will alter generated proxy class directly; meaning your fix may be lost if you update your web reference.  In the GetWebRequest function the KeepAlive property must be set to false. This can be accomplished by following these steps:

• Add a Web Reference using the normal way (if you haven’t already added one ofcourse).
• Make sure Show All Files menu item is enable in the Project menu.
• In the Solution Explorer window, navigate to:
  • Web References
    • • Reference.map
        • Reference.cs (or .vb)
• Open the Reference.cs file and add following code in the webservice proxy class:

  • protected override System.Net.WebRequest GetWebRequest(Uri uri)

    {

    System.Net.HttpWebRequest webRequest =

                    (System.Net.HttpWebRequest)base.GetWebRequest(uri);

     

          webRequest.KeepAlive       = false;

          webRequest.ProtocolVersion = HttpVersion.Version10;

     

          return webRequest;

    }

     

     

The high road, or proper way of doing this, is to subclass our webservice such that we don’t touch the auto-generated proxy class.  Here is a sample: 

using System;
using System.Net;
using System.Reflection;

// Web service reference entered in wizard was “MyWebService.MyWebServiceWse”
using MyProject.MyWebService;

namespace MyProject
{
 public class MySubClassedWebService : MyWebServiceWse
 {
  private static PropertyInfo requestPropertyInfo = null;

  public MySubClassedWebService(){}

  protected override System.Net.WebRequest GetWebRequest(Uri uri)
  {
   WebRequest request = base.GetWebRequest(uri);

   // Retrieve property info and store it in a static member for optimizing future use
   if (requestPropertyInfo==null)
    requestPropertyInfo = request.GetType().GetProperty(“Request”);

   // Retrieve underlying web request
   HttpWebRequest webRequest = (HttpWebRequest)requestPropertyInfo.GetValue(request, null);

   // Setting KeepAlive
   webRequest.KeepAlive = false;
   // Setting protocol version
   webRequest.ProtocolVersion = HttpVersion.Version10;
   return request;
  }
 }
}

Reference: http://weblogs.asp.net/jan/archive/2004/01/28/63771.aspx