Multi-threaded WebService: “Unable to connect to remote server” April 24, 2007
Posted by codinglifestyle in ASP.NET, IIS, Parallelism.Tags: IIS, Parallelism, ports, tcp/ip, threads, web service
2 comments
You know you’ve made it in to hackerdom when 4000 ports just isn’t enough. I need more power!
I have a webservice which spawns, potentially thousands, of threads. These in turn are calling a webservice in SharePoint to perform a real-time query (or else we could make a single call to the search service which relies on the index being up-to-date). I did think to include a throttle which would restrict the total number of threads spawned across all calls to the webmethod. However, even with this number safely at 100 threads I didn’t account for TCP/IP’s default settings keeping the port alive 4 minutes. It didn’t take long to put around 4000 ports in a TIME_WAIT state. When my webservice would make a call to the SharePoint webservice this would result in a System.Net.WebException:
“Unable to connect to remote server” with an inner exception of:
“Only one usage of each socket address (protocol/network address/port) is normally permitted”
The solution was simple enough and is common across a number of greedy applications. TCP/IP allows us to up the ante by changing a few parameters in the registry. This allows you to adjust the delay before a port is available again. In addition you may increase the number of ports at your disposal by nearly 60,000. If this isn’t enough, maybe a design change is in order!
Here’s how you do it:
The product handles heavy query volumes more efficiently if the following TCP/IP parameters are set in the Windows® registry:
1. Start the registry editor:
a. Click the Windows Start button.
b. Click Run.
c. Type regedit in field provided.
d. Click OK
2. Use the following directory path to navigate to the registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. In the right pane of the registry editor, look for the TcpTimedWaitDelay value name. If it is not there, add it by selecting EditNewDWORD Value from the menu bar. Type the value name TcpTimedWaitDelay in the name box that appears with the flashing cursor.
Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.
4. Double-click inside the right pane again to set the value of TcpTimedWaitDelay. Select Decimal as the Base, and enter 30 in the Value data field.
5. In the right pane of the registry editor, look for the MaxUserPort value name. If it is not there, add it by selecting EditNewDWORD Value from the menu bar. Type the value name MaxUserPort in the name box that appears with the flashing cursor.
Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.
6. Double-click inside the right pane again to set the value of MaxUserPort. Select Decimal as the Base, and enter 65534 in the Value data field.
7. You must restart Windows for these settings to take effect
Reference: http://blogs.msdn.com/dgorti/archive/2005/09/18/470766.aspx
Update!
After issuing the above changes were made and the customer finally rebooted the server I ran in to an new, exciting System.Net.WebException:
“The underlying connection was closed: An unexpected error occurred on a send.” with a status of: SendFailure
Searching for this exception lead me to exactly the information I was looking for. Let’s close down those ports as soon as we are done with them. I have no desire to use up thousands of ports and wear the hacker crown. We can do this by changing the keep alive state of our web request to false. Funny that this isn’t by default false for a webservice nor is it a public member to set like the timeout. We have two choices, the high road and the low road. First the low road:
We will alter generated proxy class directly; meaning your fix may be lost if you update your web reference. In the GetWebRequest function the KeepAlive property must be set to false. This can be accomplished by following these steps:
- Add a Web Reference using the normal way (if you haven’t already added one ofcourse).
- Make sure Show All Files menu item is enable in the Project menu.
- In the Solution Explorer window, navigate to:
- Web References
-
- Reference.map
- Reference.cs (or .vb)
- Reference.map
-
- Web References
- Open the Reference.cs file and add following code in the webservice proxy class:
-
protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
System.Net.HttpWebRequest webRequest =
(System.Net.HttpWebRequest)base.GetWebRequest(uri);
webRequest.KeepAlive = false;
webRequest.ProtocolVersion = HttpVersion.Version10;
return webRequest;
}
-
The high road, or proper way of doing this, is to subclass our webservice such that we don’t touch the auto-generated proxy class. Here is a sample:
using System;
using System.Net;
using System.Reflection;// Web service reference entered in wizard was “MyWebService.MyWebServiceWse”
using MyProject.MyWebService;namespace MyProject
{
public class MySubClassedWebService : MyWebServiceWse
{
private static PropertyInfo requestPropertyInfo = null;public MySubClassedWebService(){}
protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);// Retrieve property info and store it in a static member for optimizing future use
if (requestPropertyInfo==null)
requestPropertyInfo = request.GetType().GetProperty(“Request”);// Retrieve underlying web request
HttpWebRequest webRequest = (HttpWebRequest)requestPropertyInfo.GetValue(request, null);// Setting KeepAlive
webRequest.KeepAlive = false;
// Setting protocol version
webRequest.ProtocolVersion = HttpVersion.Version10;
return request;
}
}
}
Reference: http://weblogs.asp.net/jan/archive/2004/01/28/63771.aspx
IIS Madness: 401, Application Pool identities, and IP addresses March 1, 2007
Posted by codinglifestyle in IIS, Security.Tags: 401, application pool, aspnet_regiis, authentication, IIS, ip address, kerberos, ntlm
add a comment
|
1.
|
Start a command prompt.
|
|
2.
|
Locate and then change to the directory that contains the Adsutil.vbs file. By default, this directory is C:\Inetpub\Adminscripts.
|
|
3.
|
Type the following command, and then press ENTER:
cscript adsutil.vbs set w3svc/NTAuthenticationProviders “NTLM”
|
|
4.
|
To verify that the NtAuthenticationProviders metabase property is set to NTLM, type the following command, and then press ENTER:
cscript adsutil.vbs get w3svc/NTAuthenticationProviders
The following text should be returned:
NTAuthenticationProviders : (STRING) “NTLM”
|
Caching woes in the garden November 6, 2006
Posted by codinglifestyle in ASP.NET, IIS.Tags: caching, IIS, web garden
add a comment
Over the past few weeks I have received numerous bugs on a SharePoint project I’ve been trying to close. This project had a lot of custom webparts and added functionality. These various reports all centered on inconsistent behavior resulting from the cache being lost. The thing was everything worked fine in the development environment (of course). What was causing the server to give up its cache so quickly? I tried varying settings in the web.config, monitored application recycles, monitored if the virus scanner was touching web or machine.config. I tried PartCache, Application State, Page.Cache, and tweaked every setting they had. I was even reduced to using (shudder) hidden fields in a few cases.
Finally I wrote a webpart for testing the cache. It allowed you to submit a value or simply refresh (postback). The webpart stored and displayed the value in every available type of cache: View State, Cache, WP Shared, WP Personal, Session, AppState.
I quickly discovered an odd behavior. The cache was cycling between values. I’d submit “one”, “two”, “three” and then simply postback and see the cache cycle between these 3 values. I shut down every other web site on the server and opened task manager. Lo and behold I saw three, count em three, w3wp.exe processes running.
Finally the penny dropped and I opened the application pool properties. On the performance tab there is an innocuous setting at the bottom called Web garden. While I was away on holiday the customer had been complaining about performance so someone had set this to 3 and forgot to tell anyone. This simple action, which does not flash any sirens, bells, or whistles, completely changes the environment in which the code runs. A web garden has multiple processes; a server farm has multiple physical machines with a load balancer. The effect on cache is the same. In a garden/farm environment you must persist cache using a custom provider, namely a SQL database which persists a single copy of the cache so that multiple servers/processes all point to a single copy.
You can imagine the impact this had on a code which relied on various types of caching, either to cache data or to maintain state for an individual user. By changing this single setting back to 1 process, everything was back in order. If I had a swear jar, I would be destitute. Here is what MSDN has to say for those who read about the ramifications of their changes before doing:
“Because Web gardens enable the use of multiple processes, each process will have its own copy of application state, in-process session state, caches, and static data. Web gardens should not be used for all applications, especially if they need to maintain state. Be sure to benchmark the performance of the application before deciding whether Web garden mode is appropriate.
When using a Web garden, it is important to understand how session state and round robin works. It is also important to consider of how other application pool settings affect the application.”
So, the lesson learned? Next time you have a funky caching problem add this to your list of things to check. Also, swear jars and programmers don’t mix.
Programatically recycle an IIS application pool October 20, 2006
Posted by codinglifestyle in ASP.NET, IIS.Tags: application pool, DirectoryServices, IIS, recycle
3 comments
One of the many banes in my life is waiting for IIS to reset. When developing certain components, like a webpart or webcontrol, I often use a post-build event to gac the assembly and issue an iisreset. This is necessary for IIS to pick up the latest assembly from the GAC. Recycling the app pool also forces IIS to reload the assembly, but I never could find a command I could issue from my post-build batch file to automate this. Well, now I don’t need to look because I just wrote my own utility to recycle the application pool:
using System;
using System.Collections.Generic;
using System.Text;
using System.DirectoryServices;
namespace AppPoolRecycler
{
class Program
{
static void Main(string[] args)
{
if (args.Length != 2)
{
Console.WriteLine(“Usage:\tapppoolrecycler.exe machine apppool\n\tapppoolrecycler.exe localhost mypool”);
return;
}
try
{
string sMachine = args[0];
string sAppPool = args[1];
string sPath = “IIS://” + sMachine + “/W3SVC/AppPools/” + sAppPool;
Console.WriteLine(sPath);
DirectoryEntry w3svc = new DirectoryEntry(sPath);
w3svc.Invoke(“Recycle”, null);
Console.WriteLine(“Application pool recycled”);
}
catch (Exception ex)
{
Console.WriteLine(“Error: ” + ex.Message);
}
}
}
}
Now in my post build event I can gac my assembly and recycle the app pool which is much faster than a reset.
AppPoolRecycler.exe sanpaula mssharepointportalapppool
Take a look at this blog entry for more information: http://blogs.aspitalia.com/daniele/post555/Riciclare-Application-Pool-IIS-Codice-CSharp.aspx
———————–
Update! I’ve recently come across a way to do this using a script which should be installed on W2003:
c:\windows\system32\iisapp.vbs /a mssharepointportalapppool /r
Coding Lifestyle 