SoapExtensions: A Bad Day with HTTP 400 Bad Requests December 5, 2012
Posted by codinglifestyle in ASP.NET, CodeProject, IIS.Tags: 400, Bad Request, iis7.5, soap, SoapExtension, web service, web.config, xml
1 comment so far
You may have found this post if you were searching for:
- HTTP 400 Bad Request web service
- Response is not well-formed XML web service
- System.Xml.XmlException: Root element is missing web service
- SoapExtension impacting all web services
Yesterday I was debugging an inconsistent issue in production. Thankfully we could track trending recurring errors and began to piece together all incoming and outgoing webservices were being negatively impacted for unknown reasons. This was creating a lot of pressure as backlogs of incoming calls were returning HTTP 400 Bad Request errors. Outgoing calls were silently failing without a facility to retrigger the calls later creating manual work.
We suspected SSO or SSL leading us to change settings in IIS. Being IIS 7.5 this touched the web.config which recycles the app pool. Every time a setting in IIS was changed or an iisreset was issued it seemed to rectify the situation. But after an indeterminate amount of time the problems would resurface.
The culprit ended up being a SoapExtension. The SoapExtension modifies the soap header for authentication when making outgoing calls to a java webservice.
<?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <SOAP-ENV:Header> <h:BasicAuth xmlns:h="http://soap-authentication.org/basic/2001/10/" SOAP-ENV:mustUnderstand="1"> <Name>admin</Name> <Password>broccoli</Password> </h:BasicAuth> </SOAP-ENV:Header> <SOAP-ENV:Body> <m:echoString xmlns:m="http://soapinterop.org/"> <inputString>This is a test.</inputString> </m:echoString> </SOAP-ENV:Body> </SOAP-ENV:Envelope>
It does this with a dynamically loaded (that bit was my fault and made it a complete bitch to debug) SoapExtension taken from a legacy command line util which did this every 5 minutes:
This existed simply because nobody could figure out how to call the webservice directly within the web application. Once incorporated when the web service was called, perhaps hours from an iisreset, the SoapExtension is dynamically loaded. The bug was, even though it was coded to not affect anything but Vendavo, the checks performed were performed too late and therefore all web services, incoming and outgoing, were impacted.
Previously the check was in the After Serialize message handler. The fix was to return the original stream in the ChainStream. The hard part was knowing what webservice was making the call before ChainStream was called. The check was moved to:
public overrides void Initialize(Object initializer)
The initializer object was tested setting a flag used in ChainStream to determine which stream was returned.
So lesson learned, beware SoapExtensions may impact all soap calls. While you can specify a custom attribute to limit the extension to web methods you publish you cannot use this filtering mechanism on webservices you consume. This means you must self-filter or risk affecting all incoming and outgoing web services unintentionally.
Also, dynamically loading a setting which belongs in the web.config was a dumb idea which delayed identification of the problem. Now we use this:
<system.web>
<webServices>
<soapExtensionTypes>
<add type="SBA.Data.PMMSoapExtension, SBA.Data" priority="1" group="High" />
</soapExtensionTypes>
</webServices>
</system.web>
Ref:
http://www.hanselman.com/blog/ASMXSoapExtensionToStripOutWhitespaceAndNewLines.aspx
http://msdn.microsoft.com/en-ie/magazine/cc164007(en-us).aspx
Set IE7 Compatibility Flag for IE8 in IIS May 11, 2009
Posted by codinglifestyle in IIS.1 comment so far
Now that IE8 has officially been pushed out via Windows Update it’s time to get serious about testing your web site compatibility. As you probably know, one of the big changes in IE8 is complying to standards for increased interoperability. In other words, Microsoft won’t be going it’s own way with standards anymore. Out of the box, IE8 complies with standards that many pages developed with IE6 or IE7 in mind won’t adhere to. This means customers may have serious layout and style issues when their customers start browsing with IE8.
There is a meta flag that can be set set to tell IE8 to automatically switch to compatiblity view, which will render the page the same as IE7. This is fine if you have a masterpage, but in an older app with many pages this is not ideal. Luckily, there is a dead easy way of setting a custom HTTP header which will tell IE8 the same thing.
- Open iismgr
- Open website properties
- Click HTTP Headers tab
- Click Add
- For customer header name enter X-UA-Compatible
- For custom header value enter IE=EmulateIE7
- Click OK
Done! From now on IE8 browsers will default to compatibility mode and won’t even offer the user the option of viewing the side with compatibility mode off.
Multi-threaded WebService: “Unable to connect to remote server” April 24, 2007
Posted by codinglifestyle in ASP.NET, IIS, Parallelism.Tags: IIS, Parallelism, ports, tcp/ip, threads, web service
2 comments
You know you’ve made it in to hackerdom when 4000 ports just isn’t enough. I need more power!
I have a webservice which spawns, potentially thousands, of threads. These in turn are calling a webservice in SharePoint to perform a real-time query (or else we could make a single call to the search service which relies on the index being up-to-date). I did think to include a throttle which would restrict the total number of threads spawned across all calls to the webmethod. However, even with this number safely at 100 threads I didn’t account for TCP/IP’s default settings keeping the port alive 4 minutes. It didn’t take long to put around 4000 ports in a TIME_WAIT state. When my webservice would make a call to the SharePoint webservice this would result in a System.Net.WebException:
“Unable to connect to remote server” with an inner exception of:
“Only one usage of each socket address (protocol/network address/port) is normally permitted”
The solution was simple enough and is common across a number of greedy applications. TCP/IP allows us to up the ante by changing a few parameters in the registry. This allows you to adjust the delay before a port is available again. In addition you may increase the number of ports at your disposal by nearly 60,000. If this isn’t enough, maybe a design change is in order!
Here’s how you do it:
The product handles heavy query volumes more efficiently if the following TCP/IP parameters are set in the Windows® registry:
1. Start the registry editor:
a. Click the Windows Start button.
b. Click Run.
c. Type regedit in field provided.
d. Click OK
2. Use the following directory path to navigate to the registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. In the right pane of the registry editor, look for the TcpTimedWaitDelay value name. If it is not there, add it by selecting EditNewDWORD Value from the menu bar. Type the value name TcpTimedWaitDelay in the name box that appears with the flashing cursor.
Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.
4. Double-click inside the right pane again to set the value of TcpTimedWaitDelay. Select Decimal as the Base, and enter 30 in the Value data field.
5. In the right pane of the registry editor, look for the MaxUserPort value name. If it is not there, add it by selecting EditNewDWORD Value from the menu bar. Type the value name MaxUserPort in the name box that appears with the flashing cursor.
Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.
6. Double-click inside the right pane again to set the value of MaxUserPort. Select Decimal as the Base, and enter 65534 in the Value data field.
7. You must restart Windows for these settings to take effect
Reference: http://blogs.msdn.com/dgorti/archive/2005/09/18/470766.aspx
Update!
After issuing the above changes were made and the customer finally rebooted the server I ran in to an new, exciting System.Net.WebException:
“The underlying connection was closed: An unexpected error occurred on a send.” with a status of: SendFailure
Searching for this exception lead me to exactly the information I was looking for. Let’s close down those ports as soon as we are done with them. I have no desire to use up thousands of ports and wear the hacker crown. We can do this by changing the keep alive state of our web request to false. Funny that this isn’t by default false for a webservice nor is it a public member to set like the timeout. We have two choices, the high road and the low road. First the low road:
We will alter generated proxy class directly; meaning your fix may be lost if you update your web reference. In the GetWebRequest function the KeepAlive property must be set to false. This can be accomplished by following these steps:
- Add a Web Reference using the normal way (if you haven’t already added one ofcourse).
- Make sure Show All Files menu item is enable in the Project menu.
- In the Solution Explorer window, navigate to:
- Web References
-
- Reference.map
- Reference.cs (or .vb)
- Reference.map
-
- Web References
- Open the Reference.cs file and add following code in the webservice proxy class:
-
protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
System.Net.HttpWebRequest webRequest =
(System.Net.HttpWebRequest)base.GetWebRequest(uri);
webRequest.KeepAlive = false;
webRequest.ProtocolVersion = HttpVersion.Version10;
return webRequest;
}
-
The high road, or proper way of doing this, is to subclass our webservice such that we don’t touch the auto-generated proxy class. Here is a sample:
using System;
using System.Net;
using System.Reflection;// Web service reference entered in wizard was “MyWebService.MyWebServiceWse”
using MyProject.MyWebService;namespace MyProject
{
public class MySubClassedWebService : MyWebServiceWse
{
private static PropertyInfo requestPropertyInfo = null;public MySubClassedWebService(){}
protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);// Retrieve property info and store it in a static member for optimizing future use
if (requestPropertyInfo==null)
requestPropertyInfo = request.GetType().GetProperty(“Request”);// Retrieve underlying web request
HttpWebRequest webRequest = (HttpWebRequest)requestPropertyInfo.GetValue(request, null);// Setting KeepAlive
webRequest.KeepAlive = false;
// Setting protocol version
webRequest.ProtocolVersion = HttpVersion.Version10;
return request;
}
}
}
Reference: http://weblogs.asp.net/jan/archive/2004/01/28/63771.aspx
IIS Madness: 401, Application Pool identities, and IP addresses March 1, 2007
Posted by codinglifestyle in IIS, Security.Tags: 401, application pool, aspnet_regiis, authentication, IIS, ip address, kerberos, ntlm
add a comment
|
1.
|
Start a command prompt.
|
|
2.
|
Locate and then change to the directory that contains the Adsutil.vbs file. By default, this directory is C:\Inetpub\Adminscripts.
|
|
3.
|
Type the following command, and then press ENTER:
cscript adsutil.vbs set w3svc/NTAuthenticationProviders “NTLM”
|
|
4.
|
To verify that the NtAuthenticationProviders metabase property is set to NTLM, type the following command, and then press ENTER:
cscript adsutil.vbs get w3svc/NTAuthenticationProviders
The following text should be returned:
NTAuthenticationProviders : (STRING) “NTLM”
|
Caching woes in the garden November 6, 2006
Posted by codinglifestyle in ASP.NET, IIS.Tags: caching, IIS, web garden
add a comment
Over the past few weeks I have received numerous bugs on a SharePoint project I’ve been trying to close. This project had a lot of custom webparts and added functionality. These various reports all centered on inconsistent behavior resulting from the cache being lost. The thing was everything worked fine in the development environment (of course). What was causing the server to give up its cache so quickly? I tried varying settings in the web.config, monitored application recycles, monitored if the virus scanner was touching web or machine.config. I tried PartCache, Application State, Page.Cache, and tweaked every setting they had. I was even reduced to using (shudder) hidden fields in a few cases.
Finally I wrote a webpart for testing the cache. It allowed you to submit a value or simply refresh (postback). The webpart stored and displayed the value in every available type of cache: View State, Cache, WP Shared, WP Personal, Session, AppState.
I quickly discovered an odd behavior. The cache was cycling between values. I’d submit “one”, “two”, “three” and then simply postback and see the cache cycle between these 3 values. I shut down every other web site on the server and opened task manager. Lo and behold I saw three, count em three, w3wp.exe processes running.
Finally the penny dropped and I opened the application pool properties. On the performance tab there is an innocuous setting at the bottom called Web garden. While I was away on holiday the customer had been complaining about performance so someone had set this to 3 and forgot to tell anyone. This simple action, which does not flash any sirens, bells, or whistles, completely changes the environment in which the code runs. A web garden has multiple processes; a server farm has multiple physical machines with a load balancer. The effect on cache is the same. In a garden/farm environment you must persist cache using a custom provider, namely a SQL database which persists a single copy of the cache so that multiple servers/processes all point to a single copy.
You can imagine the impact this had on a code which relied on various types of caching, either to cache data or to maintain state for an individual user. By changing this single setting back to 1 process, everything was back in order. If I had a swear jar, I would be destitute. Here is what MSDN has to say for those who read about the ramifications of their changes before doing:
“Because Web gardens enable the use of multiple processes, each process will have its own copy of application state, in-process session state, caches, and static data. Web gardens should not be used for all applications, especially if they need to maintain state. Be sure to benchmark the performance of the application before deciding whether Web garden mode is appropriate.
When using a Web garden, it is important to understand how session state and round robin works. It is also important to consider of how other application pool settings affect the application.”
So, the lesson learned? Next time you have a funky caching problem add this to your list of things to check. Also, swear jars and programmers don’t mix.
Programatically recycle an IIS application pool October 20, 2006
Posted by codinglifestyle in ASP.NET, IIS.Tags: application pool, DirectoryServices, IIS, recycle
3 comments
One of the many banes in my life is waiting for IIS to reset. When developing certain components, like a webpart or webcontrol, I often use a post-build event to gac the assembly and issue an iisreset. This is necessary for IIS to pick up the latest assembly from the GAC. Recycling the app pool also forces IIS to reload the assembly, but I never could find a command I could issue from my post-build batch file to automate this. Well, now I don’t need to look because I just wrote my own utility to recycle the application pool:
using System;
using System.Collections.Generic;
using System.Text;
using System.DirectoryServices;
namespace AppPoolRecycler
{
class Program
{
static void Main(string[] args)
{
if (args.Length != 2)
{
Console.WriteLine(“Usage:\tapppoolrecycler.exe machine apppool\n\tapppoolrecycler.exe localhost mypool”);
return;
}
try
{
string sMachine = args[0];
string sAppPool = args[1];
string sPath = “IIS://” + sMachine + “/W3SVC/AppPools/” + sAppPool;
Console.WriteLine(sPath);
DirectoryEntry w3svc = new DirectoryEntry(sPath);
w3svc.Invoke(“Recycle”, null);
Console.WriteLine(“Application pool recycled”);
}
catch (Exception ex)
{
Console.WriteLine(“Error: ” + ex.Message);
}
}
}
}
Now in my post build event I can gac my assembly and recycle the app pool which is much faster than a reset.
AppPoolRecycler.exe sanpaula mssharepointportalapppool
Take a look at this blog entry for more information: http://blogs.aspitalia.com/daniele/post555/Riciclare-Application-Pool-IIS-Codice-CSharp.aspx
———————–
Update! I’ve recently come across a way to do this using a script which should be installed on W2003:
c:\windows\system32\iisapp.vbs /a mssharepointportalapppool /r
Coding Lifestyle 