jump to navigation

SoapExtensions: A Bad Day with HTTP 400 Bad Requests December 5, 2012

Posted by codinglifestyle in ASP.NET, CodeProject, IIS.
Tags: , , , , , , ,
1 comment so far

You may have found this post if you were searching for:

  • HTTP 400 Bad Request web service
  • Response is not well-formed XML web service
  • System.Xml.XmlException: Root element is missing web service
  • SoapExtension impacting all web services

Yesterday I was debugging an inconsistent issue in production. Thankfully we could track trending recurring errors and began to piece together all incoming and outgoing webservices were being negatively impacted for unknown reasons. This was creating a lot of pressure as backlogs of incoming calls were returning HTTP 400 Bad Request errors. Outgoing calls were silently failing without a facility to retrigger the calls later creating manual work.

We suspected SSO or SSL leading us to change settings in IIS. Being IIS 7.5 this touched the web.config which recycles the app pool. Every time a setting in IIS was changed or an iisreset was issued it seemed to rectify the situation. But after an indeterminate amount of time the problems would resurface.

The culprit ended up being a SoapExtension. The SoapExtension modifies the soap header for authentication when making outgoing calls to a java webservice.

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<SOAP-ENV:Header>
<h:BasicAuth xmlns:h="http://soap-authentication.org/basic/2001/10/"
SOAP-ENV:mustUnderstand="1">
<Name>admin</Name>
<Password>broccoli</Password>
</h:BasicAuth>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<m:echoString xmlns:m="http://soapinterop.org/">
<inputString>This is a test.</inputString>
</m:echoString>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

It does this with a dynamically loaded (that bit was my fault and made it a complete bitch to debug) SoapExtension taken from a legacy command line util which did this every 5 minutes:

Vendavo Sequence Diagram

This existed simply because nobody could figure out how to call the webservice directly within the web application.  Once incorporated when the web service was called, perhaps hours from an iisreset, the SoapExtension is dynamically loaded.  The bug was, even though it was coded to not affect anything but Vendavo, the checks performed were performed too late and therefore all web services, incoming and outgoing, were impacted.

SoapExtension Lifecycle

Previously the check was in the After Serialize message handler.  The fix was to return the original stream in the ChainStream.  The hard part was knowing what webservice was making the call before ChainStream was called. The check was moved to:

public overrides void Initialize(Object initializer)

The initializer object was tested setting a flag used in ChainStream to determine which stream was returned.

So lesson learned, beware SoapExtensions may impact all soap calls.  While you can specify a custom attribute to limit the extension to web methods you publish you cannot use this filtering mechanism on webservices you consume.  This  means you must self-filter or risk affecting all incoming and outgoing web services unintentionally.

Also, dynamically loading a setting which belongs in the web.config was a dumb idea which delayed identification of the problem. Now we use this:

<system.web>
   <webServices>
      <soapExtensionTypes>
         <add type="SBA.Data.PMMSoapExtension, SBA.Data" priority="1" group="High" />
      </soapExtensionTypes>
   </webServices>
</system.web>

Ref:
http://www.hanselman.com/blog/ASMXSoapExtensionToStripOutWhitespaceAndNewLines.aspx

http://msdn.microsoft.com/en-ie/magazine/cc164007(en-us).aspx

Multi-threaded WebService: “Unable to connect to remote server” April 24, 2007

Posted by codinglifestyle in ASP.NET, IIS, Parallelism.
Tags: , , , , ,
2 comments

You know you’ve made it in to hackerdom when 4000 ports just isn’t enough.  I need more power!

 

I have a webservice which spawns, potentially thousands, of threads.  These in turn are calling a webservice in SharePoint to perform a real-time query (or else we could make a single call to the search service which relies on the index being up-to-date).  I did think to include a throttle which would restrict the total number of threads spawned across all calls to the webmethod.  However, even with this number safely at 100 threads I didn’t account for TCP/IP’s default settings keeping the port alive 4 minutes.  It didn’t take long to put around 4000 ports in a TIME_WAIT state.  When my webservice would make a call to the SharePoint webservice this would result in a System.Net.WebException:

 

“Unable to connect to remote server” with an inner exception of:

 

“Only one usage of each socket address (protocol/network address/port) is normally permitted”

 

The solution was simple enough and is common across a number of greedy applications.  TCP/IP allows us to up the ante by changing a few parameters in the registry.  This allows you to adjust the delay before a port is available again.  In addition you may increase the number of ports at your disposal by nearly 60,000.  If this isn’t enough, maybe a design change is in order!

 

Here’s how you do it:

 

The product handles heavy query volumes more efficiently if the following TCP/IP parameters are set in the Windows® registry:

1.       Start the registry editor:

a.       Click the Windows Start button.

b.       Click Run.

c.       Type regedit in field provided.

d.       Click OK

2.       Use the following directory path to navigate to the registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

3.       In the right pane of the registry editor, look for the TcpTimedWaitDelay value name. If it is not there, add it by selecting Edit > New > DWORD Value from the menu bar. Type the value name TcpTimedWaitDelay in the name box that appears with the flashing cursor.

Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.

4.       Double-click inside the right pane again to set the value of TcpTimedWaitDelay. Select Decimal as the Base, and enter 30 in the Value data field.

5.       In the right pane of the registry editor, look for the MaxUserPort value name. If it is not there, add it by selecting Edit > New > DWORD Value from the menu bar. Type the value name MaxUserPort in the name box that appears with the flashing cursor.

Note: If you do not see the a flashing cursor and New Value # inside the box, right-click inside the right panel and select Rename from the menu, then type the value name TcpTimedWaitDelay in the name box.

6.       Double-click inside the right pane again to set the value of MaxUserPort. Select Decimal as the Base, and enter 65534 in the Value data field.

7.       You must restart Windows for these settings to take effect

 

Reference: http://blogs.msdn.com/dgorti/archive/2005/09/18/470766.aspx

 

 

 

Update! 

 

After issuing the above changes were made and the customer finally rebooted the server I ran in to an new, exciting System.Net.WebException:

 

The underlying connection was closed: An unexpected error occurred on a send.” with a status of: SendFailure

 

Searching for this exception lead me to exactly the information I was looking for.  Let’s close down those ports as soon as we are done with them.  I have no desire to use up thousands of ports and wear the hacker crown.  We can do this by changing the keep alive state of our web request to false.  Funny that this isn’t by default false for a webservice nor is it a public member to set like the timeout.  We have two choices, the high road and the low road.  First the low road:

We will alter generated proxy class directly; meaning your fix may be lost if you update your web reference.  In the GetWebRequest function the KeepAlive property must be set to false. This can be accomplished by following these steps:

  • Add a Web Reference using the normal way (if you haven’t already added one ofcourse).
  • Make sure Show All Files menu item is enable in the Project menu.
  • In the Solution Explorer window, navigate to:
    • Web References
        • Reference.map
          • Reference.cs (or .vb)
  • Open the Reference.cs file and add following code in the webservice proxy class:
    • protected override System.Net.WebRequest GetWebRequest(Uri uri)

      {

      System.Net.HttpWebRequest webRequest =

                      (System.Net.HttpWebRequest)base.GetWebRequest(uri);

       

            webRequest.KeepAlive       = false;

            webRequest.ProtocolVersion = HttpVersion.Version10;

       

            return webRequest;

      }

       

       

The high road, or proper way of doing this, is to subclass our webservice such that we don’t touch the auto-generated proxy class.  Here is a sample: 

 

using System;
using System.Net;
using System.Reflection;

// Web service reference entered in wizard was “MyWebService.MyWebServiceWse”
using MyProject.MyWebService;

namespace MyProject
{
 public class MySubClassedWebService : MyWebServiceWse
 {
  private static PropertyInfo requestPropertyInfo = null;

  public MySubClassedWebService(){}

  protected override System.Net.WebRequest GetWebRequest(Uri uri)
  {
   WebRequest request = base.GetWebRequest(uri);

   // Retrieve property info and store it in a static member for optimizing future use
   if (requestPropertyInfo==null)
    requestPropertyInfo = request.GetType().GetProperty(“Request”);

   // Retrieve underlying web request
   HttpWebRequest webRequest = (HttpWebRequest)requestPropertyInfo.GetValue(request, null);

   // Setting KeepAlive
   webRequest.KeepAlive = false;
   // Setting protocol version
   webRequest.ProtocolVersion = HttpVersion.Version10;
   return request;
  }
 }
}

 

 

 

Reference: http://weblogs.asp.net/jan/archive/2004/01/28/63771.aspx