Found the below post in my drafts section. I should have released it a year ago but better late than never. Maybe someone with a deadlock problem on IIS and .NET 4.0 will find this piece of information useful.
At the place where I am contracting at the moment, we recently (April 2013) went live with an intranet web application that needs to handle about 1000 concurrent users. This is handled by 8 web-servers running Internet Information Services (IIS) 7.5 and ASP.NET 4.0 and in the beginning, all was well.
After running for about a day, we started seeing deadlocks in the IIS worker processes. At its worst, it happened every five minutes. When a worker process deadlocks, it recycles itself and the system could not handle that. A Microsoft support ticket was opened and the problem was eventually solved by adding the following code in a random .cs file in all of the assemblies (i.e. projects) for the application:
[System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit)]
struct WorkaroundStruct
{
[System.Runtime.InteropServices.FieldOffset(0)]
[System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.I4)]
public int myField1;
}
This magically solved the deadlocks. It turns out that it is a known bug in the .NET 4.0 framework but Microsoft was not going to make a hotfix for it. Hmm.
Deadlocks are always bad but in our case, they were extra bad. All data in the application is fetched through a Windows Communication Foundation (WCF) webservice from a different company. Instantiation of a WCF client can be slow and in our case, it could take 7-9 seconds for one or more of the WCF clients. By the way, while instantiating the client, the CPU goes nuts. Even though we cache the client for each session to improve performance, when the IIS enters a deadlock and resets, all 1000 users have to instantiate a new WCF client and the servers could simply not handle the CPU load.
It is funny what weird issues we run into with technology.