Thought Flow

Tag: webservice

  • Fixing an IIS deadlock

    Found the below post in my drafts section. I should have released it a year ago but better late than never. Maybe someone with a deadlock problem on IIS and .NET 4.0 will find this piece of information useful.

    At the place where I am contracting at the moment, we recently (April 2013) went live with an intranet web application that needs to handle about 1000 concurrent users. This is handled by 8 web-servers running Internet Information Services (IIS) 7.5 and ASP.NET 4.0 and in the beginning, all was well.

    After running for about a day, we started seeing deadlocks in the IIS worker processes. At its worst, it happened every five minutes. When a worker process deadlocks, it recycles itself and the system could not handle that. A Microsoft support ticket was opened and the problem was eventually solved by adding the following code in a random .cs file in all of the assemblies (i.e. projects) for the application:

    [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Explicit)]
struct WorkaroundStruct
{
    [System.Runtime.InteropServices.FieldOffset(0)]
    [System.Runtime.InteropServices.MarshalAs(System.Runtime.InteropServices.UnmanagedType.I4)]
    public int myField1;
}

    This magically solved the deadlocks. It turns out that it is a known bug in the .NET 4.0 framework but Microsoft was not going to make a hotfix for it. Hmm.

    Deadlocks are always bad but in our case, they were extra bad. All data in the application is fetched through a Windows Communication Foundation (WCF) webservice from a different company. Instantiation of a WCF client can be slow and in our case, it could take 7-9 seconds for one or more of the WCF clients. By the way, while instantiating the client, the CPU goes nuts. Even though we cache the client for each session to improve performance, when the IIS enters a deadlock and resets, all 1000 users have to instantiate a new WCF client and the servers could simply not handle the CPU load.

    It is funny what weird issues we run into with technology.

  • Antecons developer diary part 1

    This post is about Antecons, a product recommendation engine, now part of Conversio. Antecons is no longer commercially available, but I have kept my developer diary on my website with permission.

    It has been a year since the first announcement of Antecons. What has happened since then? Absolutely nothing. The reason for this seems to be pretty straightforward: I have been occupied with full-time consulting and I have made a deliberate choice of working a five-day 40-hour work week. That leaves no room for extra work. However, this has changed slightly. Since October 1, I have had a four-day work week. This leaves one day to do other stuff and I have decided to continue my work on Antecons and I intend to blog about the (slow) progress.

    Consider this part 1 in a series of posts about my experiences, design decisions and the current state of Antecons. First, I would like to fill in the gap in the history of Antecons from last year until now. There is more to the story than just a lack of time.

    Where we left off

    One year ago, I made a promise, a prototype and a teaser. The prototype is a recommendation engine that is still running on one webshop (as far as I know). It is written in C# for .NET, uses Microsoft SQL server as database and is deployed directly on the end-user’s server using a pre-compiled DLL that can be referenced from any .NET application. As I was developing the prototype, I already had a couple of basic objections to my own design:

    1. It required the user to use .NET and SQL Server.
    2. It required me to use Visual Studio, IIS and Windows as a development environment.
    3. It required the user to integrate the component into their webshop. This included:
      1. Setting up an SQL database and tables conforming with the Antecons data format.
      2. Synchronizing webshop purchases with the Antecons database and keeping it updated.
      3. Manually running the recommendation algorithm to pre-create recommendations.
      4. Integrating Antecons into the webshop design in the appropriate places to show the pre-created recommendations for the current product being shown.

    The first two points were easily fixed by deciding to replace the pre-compiled DLL component with a webservice. Using an API for Antecons, there would be no constraints on the user regarding their server software or webshop programming environment.

    And thus was born the teaser, a web application written in Python and running on Google App Engine. It never turned into a webservice with an API. Instead it was a website that showed what the essence of Antecons was: A recommendation engine based on association rule mining (or market basket analysis). As it turned out, this was the easy part and implementing a simple API would have been too. But the third point I outlined above, that was the real killer.

    The promise

    A quote from my company website:

    Metacircle has been developing a data analysis tool for some time. It is called Antecons.

    And from the Antecons teaser website:

    The API is currently under development…

    These quotes sum up the promise I made myself (and the world): To work on Antecons. Did I succeed? No. Do I feel bad about that? Yes. What killed the project? Time and wishful thinking.

    Time grew short when I started working full-time but this is only part of the explanation. As hinted above, I was also held back by the third design objection. Setting up Antecons for actual usage was, in my opinion, too difficult and too much an obstacle for widespread usage. How could Antecons be made so it would be super easy to setup, super easy to deploy and super easy to run. The ideal scenario looked like this:

    1. User registers for Antecons and receives a unique User ID.
    2. User inserts 10 lines of Javascript code in the header of each webshop page where Antecons should show recommendations.
    3. Antecons takes care of the rest.

    This is how e.g. Google Analytics works and it is quite amazing how much information it gathers and how useful this information is, based on such a simple installation. If Antecons could be as easy to install as Google Analytics, which is installed on millions of websites, there would be a huge potential and many users could benefit from it.

    But the difficult questions piled up fast. For example, how would Antecons gather information on user’s sales data and how would it know where to show recommendations on a user’s webshop. The answers I came up with all lead to the same conclusion: The user would have to do some work and Antecons could not be fully automated, not even partially. This shattered my overall vision for Antecons and my head has been throwing around ideas ever since. Maybe the ideal scenario was wishful thinking but in any case, this was the end of the line. Until now.

    Where we are

    It has been two weeks since I started over with the Antecons project. In the next blog post, I will elaborate on what specific design obstacles that put the project on ice. This is an important pretext for what I have started working on right now. Later, I will write more about specific design decisions and actual implementation. The API should even have some partial functionality coming up very soon. Stay tuned.

  • Antecons

    “Release fast and iterate“. In some parts of the computing world this is the mantra and I think it’s time for an early announcement: Metacircle has now released an actual product (kind of). It is called Antecons, short for antecedent – consequent. Antecons is a small and simple component that suggests items based on previous usage data (such as purchases) and it is currently being tested on vildmedvin.dk.

    Why did I create Antecons? Well, there are several reasons, actually.

    • I love data mining and business intelligence
    • Good recommendation engines improve conversion rates and thus profit for e-businesses.
    • Recommendation engines are not available to everyone at a low price. Most current solutions like avail.net target big companies and corporations. Antecons targets everyone.
    • People have already shown interest in Antecons. This includes my faithful partners at MCB as well as the founder of Remove The Background

    Let’s keep it real. The business intelligence market is very competitive and many companies are years of research ahead of me. I do not plan to steal their customers. I plan on making advanced technology available to the masses.

    In its current form, Antecons is simply a .NET DLL component with a public API for interacting with Antecons. It needs access to an SQL server and will thus only work with ASP.NET applications. I am currently exploring the possibilities of making Antecons a software-as-a-service (SaaS) product and a proof-of-concept prototype is being built for the Google App Engine, although it has not been finally decided where to host the first SaaS version. You can check it out here.

    If Antecons gains traction, I will give up consulting and work solely on making it an excellent and easy-to-use product. For now, it has been a major personal success to see something get out there. I hope the interest continues.

  • Basic Http authentication from .NET to PHP webservice

    Today, I had a major problem that had a not-so-complex-but-rather-stupid solution so I thought it might be worth sharing since I only found online solutions that were rather old.

    I have remote access to a PHP webservice, written using NuSOAP. I call the methods of this webservice from a .NET application. This is no problem at all and I can easily call most webservice methods from within .NET. But some of the methods require basic authentication and now the trouble begins.

    One would think that it would be enough to just set the credentials for the webservice and make sure that it authenticates on every request. Like this:

    MyPhpWebservice ws = new MyPhpWebservice();
ws.PreAuthenticate = true;
CredentialCache cache = new CredentialCache();
cache.Add(
   new Uri(ws.Url),
   "Basic", 
   new NetworkCredential(wsUsername, wsPassword)
);
ws.Credentials = cache;
ws.CallSomeFunctionThatNeedsAuth();

    Alas, this did not work. In many places online, people say that .NET does not actually send the authentication details on the first request for some webservice function. If this was true, the solution would simply be:

    ws.CallSomeFunctionThatNeedsAuth(); // Does not auth
ws.CallSomeFunctionThatNeedsAuth(); // Should auth now?

    This did not work either. Other people in other places online say that one needs to manually add the authentication details to the HttpWebRequest header. But the Visual Studio auto-created MyPhpWebservice class does not expose its WebRequest object so it cannot be directly altered. However, we can override the method GetWebRequest for the webservice and that way manually add the authentication header. This took me a while to figure out but here is a simple solution:

    class MyOverriddenPhpWebservice : MyPhpWebservice
{
    protected override System.Net.WebRequest GetWebRequest(Uri uri)
    {
        HttpWebRequest request;
        request = (HttpWebRequest)base.GetWebRequest(uri);

        if (PreAuthenticate)
        {
            NetworkCredential cred =
                Credentials.GetCredential(uri, "Basic");
            if (cred != null)
            {
                byte[] credBuffer = new UTF8Encoding().GetBytes(
                    cred.UserName + ":" + cred.Password);
                request.Headers["Authorization"] =
                    "Basic " + Convert.ToBase64String(credBuffer);
            }
        }
        return request;
    }
}

    The code that I wrote in Listing 1 is now perfectly valid and all I need to do is exchange MyPhpWebservice with MyOverriddenPhpWebservice.

    And the best part: It works :-)