Complete independence

Jump I

I am now jobless. I am now independent. For the next few months, I am going to work on projects that do not provide any steady income and I have not planned to take on any consulting or freelance work. I am writing this post both for people that would be interested in what I am doing but also to wrap my ahead around what this is all about.

What is the plan

I am taking some time off from paid jobs. Instead, I will work for myself full time. My primary focus is improving and enhancing Antecons, a recommendation engine for webshops that is available now.

I also want to work on smaller side-projects. For example, I have an ongoing dream of creating programmatic music. Hopefully, blog posts will be more frequent in this period as well. I have a few different topics lined up but my thoughts about them have not yet materialized.

Show me the money

I am funding this time off with money that I saved up over the last year. Unless something unexpected happens, I can afford at least two months without any form of income but I hope to stretch the money to support me for half a year. I have very few obligations and expenses so it should be possible.

Some of my friends suggested that if I do not have a job then I could get unemployment benefits. My answer to that is a clear no. It would be morally wrong to do because I am not unemployed and I do not want to take advantage of the system.

“I have this cool idea”

Great! I am open to good ideas and I would love to hear what you have in mind.

Also, I have always wanted to work on open source projects or for a good cause so if you need help with a non-profit or a charity, I might be able to help you for free.

The road ahead

Honestly, I do not really know what I am doing. I am trying to think rationally and tell myself that I should have a clear plan or be more organized but planning is completely against my nature so for now, I am just going to wing it. We will see what happens down the line.


Experimental features

This post is about Antecons, a product recommendation engine, now part of Conversio. Antecons is no longer commercially available, but I have kept my developer diary on my website with permission.

Yesterday, I found out exactly what it means when Google warns about their experimental App Engine features: Your code might eventually break. Let me be clear, I am not blaming Google. They give you fair warning:

Mapreduce is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Mapreduce.

I have written about my usage of the MapReduce framework earlier. Yesterday, I updated the MapReduce framework to the latest version only to see that my custom Datastore reader suddenly had stopped working and I was seeing exceptions in my MapReduce pipeline. Bummer.

Long story short, I spent a day debugging the new code and finally got it working by:

  1. Digging through the MapReduce framework code. Hurray for open source!
  2. Dropping the idea of running FP-Growth on batches of entities and instead running the mapping function on each entity.

That second point probably requires some explanation to really grasp and I am not sure I will be able to but maybe some pseudo-Python will help. The biggest change happened in the map-step of the Frequent Patterns MapReduce pipeline. Basically I went from this:

def map_batch_of_transactions(batch):
    frequent_patterns =
    for p in frequent_patterns:
        yield p,

to this:

def map_single_transaction(transaction):
    frequent_patterns = itertools.combinations(transaction, 2)
    for p in frequent_patterns:
        yield p, 1

The MapReduce shuffler takes care of grouping together patterns with the same key so with the new method, the shuffler will have more work to do since the same patterns will be yielded more often. Let’s say we have the pattern:

a,b (support: 4)

Before, the shuffler would just receive:

('a,b', 4)

but now it will receive:

('a,b', 1)
('a,b', 1)
('a,b', 1)
('a,b', 1)

On the other hand, FP-growth does not have to run so the map-step of the pipeline has more predictable performance characteristics. It remains to be seen if the change has significant impact on the entire MapReduce process. I am currently testing this.

So anyway, the whole point of this post was: If a feature is experimental, watch out. Sounds obvious right? Well…

Industry Technology

Why you should not use Microsoft Silverlight for your next web application

You are running a Silverlight 4 application. You may experience incompatibilities as Moonlight does not have full support for this runtime yet.
The above warning message sums up everything I dislike about proprietary web technologies. Silverlight is a fairly new Microsoft technology from 2007 and Moonlight is its Open Source and not up-to-date equivalent that I have to use because Microsoft does not provide Silverlight for Linux. Since most software only run on Windows anyway why is this so upsetting? Well, everything is different on the web.

First, the premise for Rich Internet Applications (RIA) like Silverlight, Flash and even Java is ok: A website usually does not provide much “action” in itself so having an extra plugin running on the website with some access to the underlying system resources as well as built-in extra functionality will provide a better user experience. Microsoft writes on the Silverlight website:

Silverlight is a powerful development platform for creating engaging, interactive user experiences for Web, desktop, and mobile applications when online or offline

That sounds good and I do not necessarily disagree with the premise for RIAs but the question is: Why did we need another RIA platform that doesn’t really work? I don’t exactly know. However, it is understandable from Microsoft’s viewpoint, given their history of locking-in people to their platforms and the fact that they have legions of .NET developers that are probably very comfortable staying in their own environment when writing web applications. For them, Silverlight is probably bliss.

But there are some problems, one of them being its availability. As of this writing, Silverlight is only supported in roughly 61% of all browsers, according to statowl, or roughly 69%, according to riastats. As the above warning message suggests, it does not work perfectly on (my version of) Linux, even with the newest Chrome browser and Ubuntu 11.04, the arguably best supported Linux version out there. Indeed, most of the Silverlight applications that I have looked at did not work very well, including an app that I was offered to work on1 and popular services such as Netflix which currently does not work with Moonlight.

Again, the question arises: Why is this so bad? Isn’t this the same as always? No, it is a problem because the beauty of the web is its openness, and this openness is what Microsoft is challenging. The same applies to Flash. You may call Apple snobbish and manipulative if you like but Steve Jobs does have a point regarding Flash as a platform:

Most Flash websites will need to be rewritten to support touch-based devices. If developers need to rewrite their Flash websites, why not use modern technologies like HTML5, CSS and JavaScript? … New open standards created in the mobile era, such as HTML5, will win on mobile devices (and PCs too). Perhaps Adobe should focus more on creating great HTML5 tools for the future …

In my opinion, this would apply to Silverlight also. To be fair, Microsoft is embracing standards like html5 in Internet Explorer 9 but at the same time, they continue to push Silverlight forward. Some people think that this is not a problem since html5 and Silverlight are not direct competitors. But because of Microsoft’s power, Silverlight applications are shooting up everywhere which cuts off some users from using certain services of the Internet, something we have not seen on the same level with Flash. And this is a troubling development.

There is only one way of avoiding that Silverlight will dominate the next decade of web applications as Flash did in the past: Stop developing applications for it. My choice should now be clear. But I fear that I am almost alone.