Thought Flow

Technology and other things

Author: David

  • Antecons developer diary part 1

    This post is about Antecons, a product recommendation engine, now part of Conversio. Antecons is no longer commercially available, but I have kept my developer diary on my website with permission.

    It has been a year since the first announcement of Antecons. What has happened since then? Absolutely nothing. The reason for this seems to be pretty straightforward: I have been occupied with full-time consulting and I have made a deliberate choice of working a five-day 40-hour work week. That leaves no room for extra work. However, this has changed slightly. Since October 1, I have had a four-day work week. This leaves one day to do other stuff and I have decided to continue my work on Antecons and I intend to blog about the (slow) progress.

    Consider this part 1 in a series of posts about my experiences, design decisions and the current state of Antecons. First, I would like to fill in the gap in the history of Antecons from last year until now. There is more to the story than just a lack of time.

    Where we left off

    One year ago, I made a promise, a prototype and a teaser. The prototype is a recommendation engine that is still running on one webshop (as far as I know). It is written in C# for .NET, uses Microsoft SQL server as database and is deployed directly on the end-user’s server using a pre-compiled DLL that can be referenced from any .NET application. As I was developing the prototype, I already had a couple of basic objections to my own design:

    1. It required the user to use .NET and SQL Server.
    2. It required me to use Visual Studio, IIS and Windows as a development environment.
    3. It required the user to integrate the component into their webshop. This included:
      1. Setting up an SQL database and tables conforming with the Antecons data format.
      2. Synchronizing webshop purchases with the Antecons database and keeping it updated.
      3. Manually running the recommendation algorithm to pre-create recommendations.
      4. Integrating Antecons into the webshop design in the appropriate places to show the pre-created recommendations for the current product being shown.

    The first two points were easily fixed by deciding to replace the pre-compiled DLL component with a webservice. Using an API for Antecons, there would be no constraints on the user regarding their server software or webshop programming environment.

    And thus was born the teaser, a web application written in Python and running on Google App Engine. It never turned into a webservice with an API. Instead it was a website that showed what the essence of Antecons was: A recommendation engine based on association rule mining (or market basket analysis). As it turned out, this was the easy part and implementing a simple API would have been too. But the third point I outlined above, that was the real killer.

    The promise

    A quote from my company website:

    Metacircle has been developing a data analysis tool for some time. It is called Antecons.

    And from the Antecons teaser website:

    The API is currently under development…

    These quotes sum up the promise I made myself (and the world): To work on Antecons. Did I succeed? No. Do I feel bad about that? Yes. What killed the project? Time and wishful thinking.

    Time grew short when I started working full-time but this is only part of the explanation. As hinted above, I was also held back by the third design objection. Setting up Antecons for actual usage was, in my opinion, too difficult and too much an obstacle for widespread usage. How could Antecons be made so it would be super easy to setup, super easy to deploy and super easy to run. The ideal scenario looked like this:

    1. User registers for Antecons and receives a unique User ID.
    2. User inserts 10 lines of Javascript code in the header of each webshop page where Antecons should show recommendations.
    3. Antecons takes care of the rest.

    This is how e.g. Google Analytics works and it is quite amazing how much information it gathers and how useful this information is, based on such a simple installation. If Antecons could be as easy to install as Google Analytics, which is installed on millions of websites, there would be a huge potential and many users could benefit from it.

    But the difficult questions piled up fast. For example, how would Antecons gather information on user’s sales data and how would it know where to show recommendations on a user’s webshop. The answers I came up with all lead to the same conclusion: The user would have to do some work and Antecons could not be fully automated, not even partially. This shattered my overall vision for Antecons and my head has been throwing around ideas ever since. Maybe the ideal scenario was wishful thinking but in any case, this was the end of the line. Until now.

    Where we are

    It has been two weeks since I started over with the Antecons project. In the next blog post, I will elaborate on what specific design obstacles that put the project on ice. This is an important pretext for what I have started working on right now. Later, I will write more about specific design decisions and actual implementation. The API should even have some partial functionality coming up very soon. Stay tuned.

  • Complexity

    Fractal simulation
    A fractal simulation from https://davidlebech.com/fractals/
    I learned about complexity studying computer science. Complexity was often measured in space and time, denoted by Ω, Θ and Ο. We learned about the fastest algorithms, the “hardest” problems and the methods to solve them or just prove that they are hard. That is what “complexity” meant to me.

    And then I started working with real world problems and complexity started meaning something else. It is no longer about NP-completeness, lower-bound running time or even algorithms. I have worked almost for a year on the same project 1 and throughout this project, the most difficult tasks have been figuring out the requirements of the end-user and transferring these requirements into an actual user interface. The complexity of a task is not measured in upper-bound running time and space requirements but in number of meetings and emails. And while the complexity of algorithms does not change after they have been implemented, the real world task changes its complexity whenever there is a change in requirements.

    When I step back and look at the output of the current project, I see nothing else there. All the complexity lies in communication and that is something different.

  • This is not a rant

    Meaw Meaw
    A kitten that is obviously ranting :-)
    Meaw Meaw by Sergiu Bacioiu.

    I’m still here, I just have not had anything interesting to say lately. I have had a few drafts for a few different rants but I usually decide not to post them. Rants can of course be interesting and enlightening but in my case, I just did not feel like posting them in the end.

    Most of what I read around the IT blogosphere are, in essence, rants. If a piece of text is not a rant, it is usually an article in a magazine or an advertisement for some new technology. Maybe my observation is wrong (it is most certainly very generalized) but if we hold on to the thought, and interesting question can be raised: Why is it like that? I do not have the answer, only further questions:

    • Is it simply easier to complain/critique stuff than to praise stuff?
    • Could it be that rants are more satisfying to write for the author?
    • Maybe the world is just generally dysfunctional and someone needs to remind us constantly that it is so?

    It would be hypocrisy for me to say that I will stop ranting altogether. But if there is one thing I have learned this past year of doing contract work it is to count to ten, take a deep breath and think once more about the situation that is bugging me — and then complain if it makes sense to do so. There are almost always two ways of looking at things.

  • Intellectual stimulus

    After a year of doing freelance programming/consulting, I have a better understanding of why some people choose academia or research as a career path: Intellectual stimulation. I have been working mostly with website and webshop backend programming and on a day-to-day basis, the recipe is basically the same:

    1. Create a datatable in a database.
    2. Create a webpage form so userdata can be put into this table.
    3. Display this userdata in various versions on the website/webshop.

    Over time, I see myself writing the same or very similar code over and over again for every new feature that needs to be added even when I try to reuse the same code — and it bores the hell out of me sometimes.

    Actually, while studying, I was not particularly interested in theory. I always liked the programming more than the theory and the latter is usually more emphasized. It is ofted noted that college is not preparing students with proper programming skills for “the real world” and that a dedicated software development education might be a good idea. I agree.

    But now I’m going of at a tangent. My point is, when I was in school, I was displeased with theory (because “hated” is such a strong word) and I fell asleep while reading any research paper or computer science book (this still haunts me. Reading CS books seems to be important for developers. I just cannot do it). But now, I miss the intellectual stimuli from academia. This is one of the reasons I have been pushing myself to find time to work on Antecons, a project that has forced me to dig into a number of research papers on association rule learning. (By the way, the recommendations made by Antecons are actually being added by the users to the shopping carts at vildmedvin.dk. I can thus assume that it is raising profits. This is a great success, given its currently limited and quite crude functionality)

    This blog post does not have a particular message or opinion. This is thought flow and I have just been wondering what other people do to cope with boredom in their jobs. Sometimes, boring work is inevitable, I know. But when more than half of one’s work is boring, is it then time to start rethinking one’s situation?

  • Fixing weird sound in Ubuntu Linux

    In Ubuntu, my sound will go weird at random times when listening to music or looking at Youtube and other videos. Voices turn into robot sounds and music is distorted. When this happens, I simply type the following in the terminal:

    pulseaudio -k
pulseaudio --start

    Notice there is only one dash (-) in the first command and two in the second. This fixes the issue for me every time. It’s probably a good idea to pause whatever sound you are playing before doing the above.

    Update: I should note, that it sometimes works just pausing the music, waiting a few seconds, and turning it back on.