Antecons – Thought Flow

Golang and the search of the past

Gopher inspecting Go code by Anthony Starks (CC-BY-NC)

This post is mostly a status of what I have been up to here at the nearly-almost-half-year mark of 2016.

Introducing Product Search

By the end of 2015, we had already been using Elasticsearch for a while. It was the first part of a long-term strategy of moving data away from Google App Engine. Event data such as page views and clicks as well as order aggregations such as revenue-per-day for our users was being stored and calculated on Elasticsearch. Although Elasticsearch is popular for collecting log data, its main selling point is that it is a very fast full-text search engine.

During the Christmas holidays, I wanted to see how easy it would be to add a search widget, powered by Elasticsearch. After about 3-4 hours, I posted this proof-of-concept video to our Slack channel with the following message:

As it turned out, the product search feature quickly found its way onto the roadmap :-)

Unlisted Antecons

It was going to happen at some point, and in early April, we finally removed the listing of Antecons from the Shopify app store. The app continues to run and interestingly, we have some users that are still using it, even though we have contacted everyone and tried to get them to switch over to Receiptful. Loyal customers.

Popular metrics report

By the end of April, we released the report “8% of all product page traffic converts to sales”. For a short while, I think it made a little splash and was read by quite a few people. Although I did not write the article, all the data for the article was gathered by me a few months before. One of those little side tasks that spice up developer life — although doing data analysis is slightly more exciting than data gathering :-)

Go nuts with Golang

Currently, I am in Golang land. I did not think I would end up there, but when tasked with creating a new web app for some simple store metrics, I decided to create it with Go after consulting with the team. After some initial headaches (i.e. getting used to a statically typed, compiled language again), I must say that Go has some good things going for it. My colleagues mock me about using tabs, but that is the Go way.

In the same project, I also said hello to my old friend MapReduce. It is a feature of MongoDB and we use it to create pre-aggregated reports for the project. It might be a short affair though, as I am also considering other options such as Google BigQuery. We will see…

So those are the major headlines (I probably missed something). I have been meaning to write slightly more technical articles, but I do not feel like I am not in the right mindset to do so yet. Those pieces also tend to be much longer and much more difficult to write, so for now, you will have to do with these random rambles.

Competition

Note: The post below was written back in September 2014, when I was starting to feel a bit down about how poorly the Shopify sales of my app were going — especially after being featured on the front page of the app store without much attention. I did not publish it back then, because I once promised myself that I would try to mostly stay away from online rants. However, I think it provides some context to my March 2015 post about what happened next during my period of complete independence. So here you go.

I often get emotionally involved in my software. For example, I feel physically uncomfortable when I find a bug in some code I have written. I should probably write better tests to insure my well-being. Anyway, I want to talk about a bit about competition.

Antecons, the recommendation engine that I am working on, has now been around for about a year on the Shopify app store. When Antecons was first released, there were three other competing recommendation engines on the app store and one of them had been added just a few weeks before Antecons. Since then, three or four more have popped up on the app store. Recommendation engines and data analysis must be hot business because all of these competing apps seem to be doing very well and are getting nice reviews from the customers. This is a bit of a let down for me because it does not seem like Antecons is enjoying the same success.

What I am starting to realize is that it probably does not matter that Antecons always use SSL for increased privacy and security (unlike most of the other apps), that the Antecons JavaScript code is minified to reduce bandwidth for the webshop visitors (unlike at least two of the other apps) or that I do not write fake reviews (like at least one other app). No one will pat me on the shoulder for doing what I feel is a tiny bit extra “niceness” of the overall package. Oh well.

What happened next

In July last year (2014), I started working full-time on Antecons. Back then, I wrote a post about complete independence where I predicted that I could support myself for up to half a year without income. This turned out to be true, but now that we have entered a new year, it is time for an update. This is an update that I was nervous about until very recently where a story about monetary failure took an interesting turn for the better.

The challenge

A lot of people try to hide their failures, myself included. Being afraid of failure, taking risks and facing challenges, it seems like a very weird and poor life choice to give up a lucrative and comfortable position as a consultant to work without income and earning potential for an extended period of time. The explanation is simple though: Things were starting to get fun.

When Antecons launched on Shopify in 2013, it was mostly developed while doing freelance consulting work. Having a high-revenue commercial product was never really the goal. I simply wanted to see if it was possible for me to start something from scratch and make it through all the way to a finished product and not just an abandoned side-project.

When the first few customers started coming in, I realized that this product might have some potential, and it was quite a different feeling having customers that bought a service rather than buying my labor. It was… fun. So I started working on Antecons full-time in July with the vision of slowly building up a list of stable clients. I had a good relationship and regular contact with a webshop house, and they sounded very interested in offering Antecons to their webshop customers so I pushed towards a first “revenue milestone” with them as a re-seller. It was a great Summer and I built an API beta that I was satisfied with offering to potential buyers.

Fail

My plan failed. It turned out that nothing came out of my contact with the webshop re-seller and in the meantime, Shopify sales only increased very slowly. Antecons was even featured on the front page of the Shopify app store, but there were disappointingly few signups.

There is no need to go into great detail about the technicalities, because the main problem was that I neglected sales too much so I failed to sell. That makes sense now, but I did not realize it soon enough… so I failed. Period. The end.

Late Fall came, I was not making any money and the product was not going to generate significant revenue anytime soon. I had to take on a few hours of consulting work again to pay the bills which meant that I had less time to work on Antecons and in the beginning of December, I was hired full-time as a Python developer for Neodev.

I had mixed feelings about starting a new job, because it felt like a defeat to stop working on Antecons. Both the job and my colleagues were great though, and it was sometimes quite difficult to explain to friends and family how having a rewarding and well-compensated job could still feel a bit like a step back or a let-down. Nevertheless, it was difficult to leave Antecons behind. As it turned out though, the departure was short-lived.

An unexpected journey

It was exactly one day after signing my new contract that I was contacted by Adii, a successful entrepreneur with experience in the e-commerce field. Adii was looking for a recommendation engine to improve a product upsell feature in a young startup called Receiptful. Initially, I was skeptical and without hope, because I had basically given up on monetizing Antecons since it was not making any real money. But after a few weeks of communication, we decided to work together and I was “acquihired” (yes, that is a real word) and Antecons was revived before it had even drawn its final breath.

Fast forward to today and I have been with Receiptful for a month, working on integrating Antecons as part of the Receiptful system. It is really great to be able to work on Antecons and data analysis in a full-time position and it has also presented some new challenges, but that is a topic for a different post.

Working independently was a great experience. It was an unexpected journey with an unexpected ending. A new and different chapter has now begun: Life in a startup.

Complete independence

I am now jobless. I am now independent. For the next few months, I am going to work on projects that do not provide any steady income and I have not planned to take on any consulting or freelance work. I am writing this post both for people that would be interested in what I am doing but also to wrap my ahead around what this is all about.

What is the plan

I am taking some time off from paid jobs. Instead, I will work for myself full time. My primary focus is improving and enhancing Antecons, a recommendation engine for webshops that is available now.

I also want to work on smaller side-projects. For example, I have an ongoing dream of creating programmatic music. Hopefully, blog posts will be more frequent in this period as well. I have a few different topics lined up but my thoughts about them have not yet materialized.

Show me the money

I am funding this time off with money that I saved up over the last year. Unless something unexpected happens, I can afford at least two months without any form of income but I hope to stretch the money to support me for half a year. I have very few obligations and expenses so it should be possible.

Some of my friends suggested that if I do not have a job then I could get unemployment benefits. My answer to that is a clear no. It would be morally wrong to do because I am not unemployed and I do not want to take advantage of the system.

“I have this cool idea”

Great! I am open to good ideas and I would love to hear what you have in mind.

Also, I have always wanted to work on open source projects or for a good cause so if you need help with a non-profit or a charity, I might be able to help you for free.

The road ahead

Honestly, I do not really know what I am doing. I am trying to think rationally and tell myself that I should have a clear plan or be more organized but planning is completely against my nature so for now, I am just going to wing it. We will see what happens down the line.

Experimental features

This post is about Antecons, a product recommendation engine, now part of Conversio. Antecons is no longer commercially available, but I have kept my developer diary on my website with permission.

Yesterday, I found out exactly what it means when Google warns about their experimental App Engine features: Your code might eventually break. Let me be clear, I am not blaming Google. They give you fair warning:

Mapreduce is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Mapreduce.

I have written about my usage of the MapReduce framework earlier. Yesterday, I updated the MapReduce framework to the latest version only to see that my custom Datastore reader suddenly had stopped working and I was seeing exceptions in my MapReduce pipeline. Bummer.

Long story short, I spent a day debugging the new code and finally got it working by:

Digging through the MapReduce framework code. Hurray for open source!
Dropping the idea of running FP-Growth on batches of entities and instead running the mapping function on each entity.

That second point probably requires some explanation to really grasp and I am not sure I will be able to but maybe some pseudo-Python will help. The biggest change happened in the map-step of the Frequent Patterns MapReduce pipeline. Basically I went from this:

def map_batch_of_transactions(batch):
    frequent_patterns = fpgrowth.run(batch)
    for p in frequent_patterns:
        yield p, p.support

to this:

def map_single_transaction(transaction):
    frequent_patterns = itertools.combinations(transaction, 2)
    for p in frequent_patterns:
        yield p, 1

The MapReduce shuffler takes care of grouping together patterns with the same key so with the new method, the shuffler will have more work to do since the same patterns will be yielded more often. Let’s say we have the pattern:

a,b (support: 4)

Before, the shuffler would just receive:

('a,b', 4)

but now it will receive:

('a,b', 1)
('a,b', 1)
('a,b', 1)
('a,b', 1)

On the other hand, FP-growth does not have to run so the map-step of the pipeline has more predictable performance characteristics. It remains to be seen if the change has significant impact on the entire MapReduce process. I am currently testing this.

So anyway, the whole point of this post was: If a feature is experimental, watch out. Sounds obvious right? Well…