This post is mostly a status of what I have been up to here at the nearly-almost-half-year mark of 2016.
Introducing Product Search
By the end of 2015, we had already been using Elasticsearch for a while. It was the first part of a long-term strategy of moving data away from Google App Engine. Event data such as page views and clicks as well as order aggregations such as revenue-per-day for our users was being stored and calculated on Elasticsearch. Although Elasticsearch is popular for collecting log data, its main selling point is that it is a very fast full-text search engine.
During the Christmas holidays, I wanted to see how easy it would be to add a search widget, powered by Elasticsearch. After about 3-4 hours, I posted this proof-of-concept video to our Slack channel with the following message:
As it turned out, the product search feature quickly found its way onto the roadmap :-)
Unlisted Antecons
It was going to happen at some point, and in early April, we finally removed the listing of Antecons from the Shopify app store. The app continues to run and interestingly, we have some users that are still using it, even though we have contacted everyone and tried to get them to switch over to Receiptful. Loyal customers.
Popular metrics report
By the end of April, we released the report “8% of all product page traffic converts to sales”. For a short while, I think it made a little splash and was read by quite a few people. Although I did not write the article, all the data for the article was gathered by me a few months before. One of those little side tasks that spice up developer life — although doing data analysis is slightly more exciting than data gathering :-)
Go nuts with Golang
Currently, I am in Golang land. I did not think I would end up there, but when tasked with creating a new web app for some simple store metrics, I decided to create it with Go after consulting with the team. After some initial headaches (i.e. getting used to a statically typed, compiled language again), I must say that Go has some good things going for it. My colleagues mock me about using tabs, but that is the Go way.
In the same project, I also said hello to my old friend MapReduce. It is a feature of MongoDB and we use it to create pre-aggregated reports for the project. It might be a short affair though, as I am also considering other options such as Google BigQuery. We will see…
So those are the major headlines (I probably missed something). I have been meaning to write slightly more technical articles, but I do not feel like I am not in the right mindset to do so yet. Those pieces also tend to be much longer and much more difficult to write, so for now, you will have to do with these random rambles.
Why move away from app engine?
Hi Mark, I still really like App Engine, and it was great for getting started with Antecons. The limitations that we/I specifically ran into was that datastore queries started to become quite “slow” and unpredictable. For example, making a recommendation query would at times average more than two seconds and then suddenly drop down to a more ok 300-500 ms. We also started having quite a lot of writes to the datastore for product updates, and there is a rather harsh limit on concurrent writes.
The other problem we had was that the official MapReduce library could not keep up with the amount data we had which was disappointing because it was in the order of a few gigabytes, but it ran ridiculously slow at times, even for very small inputs that were only a few megabytes. For example, I once asked this question on Stackoverflow and never got any of it resolved: http://stackoverflow.com/questions/30643194/app-engine-mapreduce-performance-optimization
A third problem might also be the price, but that’s not such a big deal for us. However, it is worth noticing that while the pricing is extremely attractive initially, the costs do go up significantly when the app inevitably has to start running big instances and do a lot of datastore operations.
I guess to sum it up, I think the variance in performance is what has made us look elsewhere. I know that there are a lot of companies that do great thing on App Engine (e.g. Khan Academy), and they have made it work out for them and so could we possibly. I should note here that getting into App Engine development is a bit daunting and since we’re primarily running node.js apps, there is also an internal wish for moving away from the platform.
Cheers
David