Category: Code

Reinforcement learning

I have been looking into a machine learning technique called reinforcement learning (RL) lately. This was on my TODO for a while, and I must say, this field is incredibly exciting! I played around with some OpenAI Gym environments and re-implemented two RL algorithms mostly based on code I found from other authors.

After spending many hours on this, I can still only get my algorithm to solve the Cartpole problem, where the goal is to balance a pole on a moving cart (video below). I haven’t cracked the nut on a continuous action problem like Pendulum, where the goal is to swing the pendulum into an upright position and keep it there (video below).

Anyway, here is my implementation of the RL algorithms. Perhaps it will be useful for someone :-)

Apache Beam MongoDB reader for Python

The Apache Beam SDK for Python is currently lacking some of the transforms found in the Java SDK. I created a very minimal example of an Apache Beam MongoDB read transform for Python that might be useful for someone else looking for an answer.

I will update this post in the future if/when the Apache maintainers include support for MongoDB in the SDK. I know I could contribute to the project directly, but I don’t have time for it right now unfortunately :-)

Golly Gosh Moments

You know that time you realized that you have been doing something the wrong way for a very long time and then finally realize the wrongness. For the sake of the low profanity rating of this blog, let’s call these golly gosh moments, although the Millennials might better understand #FML. Homer just says do’h.

So it’s just a normal day at the office, and I want to see if I can make an IP address lookup to get the approximate geo-location of a website visitor. I find an IP and it starts with 10 and turns out to be part of a private IP range. The next IP is the same. And the next.

To make a long story short, it turned out that we have been saving Heroku IP addresses in our logs instead of the user IP address for all our widget tracking for all of time. Heroku is a proxy, so the actual IP address is in an X-Forwarded-For header. For educational purposes here is how to make an express.js app behave better with a trusted proxy:

// App is an express.js app
app.enable('trust proxy');

// req.ip now contains the correct
// IP address during requests.

A one-liner made a world of difference for the logging. Golly Gosh.