{"id":2805,"date":"2020-11-01T21:43:53","date_gmt":"2020-11-01T20:43:53","guid":{"rendered":"https:\/\/davidlebech.com\/thoughtflow\/?p=2805"},"modified":"2022-03-16T20:59:41","modified_gmt":"2022-03-16T19:59:41","slug":"image-captioning-model","status":"publish","type":"post","link":"https:\/\/davidlebech.com\/thoughtflow\/image-captioning-model\/","title":{"rendered":"Image captioning model"},"content":{"rendered":"\n

Progress is slow for my various hobby efforts to generate images<\/a>, create lyrics<\/a> or even detect my dog in an image<\/a>, but I do still experiment a little bit when I have the time for it.<\/p>\n\n\n\n

Today, I ran through the image captioning tutorial<\/a> from Tensorflow, because why not. I trained it on a reduced set of 5000 images compared to the tutorial, but otherwise the code is identical.<\/p>\n\n\n\n

The purpose of the model is to look at an image and predict a caption for the image, e.g. “man sits next to computer” if the model was looking at me right now.<\/p>\n\n\n\n

The results are quite hilarious at this point, and it has given me some ideas for future development that could potentially delight or confuse.<\/p>\n\n\n\n

Anyway, for now, here is a caption that is as close to being accurate as I could get today. It’s a picture of Mila, my dog, with the predicted caption “a little dog sitting on dirt field”. Not too bad.<\/p>\n\n\n\n

\"Image
Best predicted caption: a little dog sitting on dirt field<\/figcaption><\/figure>\n\n\n\n

The model has a bit of randomness in its output, so the same image might produce multiple different results. For example, the image above also produced “a dog is surfing on a field with it’s toppings in front of grass” as well as the completely nonsensical “a fire up of a green grass next to a grass in a grassy field”.<\/p>\n\n\n\n

I also tried a few other images, but I am a too lazy to look up the proper attribution for including them in this post, so here is another image of Mila with the strange caption “a dog that is standing in tall grass in the grassy field with a large dog and white dog is standing in a grassy field next to a tree”<\/p>\n\n\n\n

\"\"
Best predicted caption: a dog that is standing in tall grass in the grassy field with a large dog and white dog is standing in a grassy field next to a tree<\/figcaption><\/figure>\n\n\n\n

There is also the slightly more boring “a dog standing in front of a grass” and my personal favorite (although it’s a bit disturbing) “a dressed dog in a field eaten horse to enjoy zebras grass in the grass”.<\/p>\n\n\n\n

That is it for now. Not much content here, but I felt like writing a post. Stay tuned for possibly more silliness when the spark of energy hits.<\/p>\n","protected":false},"excerpt":{"rendered":"

Progress is slow for my various hobby efforts to generate images, create lyrics or even detect my dog in an image, but I do still experiment a little bit when I have the time for it. Today, I ran through the image captioning tutorial from Tensorflow, because why not. I trained it on a reduced […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37],"tags":[200,191,203,238],"class_list":["post-2805","post","type-post","status-publish","format-standard","hentry","category-projects","tag-classification","tag-deep-learning","tag-tensorflow","tag-text-generation"],"_links":{"self":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts\/2805","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/comments?post=2805"}],"version-history":[{"count":0,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts\/2805\/revisions"}],"wp:attachment":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/media?parent=2805"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/categories?post=2805"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/tags?post=2805"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}