{"id":2351,"date":"2019-11-30T09:34:49","date_gmt":"2019-11-30T08:34:49","guid":{"rendered":"https:\/\/davidlebech.com\/thoughtflow\/?p=2351"},"modified":"2019-11-30T09:34:49","modified_gmt":"2019-11-30T08:34:49","slug":"80-20-and-data-science","status":"publish","type":"post","link":"https:\/\/davidlebech.com\/thoughtflow\/80-20-and-data-science\/","title":{"rendered":"80\/20 and data science"},"content":{"rendered":"\n

“20% of the task took 80% of the time”. If you have ever heard someone say that, you probably know about the so-called 80\/20 rule, also known as the Pareto principle<\/a>.<\/p>\n\n\n\n

The principle is based on the observation that “for many events, roughly 80% of the effects come from 20% of the causes.”. As an example, the Wikipedia article<\/a> mentions how in some companies “80% of the income come from 20% of the customers” along with a bunch of other examples.<\/p>\n\n\n\n

In contrast to this, most of the references to the 80\/20 rule that I hear in the wild are variations of the (often sarcastic) statement at the beginning of the post, and it is also this version that is more fun to talk about.<\/p>\n\n\n\n

Estimating the finishing touch<\/h3>\n\n\n\n

In software development, the 80\/20 rule often shows up when the “finishing touches” to a task are forgotten or complexity is underestimated. An example could be if a developer forgot to factor in the time it takes to add integration tests to a new feature or underestimated the difficulty of optimizing a piece of code for high performance.<\/p>\n\n\n\n

In this context, the 80\/20 rule could thus be seen as the result of bad task management to a certain extent, but it is worth noting that it is not always this simple. Things<\/em> get in the way, like when the test suite refuses to run locally, or the optimized code cannot work without blocking the CPU, and the programming language is single-threaded, forcing the developer to take a different approach to the problem (this is purely<\/em> hypothetical of course…).<\/p>\n\n\n\n

Related to this, Erik Bernhardsson recently wrote an interesting treatise on the subject of why software projects “take longer than you think”, and I think it is worth sneaking in a reference. Here is the main claim from the author:<\/p>\n\n\n\n

I suspect devs are actually decent at estimating the median<\/em> time to complete a task. Planning is hard because they suck at the average<\/em>.<\/p>Erik Bernhardsson, Why software projects take longer than you think \u2013 a statistical model<\/a><\/cite><\/blockquote>\n\n\n\n

The message here resonated quite well with me (especially because of the use of graphs!). The author speaks of a “blowup factor” (actual time divided by estimated time) for projects, and if his claims are true, there could be some merit to the idea that the 20% of a task could easily “blow up” and take 80% of the time.<\/span>1<\/sup><\/a><\/span><\/p>\n\n\n\n

Dirty data<\/h3>\n\n\n\n

Sometimes, the perception<\/em> of data science is that most of the time “doing data science” is spent on creating models. For some data scientists, this might be the reality, but for the majority that I have spoken to, preparing data takes up a significant amount of time, and it is not the most glorious work, if one is not prepared for it.<\/p>\n\n\n\n

I recently gave an internal talk at work where I jokingly referred to this as the 80\/20 rule of data science<\/strong>: 80% of the time is spent on data cleaning (the “boring ” part), and 20% on modeling (the “fun” part).<\/p>\n\n\n\n

This is not really a 80\/20 rule, except if we rephrase it as “80% of the fun takes up only 20% of the time” or something like that.<\/span>2<\/sup><\/a><\/span><\/p>\n\n\n\n

When it comes to deploying models in production, the timescales sometimes shift even more. The total time spent on a project might be 1% on modeling and 99% on data cleaning and infrastructure setup, but it’s the 1% (the model) that gets all the attention.<\/p>\n\n\n\n

The future of data cleaning<\/h3>\n\n\n\n

In the last couple of years, there have been loads of startups and tools emerging that do automatic machine learning (or “AutoML”), i.e. they automate the fun<\/em> parts of data science, while sometimes also providing convenient infrastructure to explore data.<\/p>\n\n\n\n

If we assume that the 80\/20 rule of data science is correct, these tools are thus helping out with 20% of data science. However, the first company that solves the problem of automatically cleaning and curating dirty data is going to be a billion-dollar unicorn. Perhaps the reason that we have not seen this yet is that dealing with data is actually really difficult<\/em>.<\/p>\n\n\n\n

For now, I suspect that the “80\/20 rule of data science” will continue to be true in many settings, but that is not necessarily a bad thing. You just gotta love the data for the data itself :-)<\/p>\n","protected":false},"excerpt":{"rendered":"

“20% of the task took 80% of the time”. If you have ever heard someone say that, you probably know about the so-called 80\/20 rule, also known as the Pareto principle. The principle is based on the observation that “for many events, roughly 80% of the effects come from 20% of the causes.”. As an […]<\/p>\n","protected":false},"author":1,"featured_media":2467,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[145,146],"class_list":["post-2351","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-thoughts","tag-data-science","tag-software-engineering"],"_links":{"self":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts\/2351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/comments?post=2351"}],"version-history":[{"count":0,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/posts\/2351\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/media\/2467"}],"wp:attachment":[{"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/media?parent=2351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/categories?post=2351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davidlebech.com\/thoughtflow\/wp-json\/wp\/v2\/tags?post=2351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}