{"id":599,"date":"2014-04-05T17:01:11","date_gmt":"2014-04-06T01:01:11","guid":{"rendered":"https:\/\/formidableengineeringconsultants.com\/?p=599"},"modified":"2014-10-13T13:20:16","modified_gmt":"2014-10-13T21:20:16","slug":"big-data-little-innovation","status":"publish","type":"post","link":"https:\/\/formidableengineeringconsultants.com\/?p=599","title":{"rendered":"Big Data – Little Innovation"},"content":{"rendered":"

\"BigDataBigBuildings\"<\/a>There is a huge focus on big data nowadays. Driven by ever decreasing prices and ever increasing capacity of data storage solutions,\u00a0big data provides magical insights and new windows into the exploitation of the long tail and addressing\u00a0micro markets and their needs.\u00a0 Big data can be used to build, test and validate models and ideas Big data holds promise akin to a panacea.\u00a0 It is being pushed as a universal solution to all ills.\u00a0 But if you look carefully and analyze correctly what big data ultimately provides is what Marshall MacLuhan<\/a> described as an accurate prediction of the present<\/a>.\u00a0 Big data helps\u00a0us understand how we got to where we are today. It\u00a0tells us what people want or need or do within a framework as it exists today.\u00a0 It is bounded by today’s (and the past’s) possibilities and ideas.<\/p>\n

But big data does not\u00a0identify the next\u00a0seismic innovation.\u00a0 It does not necessarily even identify how to modify the current big thing to make it incrementally better<\/p>\n

In the October 2013 issue of IEEE Spectrum, an article<\/a>\u00a0described the work of a company named Lex Machina<\/a>. The company is a classic big data play.\u00a0 They collect, scan and analyze all legal proceedings associated with patent litigation and draw up statistics\u00a0identifying, for instance,\u00a0the companies who are more likely to settle, law firms that are more likely to win, judges who are more favorable to defendants or the prosecution, duration and cost assessments of prosecutions in different areas.\u00a0 So it is a useful tool.\u00a0 But all it does is tell you about the state of things now.\u00a0 It does not measure variables like outcomes of prosecution or settlements (for instance, if a company wins but goes out of business or wins and goes on to build a more dominant market share or wins and nothing happens).\u00a0 It does not indicate if companies protect only specific patents that have, say, an estimated future value of, say,\u00a0$X million or what metric companies might use in their internal decision making process because that is likely not visible in the data.<\/p>\n

Marissa Meyer<\/a>, the hyper-analyzed and hyper-reported-on CEO of Yahoo!, famously tests all decisions based on data<\/a>.\u00a0 Whether it is the shade of purple for the new Yahoo! logo<\/a>, the purchase price of the next acquisition or value of any specific employee –\u00a0it’s all about measurables.<\/p>\n

But how can you measure the immeasurable? \u00a0If something truly revolutionary is developed, how can big data help you decide if it’s worth it? How even can little data help you? \u00a0How can people know what they like until they have it? If I told you that I would provide you with a service that lets you broadcast your thoughts to anyone who cares to subscribe to them, you’d probably say. \u00a0“Sounds stupid. Why would I do that and who would care what I think?” \u00a0If I then told you that I forgot one important aspect of the idea, that every shared thought is limited to 140 characters, you would have likely said, “Well, now I KNOW<\/strong><\/em> it’s stupid!”. \u00a0Alas, I just described Twitter<\/a>. \u00a0An idea that turned into a company that is, as of this writing, trading on the NYSE for just over $42 per share<\/a>\u00a0with a market capitalization of about $25 billion.<\/p>\n

Will a strong reliance on big data lead us incrementally into a big corner? \u00a0Will all this fishing about in massive data sets for patterns and correlations merely reveal the complete works of Shakespeare in big enough data sets<\/a>? Is Big Data just another variant of the Infinite Monkey Theorem? Will we get the to point that with so much data to analyze we merely prove whatever it is we are looking for?<\/p>\n

Already we are seeing that\u00a0Google Flu Trends<\/a> is looking for instances of the flu and finds them where they aren’t or in higher frequencies than they actually are. \u00a0In that manner, big data fails even to accurately predict the present.<\/p>\n

It is only now that some of the issues with ‘big data’ are being considered. \u00a0For instance, even when you have a lot of data – if it is bad or incomplete, you still have garbage only just a lot more of it (that is where wearable devices, cell phones and other sophisticated but merely thinly veiled data accumulation appliances come into play – to help improve the data quality by making it more complete). \u00a0Then the data itself is only as good as the analysis you can execute on it. \u00a0The failings of Google Flu Trends are often attributed to bad search terms in the analysis but of course, there could be many other different reasons.<\/p>\n

Maybe, in the end, big data is just big hubris. \u00a0It lulls us into a false sense of security, promising knowledge and wisdom based on getting enough data but in the end all we learn is where we are right now and its predictive powers are, at best, based merely on what we want the future to be and, at worst, are non-existent.<\/p>\n","protected":false},"excerpt":{"rendered":"

There is a huge focus on big data nowadays. Driven by ever decreasing prices and ever increasing capacity of data storage solutions,\u00a0big data provides magical insights and new windows into the exploitation of the long tail and addressing\u00a0micro markets and their needs.\u00a0 Big data can be used to build, test and validate models and ideas […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[86,47,48,73,68,15,45,50],"tags":[74,46,6,24,38,89,52],"class_list":["post-599","post","type-post","status-publish","format-standard","hentry","category-big-data-2","category-data","category-databases","category-disruptive-technology","category-innovation","category-software","category-start-ups","category-storage","tag-big-data","tag-ideas","tag-new","tag-reinvent","tag-revolutionary","tag-software","tag-space"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/posts\/599","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=599"}],"version-history":[{"count":18,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/posts\/599\/revisions"}],"predecessor-version":[{"id":772,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=\/wp\/v2\/posts\/599\/revisions\/772"}],"wp:attachment":[{"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=599"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=599"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formidableengineeringconsultants.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=599"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}