I have noticed that as people age, they become finer and finer versions of themselves. Their eccentricities become sharper and more pronounced; their opinions and ideas more pointed and immutable; their thoughts more focussed. In short, I like to say that they become more perfect versions of themselves. We see it in our friends and acquaintances and in our parents and grandparents. It seems a part of natural human development.

Back in 2006, Netflix initiated the Netflix Prize with the intent of encouraging development of improvements in the accuracy of predictions about how much someone is going to enjoy a movie based on their movie preferences and rewarding the winner with $1,000,000. Contestants were given access to a set of Netflix’s end-users’ movie ratings and were challenged to provide recommendations of other movies to watch that bested Netflix’s own recommendation engine. BellKor’s Pragmatic Chaos team was announced as the winner in 2009 having manage to improve Netflix’s recommendations by 10% and walked off with the prize money.

What did they do? Basically, they algorithmically determined and identified movies that were exceptionally similar to the ones that were already liked by a specific user and offered those movies as recommended viewing. And they did it really well.

In essence what the Bellkor team did was build a better echo chamber. Every viewer is analyzed, their taste detailed and then the algorithm perfects that taste and hones it to a razor sharp edge. You become, say, an expert in light romantic comedies with a strong female lead, who lives in a spacious apartment in Manhattan, includes many dog owners, no visible children and often features panoramic views of Central Park.

Of course, therein lies the rub. A multifaceted rub at that. As recommendation engines become more accurate and discerning of individual tastes they remove any element of chance, randomness or error that might serve to introduce new experiences, genres or even products into you life. You become a more perfect version of you. But in that perfection you are also stunted. You are shielded from experimentation and breadth of experience. You pick a single pond and overfish it.

There are many reasons why this is bad and we see it reflected, most obviously, in our political discourse where our interactions with opposing viewpoints are limited to exchanges of taunts (as opposed to conversations) followed by a quick retreat to the comfort of our well-constructed echo chambers of choice where our already perfected views are nurtured and reinforced.

But it also has other ramifications. If we come to know what people like to such a degree then innovation outside safe and well-known boundaries might be discouraged. If Netflix knows that 90% of its subscribers like action/adventure films with a male hero and lots of explosions why would they bother investing in a story about a broken family being held together by a sullen beekeeper. If retail recommendations hew toward what you are most likely to buy – how can markets of unrelated products be expanded? How can individual tastes be extended and deepened?

Extending that – why would anyone risk investment in or development of something new and radically different if the recommendation engine models cannot justify it. How can the leap be made from Zero to One – as Peter Theil described – in a society, market or investment environment in which the recommendation data is not present and does not justify it?

There are a number of possible answers. One might be that “gut instincts” need to continue to play a role in innovation and development and investment and that risk aversion has no place in making the giant leaps that technology builds upon and needs in order to thrive.

A more geeky answer is that big data isn’t yet big enough and that recommendation engines aren’t yet smart enough. A good recommendation engine will not just reinforce your prejudicial tastes, it will also often challenge and extend them and that we don’t yet have the modelling right to do that effectively.  The data are there but we don’t yet know how to mine it correctly to broaden rather than narrow our horizons. This broadening – when properly implemented – will widen markets and opportunities and increase revenue.

I’ve had occasion to be interviewed for positions at a variety of technology companies. Sometimes the position actually exists, other times it might exist and even other times, the folks are just fishing for solutions to their problems and hope to save a little from their consulting budget. In all cases, the goal of the interview is primarily to find out what you know and how well you know it in a 30 to 45 minute conversation. It is interesting to see how some go about doing it. My experience has been that an interview really tells you nothing but does give you a sense of whether the person is nice enough to “work well with others“.

But now, finally folks at Google used big data to figure out something that has been patently obvious to anyone who has either interviewed for a job or was interviewing someone for a job. The article published in the New York Time details a talk with Mr. Laszlo Bock, senior vice president of people operations at Google.  In it, he shared that puzzle questions don’t tell you anything about anyone.  I maintain that they tell you if someone has heard that particular puzzle question before. In the published interview Mr. Bock, less charitably, suggests that it merely serves to puff up the ego of the interviewer.

I think it’s only a matter of time before big data is used again to figure out another obvious fact – that even asking simple or complex programming questions serves as no indicator of on-the-job success.  Especially now in the age of Google and open-source software.  Let’s say you want to write some code to sort a string of arbitrary letters and determine the computational complexity, a few quick Google searches and presto – you have the solution.  You need to understand the question and the nature of the problem but the solution itself has merely become a matter of copying from your betters and equals who shared their ideas on the Internet.  Of course, such questions are always made more useless when the caveat is added – “without using the built-in sort function” – which is, of course, the way you actually solve it in real life.

Another issue I see is the concern about experience with a specific programming language. I recall that the good people at Apple are particularly fond of Objective C to the point where they believe that unless you have had years of direct experience with it, you could never use it to program effectively.  Of course, this position is insulting to both any competent programmer and the Objective C language. The variations between these algorithmic control flow languages are sometimes subtle, usually stylistic but always easily understood. This is true of any programming language.  In reality, if you are competent at any one, you should easily be able to master any another. For instance,  Python uses indentation but C uses curly braces to delineate code blocks.  Certainly there are other differences but give any competent developer a few days and they can figure it out leveraging their existing knowledge.

But that still leaves the hard question.  How do you determine competency?  I don’t think you can figure it out in a 45 minute interview – or a 45 hour one for that matter – if the problems and work conditions are artificial.  I think the first interview should be primarily behavioral and focus on fit and then, if that looks good, the hiring entity should then pay you to come in and work for a week solving an actual problem working with the team that would be yours. This makes sense in today’s world of limited, at-will employment where everyone is really just a contractor waiting to be let go. So, in this approach, everyone gets to see how you fit in with the team, how productive you can be, how quickly you can come up to speed on a basic issue and how you actually work a problem to a solution in the true environment. This is very different from establishing that you can minimize the number of trips a farmer takes across a river with five foxes, three hens, six bag of lentils, a sewing machine and a trapeze.

I encourage you to share some of your ideas for improving the interview process.

In these days of tight budgets but no shortage of things to do, more and more companies are finding that having a flexible workforce is key. This means that having the ability to apply immediate resources to any project is paramount. But also as important, is the ability to de-staff a project quickly and without the messiness of layoffs.

While this harsh work environment seems challenging, it actually can very rewarding both professionally and monetarily and see both the employers and employees coming out winners.

The employees have the benefit of being able to work on a wide variety of disparate projects. This can yield a level of excitement unlikely to be experienced in a full time position that is usually focussed on developing deep expertise in a narrow area. The employers get the ability to quickly staff up to meet schedules and requirements and the ability to scale back just as quickly.

Of course, this flexibility – by definition – means that there is no stability and limited predictability for both employees and employers. The employees don’t know when or where they will see the next job and the employers don’t know if they will get the staff they need when they need it. While some thrive in this sort of environment, others seek the security of knowing with some degree of certainty what tomorrow brings. With enough experience with a single contractor, an employer can choose to attempt to flip the contractor from a “renter” to an “owner”. Similarly, the contractor may find the work atmosphere so enticing that settling down and getting some “equity” might be ideal.

It is a strange but mutually beneficial arrangement with each party having equal stance and in effect both having the right of first refusal in the relationship. And it may very well be the new normal in the workplace.