Expertise during a data explosion

No one really knows how much data is being created in the world. We know that most of the data that exists was created in the past couple of years. Some people say it is doubling every year. The reason that this is even possible comes back to lower barriers of entry. Until we had low cost computing, internet connectivity, and more recently the smart phone, data was isolated, segmented and verified by institutions who were given the authority to create, curate and store it. Authority in this instance was a function of finance. The cost of creating permanent information was expensive – print materials, broadcast hardware, costs of distribution all limited the ability for information to be created and shared. It meant there was far less data, but it also meant we knew where to look to find what was available.

Data has moved from being something which was structured, in know-able places, to something which is unstructured, distributed and without authority. It’s now organic, alive and rapidly evolving. Authority and tools go hand in hand. Now that the tools of creating and storing data are omnipresent and almost free, their is no authority governing it. This means two important things:

  • It will continue to increase exponentially
  • Knowledge no longer has a boundary

So how can anyone be an expert on anything?

In this environment expertise has no choice but to change. No one can know everything, even in the most niche of subjects. If we add to this the idea that the major factors of production are shared – that being 1’s and 0’s – then the potential for cross fertilisation of ideas is infinite. What is true today might be kiboshed tomorrow by new inventions, ideas and collaborations.

The new art of expertise has to become this – knowing where to look and who to rely on.

While we’ll never know everything that has ‘just happened’ and we’ll never be able to predict exactly what is next, we can study the trajectory. Pattern recognition, is quickly becoming more important than knowing. What experts will need to be able to determine in the future is how likely something is, how to assess the sentiment of future behaviour and how to be able to verify what just happened. Expertise is becoming a weird kind of reverse archaeology.

Increasingly what we need to know is how to work with the tools to uncover knowledge as it is created.  The age of memorising things for future reference is quickly becoming obsolete.

– – –

Come join me Tuesday Night in Melbourne to dig into your future and some of The Lessons School Forgot – register here. See you then, Steve.

We need to open up the digital black box

Imagine for a moment you bought something at the supermarket, a packaged food you intend to eat and it didn’t have an ingredient list, let alone a nutrition panel. You’d think twice about consuming it wouldn’t you? In the early days of packaged foods this was the way things were. It was exactly as explained – we had no ideas what was in the box. Before the early 20th century food and medicine we consumed was a total crap shoot. There were literally no regulations, or information as to what we were all consuming if it wasn’t in its raw form. Eventually, we collectively decided this wasn’t good enough.

Boxes filled with secret ingredients, sometimes dangerous ones, developed by the manufacturers so they could sell more and keep it shelf stable longer, were marketed as natural and safe. Quite often the labelling on food and medicines was down right dangerous. The changes didn’t come easy though, manufacturers (Coca Cola, Kelloggs, Phillip Morris, Johnson & Johnson, Unilever, Nestle to name a few) not only sought to mislead to sell more, but often claimed the ingredients in the box were some kind of trade secret that, were valid in a capitalist society. A black box of secrets? –  Sounds a little like something that is going on today doesn’t it?

mind control

Everything we consume in digital form today comes with its own secret set of ingredients. We are ingesting information into our minds that might shape our lives as much as food does. Yet, for some reason we don’t question the opacity of the digital black box. We need to also to be smart enough to realise that shaping peoples minds could well be more dangerous than addictive and unhealthy ingredients are. They don’t just shape us, but they can reshape geo-politics as well, maybe even democracy. The lessons of history are easy to forget, especially when others fought the battles for us before we were born. It’s possible to take for granted the toil undertaken for the civility we bask in today.

A new revolution has arrived with a new set of corporations designing our consumption patterns, and just like before, the behaviour is the same – the large companies at the front line, call ‘competitive secret’ in order to profit seek against an unaware public. We instead need to push back and ensure that the society being built, with a meta structure on top of it, is one we want to live in.

It’s time we exposed the ingredients in the algorithms that shape our digital existence. We need this in the same way we know ingredients in packaged food. Algorithmic Nutrition Panels might just be the start of a more informed society. People might finally understand why they’re addicted to Facebook or Buzzfeed and why they see what they do on the screens in in their lives. It might just start a movement, one in which people realise what they’re actually giving away and why it matters.

What goes in our mind, is at least as important as what enters our mouths. It’s worth mentioning here that we are only on the precipice of the data deluge. The pending trillion sensor economy made possible the internet of things will mean every human interaction involves an algorithm. Algorithms as trade secrets, will increasingly shape our world and our lives, for as long as we tolerate it.

Imagine if you had all the power in the world

IBM Watson

In 2011 IBM’s Watson an Artificial Intelligence unit famously won a 100 game set of jeopardy against human world champions. It was really the first time an A.I. had beaten humans in a multidimensional association test. Intelligence through inference, cognitive computing and natural language rather than mere brute force calculation and probability sets. That said, there was plenty of brute force behind Watson. Here’s some of stats from how Watson was configured at that time in 2011:

  • It had 3000 Power7 processors.
  • A cluster of 90 IBM Power 750 servers.
  • It could read the equivalent of 1 million books per second.
  • It had downloaded the entire content of Wikipedia.
  • It was not connected to the internet for the competition.
  • It was the size of a small conference room. (above)

But this was all 5 years ago. Now its the size of 3 pizza boxes.

3 Pizza Boxes stacked

Imagine if us mere mortals and startup founders could access such computing power. Well, we can, Watson is already in the cloud.

Anyone can access it’s incredible power through the Watson Developer cloud. It even includes 30 day free trials – 30 days with the worlds most powerful A.I. – I can remember when an internet cafe used to charge $20 an hour for dial up to surf sites like GeoCities and Altavista. You can check out all the ways it’s being used here. You’ll see there are incredible opportunities for all manner of business, data usage and cognitive assistance.

This opportunity simply can’t be understated. Yes, IBM benefit from the work you do (it makes Watson smarter) and it will cost money, but heck, you’re getting access to the most powerful thing of it’s kind in the world.

It turns out that while some companies continue to get stronger and more ensconced in our lives – think GAFA, there’s still a move towards democratisation if we care enough to dig around.

Follow me on SnapChat – search ‘Sammartron’ for more business insight. Click here if on mobile to add now.