Ronnie05's Blog

Is Big Data in reality only a hyperbole?

Posted in Big Data, Semantic Media and Web, Semantic Web by Manas Ganguly on September 29, 2013

The World Economic Forum (WEF) calls it “the new oil” and “a new asset class”. The vast loads of data have been likened to transformative innovations like the steam locomotive, electricity grids, steel, air-conditioning and the radio.

Big Data

There were 30 billion gigabytes of video, e-mails, Web transactions and business-to-business analytics in 2005. The total is expected to reach more than 20 times that figure in 2013, according to Cisco. Cisco estimates that in 2012, some 2 trillion minutes of video alone traversed the internet every month.

What is sometimes referred to as the internet’s first wave — from the 1990s until around 2005 — brought completely new services like e-mail, the Web, online search and eventually broadband. The next one – connected the world into social grids giving people identity and voice. For its next act, the industry has pinned its hopes, and its colossal public relations machine, on the power of Big Data itself to supercharge the economy. Some call it Web 3.0, some call it Big Data.

Is Big Data pure hyperbole….

There is just one tiny problem: the economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic. The rate of productivity growth, whose steady rise from the 1970s well into the 2000s has been credited to earlier phases in the computer and internet revolutions, has actually fallen. The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance.

All that promise of Big Data or even Social web hasn’t exactly fired the economic engines of the world as they were expected to. The promise is real – so why would such a disturbing trend start building – One theory holds that the Big Data industry is thriving more by cannibalising existing businesses than by creating fundamentally new opportunities. Online companies often eat up traditional advertising, media, music and retailing businesses, said Joel Waldfogel, an economist at the University of Minnesota. “One falls, one rises — it’s pretty clear the digital kind is a substitute to the physical kind,” he said. “So it would be crazy to count the whole rise in digital as a net addition to the economy.”

… or are these early days?

Other economists believe that Big Data’s economic punch is just a few years away, as engineers trained in data manipulation make their way through college and as data-driven start-ups begin hiring. And, of course, the recession could be masking the impact of the data revolution in ways economists don’t yet grasp. Still, some suspect that in the end our current framework for understanding Big Data and “the cloud” could be a mirage.

There is no disputing that a wide spectrum of businesses are now using huge amounts of data as part of their everyday business.

Josh Marks (CEO masFlight) helps airlines use enormous data sets to reduce fuel consumption and improve overall performance.Although his first mission is to help clients compete with other airlines for customers, Marks believes that efficiencies like those his company is chasing should eventually expand the global economy. For now, though, he acknowledges that most of the raw data flowing across the Web has limited economic value: far more useful is specialised data in the hands of analysts with a deep understanding of specific industries.

Some economists argue that it is often difficult to estimate the true value of new technologies, and that Big Data may already be delivering benefits that are uncounted in official economic statistics.

Also, infrastructure investments often take years to pay off in a big way, said Shane Greenstein, economist at Northwestern University. He cited high-speed internet connections laid down in the late 1990s that have driven profits only recently. But he noted that in contrast to internet’s first wave, which created services like the Web and e-mail, the impact of the second wave — the Big Data revolution — is harder to discern above the noise of broader economic activity.

… reproduced from Will big data prove to be an economic big dud?

Facebook, Big Data and Project Prism

Posted in Big Data by Manas Ganguly on August 24, 2012

Facebook processes 2.5 billion pieces of content and 500+ terabytes of data each day. It’s pulling in 2.7 billion Like actions and 300 million photos per day, and it scans roughly 105 terabytes of data each half hour. The speed of data ingestion keeps on increasing, and the world is getting hungrier and hungrier for data. Facebook’s latest effort is about putting all this data in some perspective, to mine this data for insights across different storage clusters with efficient use of resources and cost leading to real time live performance management on data outputs. And to achieve a seamless integration of data across huge data centres, Facebook has put in place initiatives such as Project Prism and Corona.

‘Project Prism,’ will allow Facebook to maintain data in multiple data centers around the globe while allowing company engineers to maintain a holistic view of it, thanks to tools such as automatic replication. Corona, makes its Facebooks’ Apache Hadoop clusters less crash-prone while increasing the number of tasks that can be run on the infrastructure.

So while Google is indexing information around the world, Facebook is indexing user behavior and reactions to a wide range of stimulus around the world. Now then, the only thing that Facebook would ideally want to fix is the ability to sell this data and get a good price for its share.

Tagged with: ,

Integrating the elements of convergence: The case for APIs

Posted in Internet and Search, Mobile Data & Traffic, The cloud and the open source by Manas Ganguly on February 28, 2012

Convergence has been the buzz word for a good part of the last decade and will continue to do so in this decade as well. However, for the discerning the definition or at least the meaning of convergence has now shifted from device convergence to technology convergence. The later being the superset of which devices are just another maifestation. So earlier its was the camera, the mobile phone, the GPS, the MP3 player and other such device charecterestics that really converged. However, in the present context it is the convergence of enabling technologies and the three big technologies that seem to be convergent at this time are: Mobility, Cloud Services and Big Data.

However, it is a relatively small lynchpin that drives the convergence of these three mega trends. Small in terms of what it is, but large in terms of the innovation spurts that it provides. The key here is APIs or Application programming Interfaces. APIs tie together the mega-trends in a fundamental and unalterable way. APIs are the lingua franca of the new wave in internet of all things combined with super mobility and seamless connectivity. In my mind, each of these three technology trends (on their own) will be on the fast track to commoditization and will risk facing the same fate as did most social business software plays. The magic and the premiums will come from contextual application of this innovation and smart integration.

To stake a few examples, Box.net as storage without document and device sync and collaboration is commodity. Apple’s iCloud as storage without ubiquitous local and iTunes media sync across devices is commodity. And Google Drive (as discussed here in Ben Kepes’ CloudU community) is also a commodity business not worth getting into had it not been for Google’s services such as Google Apps, Piccasa, and its media and unified communication capabilities under the Google Plus brand.

The premiums from big data, mobile access and cloud comes from
a) dynamically assembled media and content, and interpreted data in the cloud,
b) available wherever you need to consume and / or collaborate and
c) insanely focused and simple interfaces to complex backends.

Thats where money would be made in these commoditized services. APIs provide the integration through the value creation network. The only other differentiator in this case being experience!

Big Data: Controlling the beast by its horns

Posted in Mobile Data & Traffic by Manas Ganguly on February 20, 2012

(This is the third of series of posts on Big data and the Internet of Things. Read the first, second and third posts here.)

I look for hot spots in the data, an outbreak of activity that I need to understand. It’s something you can only do with Big Data.” – Jon Kleinberg, a professor at Cornell

Researchers have found a spike in Google search requests for terms like “flu symptoms” and “flu treatments” a couple of weeks before there is an increase in flu patients coming to hospital emergency rooms in a region (and emergency room reports usually lag behind visits by two weeks or so).Global Pulse, a new initiative by the United Nations, wants to leverage Big Data for global development. The group will conduct so-called sentiment analysis of messages in social networks and text messages – using natural-language deciphering software – to help predict job losses, spending reductions or disease outbreaks in a given region. The goal is to use digital early-warning signals to guide assistance programs in advance to, for example, prevent a region from slipping back into poverty.

In economic forecasting, research has shown that trends in increasing or decreasing volumes of housing-related search queries in Google are a more accurate predictor of house sales in the next quarter than the forecasts of real estate economists.

Big Data has its perils, to be sure. With huge data sets and fine-grained measurement, statisticians and computer scientists note, there is increased risk of “false discoveries.” The trouble with seeking a meaningful needle in massive haystacks of data, is that “many bits of straw look like needles.”
Data is tamed and understood using computer and mathematical models. These models, like metaphors in literature, are explanatory simplifications. They are useful for understanding, but they have their limits. A model might spot a correlation and draw a statistical inference that is unfair or discriminatory, based on online searches, affecting the products, bank loans and health insurance a person is offered, privacy advocates warn.

Despite the caveats, there seems to be no turning back. Data is in the driver’s seat. It’s there, it’s useful and it’s valuable, even hip. It’s a revolution. We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.

The age of Big data

Posted in Mobile Data & Traffic by Manas Ganguly on February 14, 2012

Mobile data traffic to increase 18-fold by 2016 – CISCO

Between 2011 and 2016 the amount of mobile data traffic will grow at a compound annual rate of 78 percent as the number of mobile devices connected to the Internet exceeds the number of people on Earth in four years’ time, according to a study by Cisco.The United Nations projects that world population will reach 7.3 billion by 2016. By that time, according to Cisco’s annual visual networking index forecast, there will be more than 10 billion devices, generating global mobile data traffic of 10.8 exabytes per month.

That translates into around 130 exabytes of mobile data per year which is equivalent to 33 billion DVDs or 813 quadrillion text messages. While most of the devices driving mobile data traffic will be smartphones, ,laptops and other portable gadgets; M2M machine-to-machine connections are gaining momentum and by 2016 and are expected to reach 2 billion. Machine-to-machine connections (M2M) include GPS systems in cars, tracking systems in fleets and ships or meters to record energy consumption.

Mobile data, especially video, will put a strain on wireless networks and while service providers are increasingly offloading mobile traffic to their fixed-line networks, there is a need for additional spectrum to keep up with demand. Quality-of-service issues are seen to be arising in major metropolitan areas due to the increase in mobile data traffic.

%d bloggers like this: