The Business Intelligence Blog

Sensors everywhere. Infinite storage. Clouds of processors. Our ability to capture, warehouse, and understand massive amounts of data is changing science, medicine, business, and technology. As our collection of facts and figures grows, so will the opportunity to find answers to fundamental questions. Because in the era of big data, more isn’t just more. More is different.

This month’s Wired magazine carries one of the most important growing concerns of the scientific community, the uncontrollable growth of data. This growth of data in many directions is nearly killing theories as everything is becoming more and more data controlled.

There are a series of articles ranging from what data miners are digging today to elaborate algorithms that predict air ticket prices to how we can monitor epidemics hour by hour.

If you are a BI entusiast or not, this month’s Wired cover story will challenge all your predictions about science and technology, even if you have a petabyte of data to support it !! Read it, like, right now !!

Here’s a follow-up on Reality Mining and Surprise Modelling, which are called as one of the 10 technologies that we think are most likely to change the way we live.

Read more from Forbes.com interview with Sandy Pentland, director of MIT’s Human Dynamics Research program.

Forbes.com: What is “reality mining?”

Sandy Pentland: Reality mining is about using sensors to understand human beings. The sensors could be security cameras, they could be devices that you wear on yourself, they could be cell phones. The point is it’s about people. Data mining is about finding patterns in digital stuff. I’m more interested specifically in finding patterns in humans. I’m taking data mining out into the real world.

What kind of reality-mining experiments have you actually performed?

We developed this thing called a sociometer, a little badge that you wear around your neck that records your body language, your motion and your tone of voice–the tone, not the words. It gives us a nice little package for reality mining.

We’ve done all sorts of interesting things with this. Just listening to peoples’ tones of voice and how they move, we can measure interest level and attention, factors that account for 40% of the variation in the outcomes of things like salary negotiation, dating scenarios, closing a sale, pitching a business plan.

Another reason why cloud computing is the in-thing these days. From WP -

Microsoft Corp sees tens of millions of corporate e-mail accounts moving to its data centers over the next five years, shifting to a business model that may thin profit margins but generate more revenue.

In an interview ahead of the Reuters Global Technology, Media and Telecoms Summit, Chris Capossela, who manages Microsoft’s Office products, said the company will see more and more companies abandon their own in-house computer systems and shift to “cloud computing,” a less expensive alternative.

Cloud computing is the trend by Internet powerhouses to array huge numbers of computers in centralized data centers to deliver Web-based applications to far-flung users.

Microsoft built its business selling software to run on local machines, both computer servers and personal computers, but, in recent years, it has invested billions of dollars in massive data centers, which are the basic infrastructure for a wide range of Web services.

An interesting analysis from PC World -

…the search market has fragmented into a few distinct size classes, analysts say: offerings from major vendors like IBM, Oracle and with its recent acquisition of FAST Search & Transfer, Microsoft; larger independents such as Autonomy; and smaller, specialized vendors.

Arnold recently wrote a nearly 300-page study for Gilbane Group, “Beyond Search,” that takes a deep dive into the facets of the enterprise search market. While in terms of size, search-focused companies are spread among only a handful of categories, but they vary widely in terms of their technological focus. These are among the sub-segments Arnold identified:

Database-centric systems, such as Teratext and Intelligenx. “Because of this, these systems are adept at handling data management, content repurposing, and generating reports from the content that reside in the system’s database,” he wrote.

Companies involved in “deep analysis” of content, which include Attensity and Siderean Software. “The use of multiple processes in iterative cascades point to the direction search and content processing is moving. Simple key word indexing is a Model-T Ford to these vendors’ finely tuned machines.”

Microsoft’s Advance08 - the advertising conference featuring James Cameroon & Bill Gates, starts today at Redmond.

The prime focus is on digital advertising and how the future of digital media will touch people’s life. This event is also looked forward as this will be Bill Gate’s one of his last public appearances as a full-time Microsoft employee.

The global advertising community’s most influential leaders have come together at advance08 to debate and discuss the factors affecting the future of advertising media:

How is new media reshaping the relationship between brands and consumers?
How do marketers work and communicate with their target audience?
Where is the future of media heading, and what does it look like?

Building on the successful format of SAS, advance08 focuses on the main themes that are reshaping the landscape of digital marketing. Learn more about these topics from prominent figures in the global advertising community, from Michael Eisner and James Cameron to visionary Bill Gates.

Varolii Corporation today announced Varolii Predictive Analytics(TM), an on-demand service that helps organizations understand, analyze and strategically target more effective customer outreach for collections, customer service and loyalty programs. Varolii unveiled Predictive Analytics at Interaction ‘08, the company’s inaugural customer conference.

Combining client data with the behavioral insight gleaned from billions of Varolii-led customer interactions, Varolii Predictive Analytics helps companies identify who to contact, when to contact them, through what channel, using which treatment, and how frequently in order to generate the greatest response. This combined data offers much more than a simple success or failure measurement; it’s a complete record of customer interaction and response on an individual level. The application is designed to gain intelligence from these conversations and progressively “learn” to predict customer behavior.
..
..
At the highest level, Varolii Predictive Analytics can help companies analyze contact strategy options (e.g., automated communications, an in-house call center, outsourced agents, or a combination of them) and select which will work best to solve their specific needs. At a more tactical level, it can help decide such things as an appropriate collection strategy based on Varolii and customer history, including probability to pay, days delinquent, and other metrics for improved results.

Read more from the Press Release.

“As an early adopter of RAP – The Trading Edition, our trading desk has been able to be more responsive to market opportunities through faster, more complete analysis of market data, both historical and real-time feeds,” said Jason Ross, director, Rates Trading Group at MUSI. “We were pleasantly surprised at the performance speed, and found we need only half the storage capacity to store the same amount of data.”

Other key beneficiaries of the solution include quantitative analysts, risk managers, and compliance officers. Analysts can run powerful quantitative models against years’ worth of time-series market data, trading histories, and reference data, with complex analysis running in minutes vs. hours; risk managers can monitor balance sheet exposure hourly to reduce firm-wide risk; and compliance officers can audit the trade flow to meet the latest market compliance requirements.

“RAP – The Trading Edition is the only integrated solution today with the ability to absorb streaming market data feeds and consolidate these with vast stores of historical data for common shared access and analysis. We offer an enterprise-class platform that can offer a single source of data to the entire trade life cycle, from quantitative analysis, through pre-trade pricing, to market risk analysis and compliance,” said Dr. Raj Nathan, Chief Marketing Officer at Sybase. “In today’s unforgiving market environment, financial services firms need best-in-class, bullet-proof solutions that afford instantaneous throughput with no down-time. Sybase® RAP – The Trading Edition is very robust, having a built-in administrative console and dependable business continuity/disaster recovery solutions to protect against system failures.”

More on RAP - The Trading Edition from the Press Release.

Anand Rajaraman, the co-founder of Kosmix also teaches Data Mining at Stanford. Here’s an interesting note from his blog.

Some of his students are working to crack algorithms for the on-going Netflix “Better Recommendation Logic” Prize of $1 million. Read it !!

Here’s how the competition works. Netflix has provided a large data set that tells you how nearly half a million people have rated about 18,000 movies. Based on these ratings, you are asked to predict the ratings of these users for movies in the set that they have not rated. The first team to beat the accuracy of Netflix’s proprietary algorithm by a certain margin wins a prize of $1 million!

Different student teams in my class adopted different approaches to the problem, using both published algorithms and novel ideas. Of these, the results from two of the teams illustrate a broader point. Team A came up with a very sophisticated algorithm using the Netflix data. Team B used a very simple algorithm, but they added in additional data beyond the Netflix set: information about movie genres from the Internet Movie Database (IMDB). Guess which team did better?

Bitam is set to showcase Bitam G6, the next gen BI tools. Here’s more info at Fox Business.

Last month, Bitam announced the release of its latest version of Enterprise Performance Management (EPM: 4.40, -0.15, -3.29%) solutions, Bitam G6. On Thursday, April 10, 2008, Bitam is hosting a live Webinar to demonstrate how companies of any size can use Bitam G6 to significantly improve their business performance. Bitam’s sixth generation of applications for EPM enables users to reach new levels of control, accuracy and flexibility in their decision making process and achieve success with greater confidence.

We will show how early adopters of Bitam G6 have captured the power of their data, aligned their metrics with corporate strategy, and optimized performance,” said David Abdo, CEO of Bitam. “Simple user interfaces allow workers at all levels to create dashboards of graphically-rich operational information, measure key performance indicators (KPIs), align strategies to corporate objectives, and drill down into data for more detailed analysis — freeing IT staff to work on other projects.”

To register for this free Webinar, visit http://www.bitam.com. The event will also be held live in Monterrey, Mexico on Thursday, April 10th at 10:15 a.m. (EDT).

Reading this Technology Review, it seems inevitable that such advanced mining technologies will pop-up in the near future. The world has a wealth of information and every single thing will be data mined in the future. And what a movement that will be.

By the way, the MIT Technology Review calls Reality Mining as one of the 10 technologies that we think are most likely to change the way we live. Exciting, Ain’t it ?

Also Surprise Modeling which combines data mining and machine learning to help people do a better job of anticipating and coping with unusual events is also one of the Top 10 Technologies listed by MIT Tech Review. This is being advocated by Eric Horvitz, Microsoft Research.

From the article on Reality Mining -

Reality mining, he says, “is all about paying attention to patterns in life and using that information to help [with] things like setting privacy patterns, sharing things with people, notifying people–basically, to help you live your life.”

Within the next few years, Pentland predicts, reality mining will become more common, thanks in part to the proliferation and increasing sophistication of cell phones. Many handheld devices now have the processing power of low-end desktop computers, and they can also collect more varied data, thanks to devices such as GPS chips that track location. And researchers such as Pentland are getting better at making sense of all that information.

To create an accurate model of a person’s social network, for example, Pentland’s team combines a phone’s call logs with information about its proximity to other people’s devices, which is continuously collected by Bluetooth sensors. With the help of factor analysis, a statistical technique commonly used in the social sciences to explain correlations among multiple variables, the team identifies patterns in the data and translates them into maps of social relationships. Such maps could be used, for instance, to accurately categorize the people in your address book as friends, family members, acquaintances, or coworkers. In turn, this information could be used to automatically establish privacy settings–for instance, allowing only your family to view your schedule. With location data added in, the phone could predict when you would be near someone in your network. In a paper published last May, ­Pentland and his group showed that cell-phone data enabled them to accurately model the social networks of about 100 MIT students and professors. They could also precisely predict where subjects would meet with members of their networks on any given day of the week.