THE "big data" phenomenon is set to be a game-changing trend for every industry, government and society in the next few years.

"Big data" is expected to join the likes of mobile and cloud computing this year as the next "must-have" competency, as the volume of digital content continues to grow.

"Big data" refers to the software tools, processes and procedures which allow companies to create and manage very large data sets that conventional database systems are unable to handle.

Companies use big-data technology for insights into new and emerging types of data and content and related trends, to make their businesses more agile.

For example, a company may analyse big data in order to monitor customer perceptions and sentiments about its brand on social media websites.

The increasing volume and detail of information, the rise of multimedia, social media, and the rise in internet access is fuelling exponential growth in data, making it more challenging to manage, according to analysts.

International Data Corporation (IDC) projects that the worldwide big data technology and services market will grow at a 31.7% compound annual growth rate — about seven times the rate of the overall information and communication technology market — with revenue reaching $23.8bn in 2016.

"The big data market is emerging at a rapid pace and incorporating technology and services from a wide range of existing and new market segments," says IDC.

Gartner predicts that 4.4-million IT jobs globally will be created to support big data by 2015.

Research by international group IBM shows that most organisations are in the early stages of big data planning and development efforts, with the majority focused either on understanding the concept or defining a roadmap related to it. However, some companies are implementing big data pilot projects or have already implemented big data solutions at scale.

In South Africa, Santam, the insurance company, is using analytics to define a new way of assessing claims from customers while also reducing business risk — in fact the claims process foiled a major fraud syndicate, saving $2.4m in the first four months of operation, says IBM big data expert Inhi Cho Suh.

"Having a handle on big data means they can enable claims settlements significantly faster than before and they lowered their operational costs by reducing the need for claims adjuster visits for low-risk claims assessments," Ms Suh says.

Martin Rennhackkamp, chief information officer of local technology firm PBT Group, has noticed that some local companies are already embracing the trend, "which of course is great for the broader industry".

However, "we do stress the importance of fully understanding the trend, as in South Africa in particular, many companies do not necessary need big data in their organisation," he says.

Many local businesses are still coming to terms with the concept — to understand what it is all about and how it will benefit them.

"The reality is that big data can be very beneficial to a particular business. However, it must be investigated correctly to ensure its benefits can be reaped," says Mr Rennhackkamp.

Ms Suh says digital data is doubling every two years. "If companies are not evolving, they will lose ground not only locally but in terms of global opportunities. It is not just large business that this applies to — it applies to every company big and small," she says.

There is no business on the planet that should ignore the value of a smartphone or mobile device, says IBM.

Big data is critically linked to other technology-related services such as social media, cloud computing and mobile. In South Africa, nine out of 10 enterprises are involved with cloud computing, says global software firm Symantec.

Gartner says big data is moving from a focus on individual projects to an influence on enterprises’ strategic information architecture. Gartner analysts say the real issue is making sense of big data and finding patterns in it that help organisations make better business decisions.

The ability to manage extreme data will be a core competency of organisations that are increasingly using new forms of information — such as text, social and context, Gartner says. This year big data is forecast to drive $34bn of information and technology spending.

• This article was amended on February 22 2013 to correct a reference to Santam, which previously said Sanlam.