Features: August 3rd, 2012

Public bodies hold vast amounts of data, but the value of this asset is unknown. What is certain is that the time of citizens and the money of taxpayers can be saved if the asset can be expoloited. Mikko Soirola explores the opportunities.

A recent report from the Policy Exchange Think-tank claimed that ‘£33bn a year could be saved from public spending if the government makes better use of data already in the system’. This highlights a long-standing issue that affects most public sector organisations today – how to harness this valuable resource to deliver extra efficiencies and reduce costs? The report describes a huge opportunity for the public sector to re-think its strategy and create something that will have an instant and lasting effect.

Cost cutting at a stroke

Decades of official form filling from tax returns and council tax details to NHS records and DVLA applications (and everything in between) has meant government departments and associated organisations have amassed huge amounts of data on the whole population. Over the past decade this data has been captured in digital format, which provides a distinct handling advantage over paper-based records. However, the distinct advantages of capturing data in digital form remain mostly unexplored and most government departments have accumulated data independently without a thought for the bigger picture.

In the recent Policy Exchange report, The Big Data Opportunity, Chris Yiu, Head of the Digital Government Unit supports the idea of sharing data between departments; “The public sector is made up of many thousands of different organisations, ranging from large departments of state to individual schools, surgeries and libraries. Each of these organisations knows many things about its operations and the people that it deals with. Finding ways to share or link this data together has the potential to save time for citizens and money for taxpayers.”

Driving factors – a perfect storm

There are a number of factors which are combining to create the ‘perfect big data storm’. They include a change in attitude regarding public sector tendering for IT services; cloud computing; legacy IT systems; increased outsourcing; and increasing crime, and I will now take a look at each in turn.

Large System Integrators

Historically, public sector IT tenders were most often won by the big technology integrators. However, following a number of high profile failures there has been a backlash against these large firms. The general realisation has been that better value and service can be had from smaller, more specialist suppliers, which can deliver more sophisticated, intuitive tools at lower overall cost and, most importantly, on budget.

Legacy systems

Decades of restricted budgets and low investment has left the public sector with many disparate IT systems filled with legacy data. The data remains of limited use as different departments cannot share and use it due to integration or formatting issues.

Cloud computing

This is the decade of the cloud and the idea of treating IT as a service is fast catching on. Driven by advances in technology, infrastructure improvements and economic factors, the cloud presents an attractive alternative. Today you can rent any IT service you want at a fraction of the cost of buying it and managing it yourself. You can divest yourself of the whole thing (using a Cloud Services Broker), or retain part responsibility (via a hybrid cloud model) which opens up numerous possibilities. Through subscription-based Software as a Service models you can ‘rent as you use’ applications that prior to cloud, required significant upfront investment. You can also scale users up (or down) at relatively short notice.


Closely connected to cloud adoption is the massively improved network infrastructure that is now available. Mobile connectivity is now commonplace and bandwidth speeds are as fast as you need. Such fast, easy and readily-available access to any cloud-based application removes the need for users to download sensitive information to personal laptops or storage devices. Could this bring to an end embarrassing stories about top secret data held on government laptops being left in taxis?

Increased outsourcing

There is an increasing trend to outsource public sector services. As the supply chain increases, so does the need to integrate and use more data from, often, proprietary external sources.


As the economy gets tougher crime rises and the numbers of criminal activities increase, especially around social security fraud and ID theft. To combat this growing threat, having access to, and being able to share information with other organisations, is essential. But the sheer volumes of data that need to be harnessed, moved and shared between departments moves us into the territory of Big Data.

The new world of Big Data

Big Data is a term used to define data sets so large and complex that they become awkward to work with. The continuing accumulation of new data is a by-product of the digital world we now all live and work in and data storage is growing almost 50% year on year in order to cope. Computing power is growing just as quickly and as social media and easy online access become the norm it all feeds the data store – so data sets continue to grow and become more and more unwieldy. The cost of coping with big data is getting out of control.

• Government departments reported spending, on average, £236,004 each year on storage, with spending reaching as high as £1.8M
• Government departments are buying, on average, at least 101TB of storage every year
Figures June 2012 from Freedom of Information Act request by Nexenta® Systems

More than a simple storage problem

Against this backdrop of change, the common aspect that remains constant is data growth. Governments are only just learning to cope with the huge quantities of data that they now generate and store on a daily basis. This big data revolution has the potential to be one of the most important global trends for the coming decade and while business has embraced the opportunity for some time now, it is long overdue for the public sector to follow suit. Although to its credit, Whitehall has now recognised Big Data can be a significant source of value and competitive advantage, rather than ‘just a storage problem’.

However, the use of any ‘data’ comes with the caveat of data privacy compliance and as Yiu points out, “Across all of this agenda, we believe government should only execute on data and analytics where it is prepared to make an open and transparent case about the public policy benefits.”

On the subject of a solution Yiu goes on to say, “With previous generations of technology, this often meant there was no alternative to storing multiple copies of the same data (or simply coping without some of the data altogether). Modern technology, however, can enable fragments of related information to be matched and linked together quickly and non-persistently. This can be used to streamline transactions – reducing the scope for errors and avoiding asking people to provide the same information multiple times.”

Automation is hugely popular

One technology that provides a cost effective answer is integration and data management solutions.

Using specialist Data Integration tools, Data Integration Service Providers work closely with a client organisation to create a ‘data exchange area’, where information from suppliers and partners is ‘captured, translated and mapped’ into a format which is recognised by all the client organisation’s systems. This fully automated process is proving hugely popular as it overcomes a need to do anything manually once the data integration map has been created. This automation has huge significance when looking at time, resource sharing and access efficiencies across independent but associated organisations.

These data exchange areas can be hosted either in the cloud or exist as part of an organisation’s own IT infrastructure. Either way, they deliver a flexibility which particularly suits industry sectors which may have to comply to security legislation such as financial institutions and government departments, or heavily regulated sectors such as pharmaceutical. They are also particularly well-suited to public sector application. Yiu’s The Big Data report supplies an illustration; “The process for obtaining a UK driving license is a good example of data sharing in action. When you apply for a driving license online, the Driver and Vehicle Licensing Agency requires a photograph and signature for your new licence. If you have a UK passport then the DVLA will try to capture these electronically from the information already held by the Identity and Passport Service.”

Big picture and little picture advantages

In conclusion, with the public sector under intense pressure to reduce costs while maintaining services, the technology and specialist suppliers are already out there to address many issues including data integration and exchange. Improved accessibility through specialised data integration and management solutions will deliver both ‘big picture’ and ‘little picture’ advantages to the public sector and increase efficiencies while reducing costs. However the public sector needs to accept that large System Integrators are not necessarily the best option.

There are barriers to adoption, most notably around individual data protection and data privacy concerns. However, the recent joint IT initiative by Hammersmith & Fulham, Kensington Chelsea and Westminster councils is breaking new ground in inter-public sector organisation co-operation, and in my view is the forerunner of things to come. It is my belief future public sector alliances and initiatives will become commonplace and automated integration and data management will not only provide a ‘cross-organisation’ solution today, but will do the same for tomorrow.

Mikko Soirola, is VP for Liaison Technologies a global provider of secure cloud-based integration and data management services and solutions.