Database management in public sector organisations is challenging because of the volumes of data involved and long retention periods. In this article Joe Kim offers four principles to reduce complexity.

Confucius wrote, “Life is really simple, but we insist on making it complicated.” This is applicable to most aspects of life, yet could not be more applicable to the way government organisations manage data. Although it may feel necessary to collect large amounts of data and store it for many years, this process is actually extremely inefficient and unnecessary. It can escalate costs through constantly having to scale up with storage solutions, while complicating processes due to the vast amounts of data which need to be sifted through.

If you’re a government database administrator, embracing the following four principles will help alleviate your pain and simplify your processes.

There’s no need to store everything
As anything and everything is connected these days, it can be tempting to collect the infinite amount of data floating around, but it doesn’t necessarily mean you should.

The key is in understanding what your organisation needs and what is important at this particular time so that you only collect what is necessary. The database administrator needs to ask: “What are our current priorities, and what type of data do we need to support these?”

Sadly, there is no ‘one-click sorting’ technology for this. It requires insight into the overall goals and objectives of the organisation. However this doesn’t have to be a daunting task involving intricate calculations, but can instead be approached by using simple algebra. By getting into the habit of organising and discarding unwanted data, processes will become faster and more efficient, making the organisation of data a much simpler feat.

Keep alerts and notifications simple
Keeping it simple also applies when setting up automated alert software. Although it may be tempting to set up alerts to notify you when just about anything happens – this can result in an alert overload, making it hard to know when an actual problem arises.

It’s better to set up a simple system consisting of both alerts and notifications, but it’s important to understand the distinct differences between the two. Alerts let you know of something which is timely and needs to be taken care of immediately, such as a security breach. Notifications, although they cannot be ignored, are less timely. They might inform you of performance issues impacting your response time, for example. A simple set-up of alerts and notifications will feed you the information you need without overwhelming you.

Prioritise your data
Though it’s not advisable to keep all data, there’s no hard and fast rule about how much data to keep or how long to keep it. Many agencies use the unwritten “seven year rule” which dictates that data should not be kept for any longer than that period of time. Yet, most agencies exceed this length of time, even though they are not required to. Instead of archiving, database administrators need to consider using a tiered storage approach. Tiered solutions help enhance efficiencies by prioritising data by date and importance, separating it from older and less useful data. For example, tier 1 storage could be used to store the newer and more important data, while tier 2 or 3 storage sets are used for older data, as they are generally slower and therefore take more time to retrieve. Again, it’s all about simplification of processes. By separating what you need from what you don’t, your database will run more efficiently.

Admit when you need help
Although external audits are never fun, they can often be very useful and eye-opening, particularly for managers who are drowning in data. Every database manager should consider hiring outside expert data analysts to provide an unbiased report of the organisation’s data management practices. These analysts are often used to being knee-deep in data and can provide expert advice on what types of data a government manager needs to collect and what to do with it, which can be invaluable. Ultimately, external analysts can help the public sector database administrator focus data management efforts – simplifying processes and data storage in a way that would make Confucius proud.

Joe Kim is SVP and CTO at SolarWinds.