Exploring Emerging Data Management Technologies
A Guide for Development Professionals
Ten years ago, data management systems for the development sector looked very different from what they do today. The primary means of collecting data were largely surveys and spreadsheets, followed by human beings who would consolidate, clean, summarize, and analyze data. A few organizations who were slightly ahead of the curve were starting to adopt Salesforce as a data platform given their extremely generous subscription fees. Technology and data systems were considered much after program implementation was underway and primarily as a means of increasing efficiency.
Today, the entire landscape of data management systems (see below) right from data entry to data visualization has undergone a transformation. State-of-the-art technology which earlier seemed only available to large for-profit enterprises is now within reach of nonprofit organizations of all sizes.
These changes are characterized by three trends:
low-code & no-code solutions
the rise of open source solutions
extremely low data-storage and computation costs of cloud data warehouses
Low- and no-code solutions
These solutions allow end-users with little or no coding knowledge to quickly build software for narrowly defined use cases.
Appsheet, for example, allows us to quickly build simple data-entry applications and run pilots with up to 10 users for free with Google Sheets acting as the database.
On the data visualization front, both Metabase and Apache Superset provide really strong user experiences and dashboards that can be consumed right from the CEO’s desk to mobile phones used by field staff.
Rise of open source
These are platforms that make code written for different use cases publicly accessible. They have allowed organizations to scale faster than what per-user licenses of closed-source software would allow.
Open source solutions, however, do require an organization to:
pay for the infrastructure, e.g. servers, to run software
invest in setting up an internal technology team or work with external vendors to set up, debug, upgrade and ensure uptime of both hardware and software.
Cloud Data Warehouses
These services (e.g. Amazon RedShift, Google BigQuery) have revolutionized the amount of data that can be ingested, stored, transformed, and finally visualized for analysis at extremely low per-unit costs. The solutions require close to zero initial setup and most data visualization tools provide first-party integrations to allow users to quickly connect and start analyzing large datasets. Further, most cloud providers offer liberal discounts and free credits to registered non-profits, which can be used for any of their cloud services (e.g. databases, automated translation APIs, file storage).
These advancements taken together mean that data systems can now be at the center of program design. Organizations can now ask deep, nuanced questions of their program data, process large and diverse datasets to answer them, and run pilots and experiments to investigate the true impact of their programs.
The authors Swapneel Rane & Aditya Krishnan, part of the team at Goalkeep, are passionate about enabling the social sector to maximize impact through data.
"Final Call: Join us Tomorrow from 4pm to 5pm for an Exciting Session on
'Impact Framework & Data Management Systems: How to Get Started?'"