Case Study
Government Data Repository
Tens of millions of public records available online in an easy-to-use, searchable database.
"Web Instinct stepped in to rescue our website rebuild at a critical juncture and delivered us a fully functioning and well-designed product in short order and for a reasonable price. Their customer service, support and work on follow-up and new projects has been equally exceptional."
Tim Hoefer, Executive Director, Empire Center
Highlights
The website provides searches across over 34 million rows of data.
The public pulls requests from this data over 4 million times each month. The majority of searches are delivered in less than 1 second.
New data can be added through CSV file uploads, which are vetted and then seamlessly integrated into existing data repositories.
Not just any server can deliver this amount of data at this load. Our can. And the data is backed up twice each day automatically.
Challenge.
The Empire Center collects millions of public records data from New York State and local governments within it to achieve its goals of achieving policy change to save taxpayers money.
While this data is shared with commentary on their website, they wanted to make this information available to the public without commentary - raw data to allow reporters and residents to easily find pertinent information about their state and local governments.
They approached us with a mish-mash of various outputs and methods - each dataset having its own solution, its own URL and its own difficulty in usability. Our job was to change all of that.
Solved.
With tens of millions of data records across a handful of datasets (such as state employee payrolls), the ability to quickly filter the data to find what you are looking for was paramount. Too, the website needed to aggregate the data (e.g. provide a total sum of everyone on payroll for your filter) and provide an easy way to share the data with the specified filter.
Our team brought all data into a single database repository. We fine-tuned the databases for optimum performance allowing most filters to return their results in under 1 second. All filters are saved and logged to provide valuable feedback as to where the public interest lies.
Keeping these datasets updated was also critical, without having to engage developers or database managers to do so. We built a multi-step process in their dashboard that allows them to upload an Excel file and seamlessly merge that data into the existing dataset, with no downtime. Any update can be rolled back with the click of a button.
Hosted on our most powerful server, the website delivers 4 million searches each month across 34 million records of data, quickly and efficiently. Each dataset is delivered in a similar mobile-friendly layout making a user's movement between datasets seamless and easy to understand.