The Open Government Data Initiative

Let a thousand real time dashboards bloom!

The Open Government Data Initiative advocates for laws that force government to proactively publish data in machine-readable formats.

It is non-partisan ; we support exposing all data, whether that might support the policies of the left, the right, both, or neither.

It does not advocate for government-provided tools to view or read data; it advocates only for the publishing of data.

( The OGDI believes in The Unix Philosophy: small tools, loosely coupled. Make the government provide the data and the free market will do the rest! )

Finally, OGDI is based in New Hampshire and advocates first and foremost for state-level laws, but we support activists in other states and look forward to sharing model legislation and discussing best practices.

Who we are

A small loose collection of interested citizens working with Travis Corcoran.

The problem

“Right to know” is

the right for people to “participate in an informed way in decisions that affect them, while also holding governments and others accountable”. It pursues universal access to information as essential foundation of inclusive knowledge societies.

At the federal level we have FOIA (the Freedom of Information Act).

In New Hampshire we have RSA 91-a.

These are better than nothing, but inadequate.

Both of these require:

  1. that a citizen know what information they want
  2. that a citizen send in a request to the government
  3. that the citizen wait for the government (often there’s a ‘deadline’, but (a) there are no penalties when the government breaks the deadline, (b) the deadline can be satisfied by the government merely replying by the specified date “this may take up to three years”).
  4. that the citizen pay for reproduction costs

Further, the data

  1. does not come back in a machine-readable format
  2. is published, at most, once, and not repeatedly (e.g. annually, quarterly, weekly…)

Contrast this to the private sector where every 1-person e-commerce business, or 4-hour-per-week software side project has a dashboard.

A typical 1-person project has a dashboard that displays in real-time (update cycle of < 1 second) and shows things like

This has been state-of-the-art in the private sector for almost 30 years.

Now compare this to even a small responsive state like New Hampshire, where almost no data is proactively published, and what is published is

For an example of both problems, look at the NH Department of Education data for Weare, NH’s high school. The most recent data is seven years old, and is presented in crappy HTML format.

What sort of data do we want to see?

All of it!

Let’s reframe that. What data do you want to see?

Pick the issue that you care about:

Children

and further: how do the figures for 2023 compare to 2022, 2021, 2020, etc? Is NH DCYF getting better at getting children needed care? …or worse?

and further: how does NH DCYF’s numbers compare to, say, Vermont’s ? or Maine’s? Are NH children who have survived trauma seeing therapists in half the time that Vermont children do? Or twice the time? Are we paying twice as much per child processed?

Police and Crime

Colleges

For any given college in the state:

Taxes and lobbying expenses

For any given town in the state:

Judges

etc.

This list is just a start.

While the above list strives to be non-partisan, it’s inevitable that certain prejudices have crept in … but others can expand the list with whatever issues they care about. Number of wetland development projects approved. Number of garbage dumps created / closed. Conviction rates for crimes against women. Spending on homeless veterans. Whatever it is that citizens care about, governments and non-profits operating under government regulation almost certainly already collect the data, and just aren’t sharing it effectively.

What does getting this data achieve?

Government, like death, is an unfortunate fact of life.

If government exists, though, it is in our interest that it do the most good (or least harm) possible, for the lowest cost.

There is an efficient frontier in all things. We can have a government that spends $0 and prevents zero cases of a given bad thing. Or we can have a government that spends $1 million and prevents 10 cases of that bad thing. Both are defensible policies.

…but what’s indefensible is a government that spends $1 million and prevents 5, or 4, or 0 cases.

Activists of all stripes believe that government is inefficient, wasteful, and focused on the wrong problems.

The problem is that, without access to data, their appeals are emotional and anecdotal, instead of evidence based.

…and further,

“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.”

― Lord Kelvin

The Solution: Open Data

Before we can craft legislation that forces government to share data, we have to consider exactly what it is we’re looking for (Remember the old adage: “be careful what you wish for”).

Small tools, loosely coupled

Different citizens and groups will have different interests.

Each citizen group should be encouraged to build tools that do the analysis and reporting that they care about.

Again, the OGDI believes in The Unix Philosophy: small tools, loosely coupled. Make the government provide the data and the free market will do the rest! )

To support this low-effort modular approach to data import and digestion, we need to think about …

Data formats

Data formats vary from human readable (a JPG of a graph, or a PDF file of a budget) to the machine readable.

Machine readable data formats vary from the opaque (a binary file, or a spreadsheet with zero column headings) to somewhat transparent (a spreadsheet in proprietary Microsoft Excel format), to the very transparent (a CSV file with column headers, or an XML file with a DTD (Document Type Definition).

Data varies from the easily comprehensible overviews( the list of NH town tax rates), to the complete (the full 1,284 pages of the New Hampshire 2023 budget ).

There is a tension between the desire to get every scrap of data that’s available, on the one hand, and drowning users in minutia that may make it effectively impossible to make use of the data (can you look at the almost 1,300 page budget and - even using search in the PDF, find how much money goes to road building and maintenance?).

Additionally, there is the issue of …

Data format consistency over time

I recently did a small project importing NHLA spreadsheet data going back ~10 years into a Rails app, and because I could export each sheet as CSV, it was trivial to import … but even then, column headers changed from year to year, and I ended up doing a lot of special casing to, e.g., pull out canonical names which appear in 3 or 4 different formats over various years.

That’s from great, clean, simple data only going back a few years.

You can imagine the challenges in trying to parse out the DCYF component of the state budget over 20 years, especially as divisions get renamed, programs come and go, etc.

So we’d like data that is easy to compare across time (especially given that one bureaucratic trick is to change measurements, or change the definition of terms over time, to intentionally thwart measurability). This pushes us in the direction of wanting to carefully specify, in legislation, exactly what must be provided.

…however, this is all in tension with the ideal of …

Re-use of existing data whenever possible

The state collects absolute mountains of data. Every town has a budget generated in a spreadsheet. Every school system, likewise. DCYF generates reams of internal reports. Every police department shares data with the FBI.

The key is to, when possible, discover existing data sources that the government already creates internally, and then mandate sharing.

So … would we rather a priori, when crafting legislation, specify the 12 fixed things we want, each year, in a time sequence … or would we rather get massive data dumps?

I think the answer is “both”.

Existing data sources, laws, best practices, other resources

Overview of laws

Data formats

NH open data

List of data sets that might exist / might be of interest

Tools / existing data sources

Legislative Approach

legislation -> data -> evangelism -> legislation -> …

I have submitted bills over my first two-year term (2023-2024). None got beyond the committee stage.

I am optimistic that we can do better in the future.

I intend to submit one or more bills this term, but I see whatever bills we craft as the start of a long salami slicing tactic - we get one bill through this year, ideally, perhaps one that forces the state police to share data. Then we reach out to grad students in political science, government, economics, etc. and point them to this data source. We also reach out to friendly media and organizations (Reason magazine, NHLA, etc etc) We also loop in Dems and point out how the data supports some of their goals.

We document those successes and take another bite next year - getting DCYF to share information, maybe, or Department of Ed.

Then we keep circling back adding more and more data requirements, to get additional fields.

Simultaneously perhaps we write a few data visualization tools ourselves, and open source them. (Perhaps “write tools” overstates it; maybe we just write some blank spreadsheets that can import from the data sources, or write a 5 line bash file that invokes curl several times to download files, etc.) The point of this exercise is to create both a community (“evangelization”) and also an ecosystem of tools.

Goals over time

Near term

The near term goal is to get some data, any data, in New Hampshire available, as a camel nose under the tent.

If we could get bipartisan support (exposing some data that speaks to left-of-center concerns), that would be great.

Medium term

The medium term goal is

the Open Government Data Initiative can serve as a catalog of data analysis tools / streams. One can imagine a complete ecosystem where government data sources exist on the web, and certain analysis tools operate on them as streams / filters / webapps. Perhaps one activist writes a tool that takes in the full state budget and produces a much simpler, easy to digest

Long term

The long term goal is to spread the idea to other states. One tool to do this is to publish model legislation, and reach out to activists in other states via national level groups.

Once the idea spreads, we can imagine an ecosystem of tools that processes and displays data from multiple states. We should have dashboards that show the mean, best, and worst NH towns on dollars-spent-per-student-achievement, and compare that with data from Vermont and Maine. We should have dashboards that find demographically similar “peer” towns in various states and compare their total spending. We should have dashboards that compare policing effectiveness across towns and states.

Allied groups

I believe that the following groups are likely to be sympathetic to our goals, and we may have success talking to them, evangelizing them, and asking them for introductions to like-minded legislators.

The presence of a group on this list is not indicative of their endorsement!

How you can help

I’d love to hear your thoughts on any/all of this!

Next steps:

Historical / Log