Open Data 101:
Web Development in the Era of Open Data
Matthew Schwartz, Todd Davis
EPA Internet Service Center - CSC
What is "Data"?
In this context, data refers to information that is meant to be handled by other computers rather than directly read by people.
- Structured
- Machine Readable
- Raw - not already interpreted
Very roughly: think databases not PDFs, spreadsheets not white papers
What is "Open"?
Data becomes "open" when it is published in a way that makes it accessible to all consumers.
- Posted in publicly accessible locations
- Established, fee-free, openly defined formats
- Distributed with terms allowing both the use and re-purposing of the data
What is Open Data not?
As with any set of "buzz words" there are a lot of interpretations of what fits into a category.
When we use the term "open data" we aren't referring to:
- Widgets / Gadgets
- Web pages or applications
- Any tools constructed to display data
These are all tools built with open data.
Open Data Glossary: XML, CSV, JSON, etc
Text files used to structure, store and transport data.
- Designed to carry data, not display it.
- Not computer programs - just plain text files meant to be used by programs.
- Open standards exist for all these formats.
Open Data Glossary: KML / Geodata
Same idea as XML - text files transporting information.
Used specifically for Geodata.
Open Data Glossary: APIs
"Application Programming Interface" - a method for computer programs to talk to each other.
In Open Data, APIs are a way for programs to request certain chunks of information over the web rather than the whole data set.
Distributor keeps the data on their computers and provides a "drive through window" for other people to grab small chunks through.
Most effective when dealing with very large or rapidly changing datasets.
- Google Maps
- Flickr images
- Twitter posts
Open Data Glossary: APIs
Visit a specially formatted web address:
Receive a machine formatted response:
Raw Data, Not Forms
If the contents of USAjobs was offered also as "open" data the public could potentialy create a variety of new and interesting uses for this data.
Not only can the public create new uses for the data it is also portable.
Open Data is:
- Raw
- Meant to be read by machines
- In an open format
- Published with permission to use and reuse
Why Provide Open Data?
- Government as Platform
- External Contribution
- Encourage Experimentation
- Efficiency
Government as Platform
IT "platforms" are products that allow others to build on their success. They create opportunities.
The biggest success stories in the IT world often involve products becoming "platforms."
- Microsoft "put a PC on every desk.": Millions use their product to more efficiently run businesses, enhance education, and manage their households.
- Google: creates tools that allow people and busniesses to get more out of the web
- Apple: leads the way in re-inventing mobile computing and provides the leading development marketplace with their App Store
Government as Platform
How can opening government data build a platform?
- Data is "raw material" for private sector innovation: easy access to raw data founds companies, fuels research, and inspires education.
- Allows the results of core government products to be "re-purposed" by the public. (ie: GPS)
- Open data is a catalyst: standards compliant, discoverable data provides tools for innovation.
External Contribution
Stakeholders have the most to gain and the most to offer in return
- Research Institutions
- Regulated Industries
- State, Local, Tribal governments
- Advocacy groups / NGOs
Encourage Experimentation
It is hard for the government to be innovative on the web
- Limited budget / Limited People-power
- Regulatory burdens can slow down innovation
These barriers can be lower for the private sector. However they lack easy access to the data that could be used to innovate.
Opening up the raw data that many government sites are based upon allows the public to build supplemental and competing uses for the government's data.
Encourage Experimentation
Enabling the public to utilize open data in its innovations is good for the government as well.
- Learn from well received projects to see what the public is interested in or has a need for.
- Identifies what doesn't work or is problematic.
- Adopt innovations that are successful.
All of this means government web sites aren't saddled with finding "the one true way" right out of the gate.
Efficiency
Data tucked away in open formats doesn't "age" in the same way web sites do. No re-designs, re-templating, or need to be moved to up-to-date platforms.
Assist in the "blending" of data across agency boundaries. Federal to Federal or Federal to State/Local/Tribal
Increase in "visibility": Innovators can expose groups who might not have been the original intended target of the data.
Marketing tool: Attention is drawn to the source of the data as it is discovered and used by the public.
Examples
Data.gov
http://www.data.gov/
"The purpose of Data.gov is to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. Although the initial launch of Data.gov provides a limited portion of the rich variety of Federal datasets presently available, we invite you to actively participate in shaping the future of Data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of your Federal data."
Examples: Data.gov and Apps for America
Apps for America is a project by the Sunlight Foundation. Sunlight is a non-profit group that advocates transparency in Congress.
- Web application building contest awarding over $21,000 to developers that build open source applications that use government data.
- In this year's contest developers will use datasets from Data.gov to build their web applications.
- Last year's contest used Sunlight's own data sets based on Congress. Winners included:
Examples: Jobless Rate
The Jobless Rate for People Like You is a data visualization tool built by the New York Times.
Using raw data from the Bureau of Labor Statistics this tool breaks out government data to be more accessible to lay people. It is also a great example of the public (private sector) coming up with innovations that could be incorporated back into government web sites.
