đŸȘŽ Anil's Garden

GH Archive

02 Apr 20257 min read

GH Archive

Excerpt

GH Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis.

Open-source developers all over the world are working on millions of projects: writing code & documentation, fixing & submitting bugs, and so forth. GH Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis.

GitHub provides 15+ event types, which range from new commits and fork events, to opening new tickets, commenting, and adding members to a project. These events are aggregated into hourly archives, which you can access with any HTTP client:

QueryCommand
Activity for 1/1/2015 @ 3PM UTCwget https://data.gharchive.org/2015-01-01-15.json.gz
Activity for 1/1/2015wget https://data.gharchive.org/2015-01-01-{0..23}.json.gz
Activity for all of January 2015wget https://data.gharchive.org/2015-01-{01..31}-{0..23}.json.gz

Each archive contains JSON encoded events as reported by the GitHub API. You can download the raw data and apply own processing to it - e.g. write a custom aggregation script, import it into a database, and so on! An example Ruby script to download and iterate over a single archive:

require 'open-uri'
require 'zlib'
require 'yajl'
gz = open('http://data.gharchive.org/2015-01-01-12.json.gz')
js = Zlib::GzipReader.new(gz).read
Yajl::Parser.parse(js) do |event|
print event
end
  • Activity archives are available starting 2/12/2011.
  • Activity archives for dates between 2/12/2011-12/31/2014 was recorded from the (now deprecated) Timeline API.
  • Activity archives for dates starting 1/1/2015 is recorded from the Events API.

For the curious, check out The Changelog episode #144 for an in-depth interview about the history of GH Archive, integration with BigQuery, where the project is heading, and more.

Analyzing event data with BigQuery

The entire GH Archive is also available as a public dataset on Google BigQuery: the dataset is automatically updated every hour and enables you to run arbitrary SQL-like queries over the entire dataset in seconds. To get started:

  1. If you don’t already have a Google project

  1. Go to BigQuery, and select your newly created project from the dropdown in the header bar.
  2. Execute your first query against the public “githubarchive” dataset. You can just copy and paste the query below and run, once you’ve selected your project. You can also look through the public dataset itself, but you will not have permission to execute queries on behalf of the project.
/* count of issues opened, closed, and reopened on 2019/01/01 */
SELECT event as issue_status, COUNT(*) as cnt FROM (
SELECT type, repo.name, actor.login,
JSON_EXTRACT(payload, '$.action') as event,
FROM `githubarchive.day.20190101`
WHERE type = 'IssuesEvent'
)
GROUP by issue_status;

For convenience, note that there are multiple tables that you can use for your analysis:

  1. year dataset: [2011](https://bigquery.cloud.google.com/table/githubarchive:year.2011), [2012](https://bigquery.cloud.google.com/table/githubarchive:year.2012), [2013](https://bigquery.cloud.google.com/table/githubarchive:year.2013), [2014](https://bigquery.cloud.google.com/table/githubarchive:year.2014), and [2015](https://bigquery.cloud.google.com/table/githubarchive:year.2015) tables contain all activities for each respective year.
  2. month dataset: contains activity for each respective month - e.g. [201501](https://bigquery.cloud.google.com/table/githubarchive:month.201501).
  3. day dataset: contains activity for each day - e.g. [20150101](https://bigquery.cloud.google.com/table/githubarchive:day.20150101).

The schema of above datasets contains distinct columns for common activity fields (see same response format), a "payload" string field which contains the JSON encoded activity description, and "other" string field containing all other fields.

  • The content of the "payload" field is different for each event type and may be updated by GitHub at any point, hence it is kept as a serialized JSON string value in BigQuery. Use the provided JSON functions (e.g. see query example above with JSON_EXTRACT()) to extract and access data in this field.
  • The content of the "other" field is a JSON string which contains all other data provided but GitHub that does not match the predefined BigQuery schema - e.g. if GitHub adds a new field, it will show up in “other” until and unless the schema is extended to support it.

Note that you get 1 TB of data processed per month free of charge. In order to make best use of it, you can restrict your queries to relevant time ranges to minimize the amount of scanned data. To scan multiple tables at once, you can use table wildcards:

/* count number of pushes between Jan 1 and Jan 5 */
SELECT
COUNT(*)
FROM `githubarchive.day.2015*`
WHERE
type = 'PushEvent'
AND (_TABLE_SUFFIX BETWEEN '0101' AND '0105')
/* count number of watches between Jan~Oct 2014 */
SELECT COUNT(*)
FROM `githubarchive.month.2014*`
WHERE
type = 'WatchEvent'
AND (_TABLE_SUFFIX BETWEEN '01' AND '10')
*/
/* count number of forks in 2012~2014 */
SELECT COUNT(*)
FROM `githubarchive.year.20*`
WHERE
type = 'ForkEvent'
AND (_TABLE_SUFFIX BETWEEN '12' AND '14')

Daily reports

Changelog Nightly is the new and improved version of the daily email reports powered by the GH Archive data. These reports ship each day at 10pm CT and unearth the hottest new repos on GitHub. Alternatively, if you want something curated and less frequent, subscribe to Changelog Weekly.

Research, visualizations, talks


Have a cool project that should be on this list? Send a pull request!

Graph View

Backlinks

  • No backlinks found