Open Library logo
Together, let's build an Open Library for the World. Sponsor a Book
Last edited anonymously
September 25, 2019 | History

About the Technology


So we created Infobase, a new database framework that gives us this flexibility. Infobase stores a collection of objects, called "things". For example, on the Open Library site, each page, book, author, and user is a thing in the database. Each thing then has a series of arbitrary key-value pairs as properties. For example, a book thing may have the key "title" with the value "A Heartbreaking Work of Staggering Genius" and the key "genre" with the value "Memoir". Each collection of key-value pairs is stored as a version, along with the time it was saved and the person who saved it. This allows us to store full structured data, as well as travel back thru time to retrieve old versions of it.

Infobase is built on top of PostgreSQL, but its interface is abstract enough to allow it to be moved to other backends as performance requires. The current schema of Infobase tables looks like:

Table site
    name (string)
TABLE thing
  site_id (references site)
  key (string)
  [(site_id, key) combinations are unique]
TABLE version
  revision (int)
  thing_id (references thing)
  author_id (references thing)
  ip (ip address)
  comment (string)
  created (datetime)
  [(thing_id, revision) combinations are unique]
TABLE datum
  thing_id (references thing)
  begin_revision (int)
  end_revision (int)
  key (string)
  value (string)
  datatype ('string', 'reference', 'int', 'float', or 'date')
  ordering (integer, default null)

From Python, the infobase interface looks like this:

# retrieve the book object
foo = site.get('/foo')
assert foo.title == "The Story of Foo"
# query for books by that author
foos = site.things(dict(author="Joe Jacobson"))
assert foos[0].title == "The Story of Foo"

Infobase also has a programmable API, which can be used to build applications using the Open Library data.


Simply building a new database wasn't enough. We needed to build a new wiki to take advantage of it. So we built Infogami. Infogami is a cleaner, simpler wiki. But unlike other wikis, it has the flexibility to handle different classes of data. Most wikis only let you store unstructured pages -- big blocks of text. Infogami lets you store structured data, just like Infobase does, as well as use infobase's query powers to sort through it.

Each infogami page (that is, a "web" page; something with a URL) has an associated type. Each type contains a schema that states what fields can be used with it and what format those fields are in. Those are used to generate view and edit templates which can then be further customized as a particular type requires.

The result, as you can see on the Open Library site, is that one wiki contains pages that represent books, pages that represent authors, and pages that are simply wiki pages, each with their own distinct look and edit templates and set of data.


September 25, 2019 Edited by Edited without comment.
September 6, 2019 Edited by state (more explicitly) that an infogami _page_ is ... a "web" page.
June 3, 2019 Edited by Charles Horn reverted to revision 15
June 3, 2019 Edited by test
March 4, 2009 Created by webchick Creating .de /about/tech page