Click here to skip to this page's main content.

 Hello!   The State of Virginia is participating in our eBook lending program. Browse the growing lending library of over 200,000 eBooks!

Site Search

Site Search
More search options
Last edited by Tom Morris
March 17, 2013 | History

How to install your own copy of the Open Library

New install instructions

New install instructions are available at http://code.openlibrary.org/bootstrap.html


The instructions below are of historical interest only.

Install Script

This install script will install the OL system on a Linux system. It has been tested with a clean install of Ubuntu 10.04.

Dependencies

  • PostgreSQL 8.2 or later (psql)
  • Python 2.5 or later (python)
  • Git (git)
  • Java Runtime (tested with openjdk-6-jre)
  • At least 1GB of RAM on your dev machine or virtual machine

On Debian or Ubuntu run: 

$ sudo apt-get install postgresql git-core openjdk-6-jre-headless

Python Packages:

  • web.py (web) version 0.33 (Open Library doesn't work well with web.py version 0.34)
  • Babel (babel)
  • pyyaml (yaml)
  • psycopg2 (psycopg2) - Python driver for PostgreSQL
  • simplejson (simplejson)- Fast JSON codec for Python
  • python-memcached (memcache) - Python memcache client
  • lxml (lxml) - Library for processing XML
  • PIL (Image) - Python Imaging Library
  • pymarc (pymarc) - Python MARC library

Install them using easy_install (you must install pymarc using easy_install): 

$ sudo easy_install web.py==0.33 Babel pyyaml psycopg2 simplejson python-memcached lxml PIL pymarc genshi

(On Ubuntu, you must install the python-dev and the libpq-dev package before you can easy_install psycopg2, and you will need to install libxml2-dev and libxslt-dev in order to easy_install lxml)

Or using apt-get and dpkg: 

$ sudo apt-get install python-setuptools python-webpy python-pybabel python-simplejson python-yaml python-memcache python-psycopg2 python-lxml python-imaging python-genshi
$ wget http://home.us.archive.org/~edward/python-pymarc_2.31-1_all.deb
$ sudo dpkg -i python-pymarc_2.31-1_all.deb

Source Code

Checkout the source code. We recommend that you create a free account on GitHub and fork the openlibrary/openlibary project. Then you can checkout your fork and be able to push changes back to to your account on GitHub. There are guides for setting up a GitHub account and forking a project. 

$ git clone git://github.com/{your github username}/openlibrary.git

OR 

$ git clone git://github.com/openlibrary/openlibrary.git # clone the read-only openlibrary master

Run the openlibrary setup script. This will checkout other source code including infogami. 

$ cd openlibrary
$ ./setup.sh

Open Library Web Server

Create a local database. You should be able to run the "createdb" command from your user account. If not you will need to set up Postgres. See Installing Postgres on Ubuntu - Alternative Server Setup for a simple setup. (sudo -u postgres createuser -d -S -R username) 

$ createdb openlibrary

Get the configuration file and edit it to correct db username and solr settings in plugin_worksearch section. (*Open Library developers can look at /olsystem/etc/openlibrary.yml on any OL node for reference*) 

$ cp conf/sample_openlibrary.yml openlibrary.yml

Edit the database user in openlibrary.yml 

$ vi openlibrary.yml
# under db_parameters, change username from "anand" to whatever you used for the createdb step above.

Bootstrap. 

$ ./scripts/openlibrary-server openlibrary.yml install

Start the server and keep it running. 

$ ./scripts/openlibrary-server openlibrary.yml
http://0.0.0.0:8080/

To start the server on a different port: 

$ ./scripts/openlibrary-server openlibrary.yml startserver 9999
http://0.0.0.0:9999/

Create an account by following the Sign Up link on your dev instance. You will need to verify the account by looking in the console output of the web server for the verification link (open it in a web browser). In development mode messages that would be sent by email are printed to the web server output instead of actually being sent out.

Add "/people/yourname" to admin and api groups. 

http://0.0.0.0:8080/usergroup/admin?m=edit
http://0.0.0.0:8080/usergroup/api?m=edit

Copy templates, macros and some config from openlibrary.org website. 

$ ./scripts/copydocs.py --src http://openlibrary.org/ --dest http://0.0.0.0:8080/ /upstream/*
$ ./scripts/copydocs.py --src http://openlibrary.org/ --dest http://0.0.0.0:8080/ /config/edition

Restart the server.

In order to create additional user accounts you need to create the AccountBot user. The special AccountBot user will be logged as the "author" in the Open Library logs when new user accounts are created. 

Load http://0.0.0.0:8080/people/AccountBot?m=edit
Set the page type to /type/user
Save the page

Infobase Server

Create a database if it is not already created. 

$ createdb openlibrary

Get the configuration file and edit the db_parameters setting. 

$ cp conf/sample_infobase.yml infobase.yml

Create the directories. 

$ mkdir -p coverstore/localdisk

Start the server at port 7000. 

$ ./scripts/infobase-server infobase.yml 7000

Now, edit the openlibrary.yml and uncomment the infobase_server to make OL web server use this. Restart the OL web server and you should see some "200 OK" statuses printed in the output of the infobase server.

Coverstore Web Server

Create and initialize the coverstore database. 

$ createdb coverstore
$ psql coverstore < openlibrary/coverstore/schema.sql

Copy the configuration and edit the db_parameters section with your database username. 

$ cp conf/sample_coverstore.yml coverstore.yml

Run the server at port 8070. 

$ ./scripts/coverstore-server coverstore.yml 8070

You may need to correct the openlibrary.yml file if you are running coverstore on a different machine/port. Similarly, if you are running OL webserver on a different port, you need to correct it the coverstore.yml file. Note that the coverstore_url is used as the base for cover image URLs sent to the client's web browser, so you should use an externally resolvable host name or IP address if you want to be able to see covers from another machine.

Note: some URLs in the coverstore are still hardcoded to point to coverstore.openlibrary.org. Working on it (mang).

Solr Search Engine

Run the Solr setup script. This will download solr and put the stock files into vendor/apache/solr-1.4.0. The parts relevant to Open Library are copied into vendor/solr. The base config is in vendor/solr/solr/solr.xml. The setup script sets some default values. 

$ ./scripts/setup_solr.py

Start the works search engine: 

$ cd vendor/solr
$ java -jar start.jar

You should see a lot of output followed by these lines which shows what address (0.0.0.0 for localhost) and port the works search engine is listening on: 

  [...]
  INFO: SolrUpdateServlet.init() done
  2010-06-02 15:49:57.869::INFO:  Started SocketConnector @ 0.0.0.0:8983

You can verify that solr is running by accessing http://0.0.0.0:8983/solr You should see "Welcome to Solr!" 

$ curl -s http://0.0.0.0:8983/solr/ | head
<html>
<head>
<link rel="stylesheet" type="text/css" href="solr-admin.css">
<link rel="icon" href="favicon.ico" type="image/ico"></link>
<link rel="shortcut icon" href="favicon.ico" type="image/ico"></link>
<title>Welcome to Solr</title>
</head>

You can verify that queries can be processed: 

$ curl -s http://0.0.0.0:8983/solr/authors/select?q=foo
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">366</int><lst name="params"><str name="q">foo</str></lst></lst>
<result name="response" numFound="0" start="0"/>
</response>

Create ~/.olrc

The ~/.olrc file is used to store login credentials when running scripts (e.g. copydocs.py) The account should exist and have admin privileges.

Here is a sample .olrc using the builtin admin account (useable only on local machine) 

[0.0.0.0:8080]
username = admin
password = admin123

Start Solr update script

The Solr update script looks for changes in the Infobase server and writes them into the Solr search engine so, for example, a newly added book can be searched for. 

$ PYTHONPATH=$PYTHONPATH:. python scripts/solr_update.py --server=0.0.0.0:8080

Starting Open Library

When no Open Library services are running they should be started in this order:

Infobase: 

$ ./scripts/infobase-server infobase.yml 7000
loaded plugin openlibrary.plugins.ol_infobase
http://0.0.0.0:7000/

Cover Store: 

$ ./scripts/coverstore-server coverstore.yml 8070
http://0.0.0.0:8070/

Solr search engine: 

$ cd vendor/solr/
$ java -jar start.jar
[...]
INFO: SolrUpdateServlet.init() done
2010-06-08 14:59:17.761::INFO:  Started SocketConnector @ 0.0.0.0:8983

Solr update script (not currently working in git): 

$ PYTHONPATH=$PYTHONPATH:. python scripts/solr_update.py --server=0.0.0.0:8080

Open Library web server: 

$ ./scripts/openlibrary-server openlibrary.yml
http://0.0.0.0:8080/

Tips & Tricks

  • Copy docs from openlibrary.org to dev instance. 

    $ ./scripts/copydocs.py /books/OL1M --recursive

    If the OL webserver is running at a different port, then: 

    $ ./scripts/copydocs.py --dest http://0.0.0.0:12345 /books/OL1M --recursive

    To copy docs from dev instance to openlibrary.org: 

    $ ./scripts/copydocs.py --src http://0.0.0.0:8080 --dest http://openlibrary.org /sandbox

  • Looking at error debugging output

    If there is an error on a specific page you can see nice debugging output by adding &debug=true to the page URL. Errors are also stored on disk. In the web server output log you will see a line like this: 

    error saved to errors/2010-06-11/000850041215.html

  • Edit /upstream/templates/site/alert.tmpl to put a note that you're on a dev host at the top of every page.

  • To update the submodules in the vendor directory run the following command. Note: if you would like to modify the submodules please be brush up the use of submodules and the Pitfalls of Submodules. 

    $ ./setup.sh

  • If you would like to edit templates, macros and pages on disk instead of through the web UI you can disable the wikitemplates plugin by commenting out that line in openlibrary.yml. Now you can e.g. edit openlibrary/plugins/upstream/templates/site/alert.html to change the line displayed at the top of every page.

History Created April 11, 2008 · 75 revisions

March 17, 2013 Edited by Tom Morris Make it clear that the new instructions supersede
October 6, 2011 Edited by Anand Chitipothu Edited without comment.
January 31, 2011 Edited by Anand Chitipothu Edited without comment.
January 4, 2011 Edited by mangtronix Edited without comment.
April 11, 2008 Created by an anonymous user moving dev docs