![](\"http://invisible.net/openlibrary/ui_files//login.png\")
The Open Library interface is powered by infogami -- a cleaner, simpler alternative to other wiki applications. \r\nBut unlike other wikis, Infogami has the flexibility to handle different classes of data. Most wikis let you store unstructured pages -- big blocks of text. \r\nInfogami lets you store structured data.\r\n\r\n[1]\r\n
\r\n\r\nIn addition to this, infogami facilitates the creation of dynamic HTML templates and macros. This flexible environment enables users to create, \r\nshare and build collaborative interfaces. With Open Library in particular, we are focused on building a productive and vital community focused on the discovery of books.
\r\n\r\nIn this document, you'll learn how to develop for infogami, including building new templates for displaying your own data, running your own copy, and developing new features and plugins.
\r\n\r\nTo discuss this document, join the General Discussion mailing list.
\r\n\r\n\r\n\r\n\r\n
| \r\n\r\n
|
\r\nThis document describes the internal workings of the Open Library\r\nsoftware, from a developers' point of view. You should read it if you\r\nare:
\r\n\r\n1) a software developer wanting to add new features to Open Library. To\r\ndo this, you will also have to be a good Python programmer experienced\r\nin writing web server applications (not necessarily in Python). The\r\ndocument will explain the Open Library's software architecture and its\r\ninternal interfaces, and will explain how to write extensions (plugins),\r\nbut the sections about plugin writing will assume that you are familiar with Python and web programming\r\nin general.
\r\n\r\n If you do not yet know Python, you should first study the\r\n[Python documentation](http://www.python.org/doc/) or the free book\r\n[Dive into Python](http://diveintopython.org) . Python is an easy language to learn, but\r\nthe OL codebase is probably not understandable by complete beginners.\r\nFor web server principles, the somewhat dated [Philip and Alex's Guide to\r\nWeb Publishing](http://philip.greenspun.com/panda) is still an\r\ninformative read, though maybe someone can suggest something newer. You\r\nshould also understand the principles of software security --\r\nsee [David A Wheeler's page](http://www.dwheeler.com) for many documents.\r\n\r\n2) A user or web designer wanting to improve or customize the Open\r\nLibrary's user interface, either for yourself or for our whole\r\ncommunity. You will mainly want to study the section about template\r\nprogramming. You will need to know how to write HTML and it will help\r\nif you've done some server-side template programming (such as in PHP).\r\nIt will also help if you've had some exposure to Python, but the\r\nprogramming skills you'll need for template writing are generally less\r\nintense than they'd be for extension writing.
\r\n\r\n
3) A general user just wanting to know how the software works behind the\r\nscenes. You might not understand all the details, but reading the doc\r\nshould give you a general impression of how sites like this are put\r\ntogether.
\r\n\r\n4) A librarian or metadata maintainer wanting to process large volumes of\r\nmetadata for import into the Open Library. If you only want to import a few books, it's\r\nprobably easiest to use the web interface (or the Z39.50 interface once we have one). To\r\nimport bulk data, you'll have to process it into a format that Open Library can understand,\r\nwhich may require programming, but you can use your own choices of languages and\r\nplatforms for that purpose since you only have to create uploadable files, rather than\r\nuse language-specific interfaces. You'll mainly want to look at the section about data\r\nformats and schemas.\r\n\r\n
If you just want to be an OL user accessing or editing book data, you do\r\nNOT need to read this doc. The doc is about how to customize and extend\r\nthe software, not how to use it. As developers and designers, our goal\r\nis to make the site self-explanatory for users and not need much separate\r\ndocumentation, but we do have some user docs at (url's....).\r\n
\r\n\r\nInfogami is a wiki application framework built on web.py. Actual\r\napplications (like Open Library) are written by extending Infogami\r\nthrough two layers: plugins and templates. Plugins are Python modules\r\nthat get loaded into Infogami through a special API. They\r\nare invoked by submitting HTTP requests to the application, either HTML\r\nform posts or direct GET requests. Plugins can use any library or\r\napplication code that they wish, and they create Python objects to\r\nrepresent results, that then get expanded to HTML by templates.\r\nTemplates are a mixture of HTML text and user-written code,\r\napproximately in the spirit of PHP templates.\r\nThe user-written code is in a special-purpose scripting language that is\r\napproximately a Python subset, which runs in a hopefully-secure\r\nserver-side interpreter (embedded in the Python app) that has limited\r\naccess to system functions and resources.
\r\n\r\n\r\nThe lighttpd HTTP server runs Infogami through a FastCGI interface using Flup. (There can be multiple concurrent infogami instances that the lighttpd server distributes requests between, although Open Library currently just runs one.) Infogami is written\r\n in Python (Open Library currently requires 2.5 or greater) and uses web.py and ThingDB. ThingDB uses PostgreSQL as its data store. Psycopg2 is the Python driver for PostgreSQL. Open Library uses supervise (see also daemontools) to make sure everything keeps running. [2]\r\n
\r\n\r\nThe infogami application relies on various Web templates (these are code+HTML snippets). The initial templates are static files but they get edited through the wiki interface, and new ones get added through the wiki, so the real versions live entirely in the database. \r\n
\r\n\r\nThe goal of web.py is to build the ideal way to make web apps. web.py allows you to build HTTP responses, makes the database\r\neasier to use, and creates a templating evironment that tries to bring Python into HTML. [*]
\r\n\r\nEach infogami page (i.e. something with a URL) has an associated type. Each type contains a schema that states what fields can be used with it and what format those fields are in. Those are used to generate view and edit templates which can then be further customized as a particular type requires.
\r\n\r\n[ See State of the UI ] ... [3]
\r\n\r\n\r\nInfogami also accepts plug-ins and Open Library uses one for the Solr search engine. Solr is a JSP currently sitting in a Jetty http server, so it communicates with Infogami through a local http socket. Solr itself wraps the Lucene search library. These run under Java (Open Library currently uses Java 1.5). Solr is built under Apache Ant and has a few config and schema files, plus a startup script (solr.sh) that has to be manually edited to set the port number. I think we currently use Lucene as a downloaded .jar file so we don't build it.\r\n
\r\n\r\nThe solr-infogami plugin also calls out to an archive.org PHP script that expands basic search queries to advanced queries. It may also start using the openlibrary.org flipbook (with some possible customizations) to display OCA scans for pages containing fulltext search results.\r\n
\r\n\r\n\r\nOpen Library has a bunch of catalog data and fulltext acquired from various sources, either sitting in the Archive or to be uploaded to there. I think the acquisition processes (including web crawling scripts for some of the data) is outside the scope of an Open Library software install. There are a bunch of additional scripts to make the stuff usable in openlibrary and these need to be documented. These include TDB Conversion Scripts written by dbg, and (for OCA fulltext) Archive Spidering and Solr Importing scripts written by phr.\r\n
\r\n\r\n\r\nThere are two kinds of types in infogami. Primitive types and compound types. Primitive types are for representing integers, strings etc.
\r\n\r\nOther primitive and compound types can be added by creating a page in the wiki with type type/type.\r\n
\r\nSometimes you might want to link two different types. For example, book type has an author property and you want to add a books field to the author, which contains all books where author property of the book is this author. This is achieved by adding a backreference to the author type.\r\n
\r\n\r\nAvailable primitive types are:
\r\n\r\n\r\ntype/int
\r\ntype/boolean
\r\ntype/string - for single line text
\r\ntype/text - for multiline text
\r\n
Compound types specify what properties a thing of that type must have. Available compound types are:
\r\n\r\ntype/type
\r\n Type for representing all types. It defines the following properties ... \r\n
type/property
\r\n Type for representing properties in a type. It defines the following properties ...\r\n
type/backreference
\r\n later ...\r\n
type/page
\r\n Type for wiki pages. It defines the following properties ...\r\n
type/template
\r\n Type for template pages. It defines the following properties ...\r\n
Infogami uses templates to control the look and feel of the site. Custom templates can be provided in the wiki to enhance functionality or customize the look and feel of the site.
\r\n\r\nThere are two kinds of templates. Regular templates and type-specific templates. Type-specific templates are used to render a wiki page based on its type.
\r\n\r\nRegular templates must be placed at path templates/ and end with .tmpl extension. type-specific templates must be placed at type/ and end with .tmpl extension.
\r\n\r\nWith Open Library, only site administrators can modify the templates that reside in the templates/ and type/ directories. If users want to override these templates, they must create their own set of templates in their user directory, and change their template root in their user preferences. For more information, see Custom Templates.\r\n\r\n
There can be 4 type-specific templates for every type.\r\n
\r\nFor more information on how the repr and input templates are used, please see Useful Functions.
\r\n\r\nIt is not compulsory to specify all these templates for every type. When a type-specific template is not specified for a type, a default template is used. The following are the default templates:\r\n
\r\nThere are many other interesting regular templates. Some of them are:\r\n
\r\n\r\n\r\nA user can create their own set of templates by creating a subdirectory in the 'user' directory ('user/[username]'). \r\nA user can create a new template for each set of templates they want to override. If a template does not exist in a user's subdirectory, \r\nthen the default set of templates will be used. Templates are created in the same manner as any new page is created in a wiki.\r\n
\r\n\r\nFor purposes of illustration, we will view the contents of ('user/brewster/*')\r\n
To view the Open Library using a set of custom templates, you must change your template preferences:
\r\n\r\nNOTE: Changing the template root to 'user/brewster' will load the templates used in this example.
\r\n\r\nIt is also possible to create custom Cascading Style Sheet (CSS) files and JavaScript files in the wiki. \r\n\r\n
To create a custom CSS file in the wiki, \r\ncreate a page in your user directory entitled 'user/[username]/[filename].css' and select 'rawtext' for the pagetype. Once the CSS file has been successfully created,\r\n it can be linked to the template using the '$add_stylesheet' command:
\r\n\r\n![in this example.](\"http://invisible.net/openlibrary/ui_files//brewster.skin.png?keepThis=true&TB_iframe=true&height=599&width=800\"){kind=link}
\r\n$add_stylesheet('/static/css/search.css')\r\nJavaScript files can be created in the same manner, using a '.js' extension and the '$add_javascript' command:
\r\n\r\n\r\n$add_javascript('/static/js/randomsearch.js')\r\nIt is important to note that when a page with the type 'rawtext' is added to the wiki, the editing function is not readily apparent. \r\nTo edit a rawtext page, simply add '?m=edit' to the url.
\r\n\r\nExample:\r\n
There are some useful functions, which are very handy in writing view and edit templates.\r\n
\r\n\r\nLets take an example. Suppose there are many types, which need to display an image. Probably they will do that by defining a type type/image for representing images. Instead of worrying how to display and edit (upload) images, these types can just use thingrepr and thinginput macros and define repr and input templates for type/image. Later in future, if you want to change how to display/upload image, you don't have to change in multiple places.
There are some global functions, which are accessible by every template. For more information see Infogami API > Decorators.\r\n
\r\n\r\n\r\n\r\n\r\nMacros are like functions in any programming language.\r\n Macros look very much like templates, but they can be called from wikipages and other templates and macros. \r\n Macros can be written in the wiki just like templates. \r\n Macros must be placed in macros/ path and type must be type/macro. As a convention, they are named in CamelCase. \r\n
Macros introduce an element of modularity when designing web templates in infogami. A common practice is to use macros for any repeatable design elements. This can be as simple as an HTML callout box, or a script. For instance, the \"SAVE\", \"PREVIEW\", and \"DELETE\" buttons on every edit form on the Open Library site are generated by the \"EditButtons\" Macro code below:\r\n\r\n
\r\n<p><strong>Edit summary</strong> (briefly describe the changes you have made):
\r\n<input type=\"text\" name=\"_comment\" style=\"width: 100%\" value=\"\" />
\r\n</p>
\r\n<p>
\r\n<button class=\"button\" type=\"submit\" value=\"$_.SAVE\" name=\"_save\" title=\"Save\"><img src=\"/static/images/save.png\" alt=\"Save\" /></button>
\r\n<button class=\"button\" type=\"submit\" value=\"$_.PREVIEW\" name=\"_preview\" title=\"Preview\"><img src=\"/static/images/preview.png\" alt=\"Preview\" /></button>
\r\n<button class=\"button\" type=\"submit\" value=\"$_.DELETE\" name=\"_delete\" title=\"Delete\"><img src=\"/static/images/delete.png\" alt=\"delete\" /></button>
\r\n\r\n
Intelligent use of macros results in clean, readable code that can be easily understood and repurposed by other users. For instance, the author view template is a good example of a template with a highly modular and streamlined design. View Page Using Template\r\n\r\n\r\n
Macros can be invoked either through the wiki interface when editing a page in Markdown, or as a script when building web templates.\r\n\r\n
The syntax to call a macro from a wiki page is:\r\n
\r\n{{HelloWorld()}} or {{MyMacro("arg1", "arg2")}}
\r\n\r\nThe syntax to call a macro from templates and macros is:\r\n
\r\n$:macros.HelloWorld() and $:macros.MyMacro("arg1", "arg2")
\r\nThe edit and view templates are defined by two lines of code that must be at the \r\ntop of the template file in order for them to function properly (The absence of either \r\nof these two lines will generate an error). For instance, this example is from the \r\ndefault edition template:
\r\n\r\n\r\n \r\n\t$def with (page)
\r\n
\r\n\t$var title: $page.title\r\nAdditional variables can be controlled in these commands. For instance, in an edit \r\ntemplate, the preview function can be disabled as follows:
\r\n\r\n\r\n \r\n\t$def with (page, preview=False )\r\nDisplay of the <TITLE> of the page can be constructed by making a call to a universal \r\nlanguage file. For instance:
\r\n\r\n\r\n \r\n\t$var title: $_.EDIT $page.title\r\nwould display \"edit Tom Sawyer\" in the edit view of an edition page by accessing the verbiage in the language file for '$_.EDIT' (edit)\r\n and the 'title' key in the 'edition' type [ See http://www.openlibrary.org/type/edition/].\r\n\r\n
and:
\r\n\r\n\r\n \r\n\t$var title: $_.VIEW $page.d.get('name')\r\nwould display \"View Mark Twain\" when viewing an author page by accessing the 'name' \r\nattribute in the 'author' type [See http://www.openlibrary.org/type/author/].\r\n\r\n
The page 'get' statements make a call to the document type to pull the corresponding \r\ninformation for display on the page. For instance the following table illustrates our \r\ncurrent author type:
\r\n\r\n![\"Name\"](\"/static/images/name.schema.png\") | \r\n ![\"Type\"](\"/static/images/type.schema.png\") | \r\n
| description | \r\n\r\n | \r\n
| is_primitive | \r\n\r\n\r\n | \r\n
| properties | \r\nwebsite of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\nbio of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\nphotograph of type type/image\r\n\r\n\r\n | \r\n
| properties | \r\nname of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\ndeath_date of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\nalt_names of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\nlocation of type type/string\r\n\r\n\r\n | \r\n
| properties | \r\nbirth_date of type type/string\r\n\r\n\r\n | \r\n
| backreferences | \r\nBack-reference to authors property of type/edition\r\n\r\n\r\n | \r\n
Therefore using this command:
\r\n\r\n \r\n\t$page.d.get('name')\r\non the view template will display the author name wherever used on an author template.
\r\n\r\n'If' statements are used in much the same manner as in other scripting languages, \r\ninvoking an action if a certain state exists. For instance, if page preview is activated in an edit template as follows:
\r\n \r\n \r\nif $def with (page, preview=True ), \r\nthe following command would format the HTML contents of the page \r\nwherever it is inserted in the edit template:
\r\n\r\n \r\n$if preview:
\r\n
\r\n <p>$:format(page.body)</p>\r\nThe 'if' statement can also be used in conjunction with the 'get' statement to hide \r\nempty data fields on a page. For instance:
\r\n\r\n \r\n\t$if page.d.get('language'):
Language: $page.language\r\nwould display the language a book was published in if that data is available for that \r\nbook, or:
\r\n\r\n \r\n\t$if page.d.get('birth_date'): $page.d.get('birth_date') -
\r\n\t$if page.d.get('death_date'): $page.d.get('death_date')\r\nwould display the date of birth and date of death for a given author on the author \r\nview template if they exist.
\r\n\r\n\r\nFunctionality of infogami can be extended by adding new plugins to it. \r\n Plugin in infogami is a python package with the following directory structure.\r\n
\r\nsee sample_run.py in infogami repository to find out how to add a plugin to infogami.\r\n
A plugin can add special pages to infogami and add new modes to wiki pages. \r\n Special pages are paths which are treated differently than regular wiki pages. login, logout, sitepreferences etc. are some examples of special pages. Modes are actions that be done on a wiki page. view, edit, history, diff etc. are some examples of wiki modes.\r\n
\r\ninfogami.utils.delegate module provides two special classes page and mode. Any class that extends from page becomes a special page with path same as the class name and any class that extends from mode becomes a wiki mode with name of the mode same as the class name.\r\n
# hello/code.py\r\n class hellopage(delegate.page)\r\n def GET(self, site)\r\n return "hello page!"\r\n\r\n class hello(delegate.mode)\r\n def GET(self, site, path)\r\n return "hello mode for " + path\r\nA plugin can also add new actions to the system. Action is a utility function, which can be invoked from the command line as shown in the following example.\r\n
$ python run.py action_name arg1 arg2
\r\n
\r\n$ python run.py startserver 9000
\r\n$ python run.py help
\r\n$ python run.py movetemplates
\r\nThe default action is startserver and that is invoked when no action is specified. \r\n
To convert a function in to an action, it must decorated with infogami.action decorator. The following is an example of action.\r\n\r\n
@infogami.action
\r\ndef hello(name="world"):
\r\n print "hello, " + name + "!"
\r\nOnce this plugin is added to infogami, this action can be used.\r\n
\r\n$ python run.py hello
\r\nhello, world!
\r\n$ python run.py infogami
\r\nhello, infogami!
\r\nThe help action prints the list of all available actions.\r\n
There are some important actions, that every plugin author must know. \r\n
\r\ninstall: The install action runs all the install hooks. Install hook is a function without any arguments decorated by infogami.install_hook. Any plugin can add a new install hook to do some actions to initialize an infogami instance. Some of the actions are install hooks too.\r\n
movefiles: The movefiles action moves all the files in $plugin/files/ from every plugin to static/ directory. This is also an install hook. If a plugin needs to access any static resources, they must be put in files/ directory of that plugin. Keeping resources directly in static/ is not a good idea and should be discouraged.\r\n\r\n
movetemplates: The movetemplates action moves templates from every plugin to wiki. Optional prefix can be specified to limit the templates that are moved into the wiki. This is also an install hook.\r\n
To make a function accessible in templates, that function must be decorated with infogami.utils.view.public decorator. \r\n Since templates can be written by untrusted users, utmost care must be taken when exposing a function in templates.\r\n
The basic flow is: the user submits a url or HTML form; the FCGI handler\r\nmatches the URL patch against a series of regexps to dispatch the\r\nrequest to the proper plugin; the plugin receives the GET or POST data\r\nand creates results as Python objects (e.g. a list of strings or ints);\r\nthe plugin then loads a template (designated in the plugin code) which\r\nreceives the result objects as arguments. The template code then\r\nexpands the result to HTML which gets presented to the user's browser.\r\nThus, a user can \"re-skin\" the application by modifying the static HTML\r\nor the script code in the templates that he or she uses. A few initial\r\ntemplates (such as the application's top-level HTML page) are included\r\nin the application source tree, and the rest are created and edited\r\nthrough the wiki interface.
\r\n\r\nPlugins are written by the application developers and are part of the\r\nsource code tree for the application. Templates are entered and updated\r\nthrough the wiki HTTP interface, are stored in the database just like\r\nother wiki pages, and can be written by arbitary users. Thus, templates\r\nare potentially hostile and therefore require security precautions\r\nduring execution. Templates give a flexible means of creating\r\npresentation HTML, but for security reasons, operations requiring access\r\nto system functions must be implemented in plugins, not templates.\r\nTemplates can be application-wide or can be specific to a given user\r\naccount.
\r\n\r\n\r\n\r\n\r\n\r\n\r\nQ. I can't change my page type when I create a new page in the wiki using my new \r\n template set. Whenever I change the page type (to 'edition' instead of 'page', for \r\n instance), it reverts back to the original type.\r\n
\r\nA. Make sure your edit template contains the box for 'page type'. It is possible that \r\n it was removed when the form was designed. The absence of this box means that the \r\n template loses the type information whenever it is saved.\r\n
\r\nQ. I am trying to use custom graphics using the type for my particular page template \r\n set, however it generates this error: error in processing template: AttributeError: \r\n 'str' object has no attribute 'png' (falling back to default template). What gives?\r\n
\r\nA. Aha. You have to say ${key}.png because otherwise it thinks you want the png field \r\n of the key variable (as in $page.doi).\r\n
\r\n\r\nQ. Why doesn't the HTML formatting I enter in my edit template show up correctly on my \r\n view template? I added carraige returns and page breaks in all the appropriate places, \r\n but it runs together in one long paragraph.\r\n
\r\nA. You need to use the '$:format' command to support HTML formatting. For instance, to \r\n properly format the appearance of a biography on the author view page you would need \r\n to add the following the script in the appropriate place on the page:\r\n
\r\n$:format(page.bio)\r\n
\r\nQ. My HTML code for any images I add via a page edit form displays on the view page. \r\n For example, instead of a photo of Dave Eggers, I see "<img src="http://ia350629.us.archive.org/3/items/ol-images/daveeggers.gif" align="left">"\r\n\r\n
\r\nA. Again, you need to use:\r\n
\r\n$:format(page.bio)\r\n instead of\r\n $page.bio\r\n
\r\nif you want to support HTML formatting.\r\n
\r\nQ. Help! My edit template for [this page] is royally hosed. I was trying to do \r\n something fancy with [form layout | javascript | etc. ] and broke the edit template so \r\n badly that I can't get back in and correct it.\r\n
\r\nA. Use the emergency override to fall back to a very simplified template version and \r\n see if you can make your corrections there: \r\n
\r\n[url]?m=edit&rescue=true\r\n
\r\n