- official name

- remove the switch readme

LICENSE

@@ -1,7 +1,7 @@
 This is the MIT license: http://www.opensource.org/licenses/mit-license.php
 
-Copyright (c) 2005, 2006, 2007, 2008, 2009 Michael Bayer and contributors. SQLAlchemy is a trademark of Michael
-Bayer.
+Copyright (c) 2005, 2006, 2007, 2008, 2009, 2010 Michael Bayer and contributors. 
+SQLAlchemy is a trademark of Michael Bayer.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software

MANIFEST.in

@@ -1,2 +1,2 @@
-recursive-include doc *.html *.css *.txt *.js
-prune doc/build/output
+recursive-include doc *.html *.css *.txt *.js *.jpg
+prune doc/build/output

README.py3k

@@ -0,0 +1,45 @@
+=================
+PYTHON 3 SUPPORT
+=================
+
+Current Python 3k support in SQLAlchemy is provided by a customized
+2to3 script which wraps Python's 2to3 tool.
+
+This document will refer to the Python 2.6 interpreter binary as
+"python26" and the Python 3.xx interpreter binary as "python3".
+
+To build the Python 3K version, use the Python 2.6 interpreter to
+run the 2to3 script on the lib/ directory, and optionally the test/
+directory.   The -w flag indicates that the new files should be
+written.
+
+    python26 sa2to3.py ./lib/ ./test/ -w
+
+You now have a Python 3 version of SQLAlchemy in lib/.   
+
+Current 3k Issues
+-----------------
+
+Current bugs and tickets related to Py3k are on the Py3k milestone in trac:
+
+http://www.sqlalchemy.org/trac/query?status=new&status=assigned&status=reopened&milestone=py3k
+
+Running Tests
+-------------
+
+The unit test runner, described in README.unittests, is built on
+Nose, and uses a plugin that is ordinarily installed using setuptools
+entry points.  At the time of this writing setuptools isn't available 
+for Python 3 although the "Distribute" project does seem to provide support.
+Additionally, Nose itself is only available in an old version for Python 3,
+which is available at http://bitbucket.org/jpellerin/nose3/ .
+
+To run the unit tests using the old version of nose and without the usage of 
+setuptools, use the "sqla_nose.py" script:
+
+    python3 sqla_nose.py
+    
+When running with Python 3, lots of debug output is dumped to the console.
+This is due to hacking around the old version of Nose to support the 
+SQLAlchemy test plugin without setuptools (details at 
+http://groups.google.com/group/nose-dev/browse_thread/thread/c6a25531baaa2531).

README.unittests

@@ -10,10 +10,19 @@ downloads for nose are available at:
 
 http://somethingaboutorange.com/mrl/projects/nose/0.11.1/index.html
 
-
 SQLAlchemy implements a nose plugin that must be present when tests are run.
 This plugin is available when SQLAlchemy is installed via setuptools.
 
+INSTANT TEST RUNNER
+-------------------
+
+A plain vanilla run of all tests using sqlite can be run via setup.py:
+
+    $ python setup.py test
+    
+Setuptools will take care of the rest !   To run nose directly and have
+its full set of options available, read on...
+
 SETUP
 -----
 
@@ -41,6 +50,11 @@ intersesting:
 
 RUNNING INDIVIDUAL TESTS
 -------------------------
+Any directory of test modules can be run at once by specifying the directory
+path:
+
+    $ nosetest test/dialect
+
 Any test module can be run directly by specifying its module name:
 
     $ nosetests test.orm.test_mapper
@@ -67,11 +81,61 @@ DATABASE TARGETS
 Tests will target an in-memory SQLite database by default.  To test against
 another database, use the --dburi option with any standard SQLAlchemy URL:
 
-    --dburi=postgres://user:password@localhost/test
+    --dburi=postgresql://user:password@localhost/test
 
-Use an empty database and a database user with general DBA privileges.  The
-test suite will be creating and dropping many tables and other DDL, and
-preexisting tables will interfere with the tests
+Use an empty database and a database user with general DBA privileges.  
+The test suite will be creating and dropping many tables and other DDL, and
+preexisting tables will interfere with the tests.
+
+Several tests require alternate schemas to be present.   This requirement
+applies to all backends except SQLite and Firebird.   These schemas are:
+
+    test_schema
+    test_schema_2
+
+Please refer to your vendor documentation for the proper syntax to create 
+these schemas - the database user must have permission to create and drop
+tables within these schemas.  Its perfectly fine to run the test suite
+without these schemas present, it only means that a handful of tests which
+expect them to be present will fail.
+
+Additional steps specific to individual databases are as follows:
+
+    ORACLE: the test_schema and test_schema_2 schemas are created as
+    users, as the "owner" in Oracle is considered like a "schema" in
+    SQLAlchemy.
+    
+     The primary database user needs to be able to create and drop tables,
+    synonyms, and constraints in these schemas. Unfortunately, many hours of
+    googling and experimentation cannot find a GRANT option that allows the
+    primary user the "REFERENCES" role in a remote schema for tables not yet
+    defined (REFERENCES is per-table) - the only thing that works is to put
+    the user in the "DBA" role:
+    
+     grant dba to scott;
+    
+     Any ideas on what specific privileges within "DBA" allow an open-ended
+    REFERENCES grant would be appreciated, or if in fact "DBA" has some kind
+    of "magic" flag not accessible otherwise. So, running SQLA tests on oracle
+    requires access to a completely open Oracle database - Oracle XE is
+    obviously a terrific choice since its just a local engine. As always,
+    leaving the schemas out means those few dozen tests will fail and is
+    otherwise harmless. 
+    
+
+    MSSQL: Tests that involve multiple connections require Snapshot Isolation
+    ability implented on the test database in order to prevent deadlocks that
+    will occur with record locking isolation. This feature is only available
+    with MSSQL 2005 and greater. You must enable snapshot isolation at the
+    database level and set the default cursor isolation with two SQL commands:
+    
+     ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON
+    
+     ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON
+
+    MSSQL+zxJDBC: Trying to run the unit tests on Windows against SQL Server
+    requires using a test.cfg configuration file as the cmd.exe shell won't
+    properly pass the URL arguments into the nose test runner.
 
 If you'll be running the tests frequently, database aliases can save a lot of
 typing.  The --dbs option lists the built-in aliases and their matching URLs:
@@ -80,24 +144,23 @@ typing.  The --dbs option lists the built-in aliases and their matching URLs:
     Available --db options (use --dburi to override)
                mysql    mysql://scott:tiger@127.0.0.1:3306/test
               oracle    oracle://scott:tiger@127.0.0.1:1521
-            postgres    postgres://scott:tiger@127.0.0.1:5432/test
+            postgresql    postgresql://scott:tiger@127.0.0.1:5432/test
     [...]
 
 To run tests against an aliased database:
 
-    $ nosetests --db=postgres
+    $ nosetests --db=postgresql
 
 To customize the URLs with your own users or hostnames, make a simple .ini
 file called `test.cfg` at the top level of the SQLAlchemy source distribution
 or a `.satest.cfg` in your home directory:
 
     [db]
-    postgres=postgres://myuser:mypass@localhost/mydb
+    postgresql=postgresql://myuser:mypass@localhost/mydb
 
 Your custom entries will override the defaults and you'll see them reflected
 in the output of --dbs.
 
-
 CONFIGURING LOGGING
 -------------------
 SQLAlchemy logs its activity and debugging through Python's logging package.
@@ -157,23 +220,3 @@ always possible.  If you hit a wall, join us on the mailing list or, better,
 IRC!
 
 
-TIPS
-----
-PostgreSQL: The tests require an 'alt_schema' and 'alt_schema_2' to be present in
-the testing database.
-
-PostgreSQL: When running the tests on postgres, postgres can get slower and
-slower each time you run the tests.  This seems to be related to the constant
-creation/dropping of tables.  Running a "VACUUM FULL" on the database will
-speed it up again.
-
-MSSQL: Tests that involve multiple connections require Snapshot Isolation
-ability implented on the test database in order to prevent deadlocks that will
-occur with record locking isolation. This feature is only available with
-MSSQL 2005 and greater. For example::
-
-    ALTER DATABASE MyDatabase
-    SET ALLOW_SNAPSHOT_ISOLATION ON
-    
-    ALTER DATABASE MyDatabase
-    SET READ_COMMITTED_SNAPSHOT ON

convert.py

@@ -1,228 +0,0 @@
-import os
-import subprocess
-import re
-
-def walk():
-    for root, dirs, files in os.walk("./test/"):
-        if root.endswith("/perf"):
-            continue
-        
-        for fname in files:
-            if not fname.endswith(".py"):
-                continue
-            if fname == "alltests.py":
-                subprocess.call(["svn", "remove", os.path.join(root, fname)])
-            elif fname.startswith("_") or fname == "__init__.py" or fname == "pickleable.py":
-                convert(os.path.join(root, fname))
-            elif not fname.startswith("test_"):
-                if os.path.exists(os.path.join(root, "test_" + fname)):
-                    os.unlink(os.path.join(root, "test_" + fname))
-                subprocess.call(["svn", "rename", os.path.join(root, fname), os.path.join(root, "test_" + fname)])
-                convert(os.path.join(root, "test_" + fname))
-
-
-def convert(fname):
-    lines = list(file(fname))
-    replaced = []
-    flags = {}
-    
-    while lines:
-        for reg, handler in handlers:
-            m = reg.match(lines[0])
-            if m:
-                handler(lines, replaced, flags)
-                break
-    
-    post_handler(lines, replaced, flags)
-    f = file(fname, 'w')
-    f.write("".join(replaced))
-    f.close()
-
-handlers = []
-
-
-def post_handler(lines, replaced, flags):
-    imports = []
-    if "needs_eq" in flags:
-        imports.append("eq_")
-    if "needs_assert_raises" in flags:
-        imports += ["assert_raises", "assert_raises_message"]
-    if imports:
-        for i, line in enumerate(replaced):
-            if "import" in line:
-                replaced.insert(i, "from sqlalchemy.test.testing import %s\n" % ", ".join(imports))
-                break
-    
-def remove_line(lines, replaced, flags):
-    lines.pop(0)
-    
-handlers.append((re.compile(r"import testenv; testenv\.configure_for_tests"), remove_line))
-handlers.append((re.compile(r"(.*\s)?import sa_unittest"), remove_line))
-
-
-def import_testlib_sa(lines, replaced, flags):
-    line = lines.pop(0)
-    line = line.replace("import testlib.sa", "import sqlalchemy")
-    replaced.append(line)
-handlers.append((re.compile("import testlib\.sa"), import_testlib_sa))
-
-def from_testlib_sa(lines, replaced, flags):
-    line = lines.pop(0)
-    while True:
-        if line.endswith("\\\n"):
-            line = line[0:-2] + lines.pop(0)
-        else:
-            break
-    
-    components = re.compile(r'from testlib\.sa import (.*)').match(line)
-    if components:
-        components = re.split(r"\s*,\s*", components.group(1))
-        line = "from sqlalchemy import %s\n" % (", ".join(c for c in components if c not in ("Table", "Column")))
-        replaced.append(line)
-        if "Table" in components:
-            replaced.append("from sqlalchemy.test.schema import Table\n")
-        if "Column" in components:
-            replaced.append("from sqlalchemy.test.schema import Column\n")
-        return
-        
-    line = line.replace("testlib.sa", "sqlalchemy")
-    replaced.append(line)
-handlers.append((re.compile("from testlib\.sa.*import"), from_testlib_sa))
-
-def from_testlib(lines, replaced, flags):
-    line = lines.pop(0)
-    
-    components = re.compile(r'from testlib import (.*)').match(line)
-    if components:
-        components = re.split(r"\s*,\s*", components.group(1))
-        if "sa" in components:
-            replaced.append("import sqlalchemy as sa\n")
-            replaced.append("from sqlalchemy.test import %s\n" % (", ".join(c for c in components if c != "sa" and c != "sa as tsa")))
-            return
-        elif "sa as tsa" in components:
-            replaced.append("import sqlalchemy as tsa\n")
-            replaced.append("from sqlalchemy.test import %s\n" % (", ".join(c for c in components if c != "sa" and c != "sa as tsa")))
-            return
-    
-    line = line.replace("testlib", "sqlalchemy.test")
-    replaced.append(line)
-handlers.append((re.compile(r"from testlib"), from_testlib))
-
-def from_orm(lines, replaced, flags):
-    line = lines.pop(0)
-    line = line.replace("from orm import", "from test.orm import")
-    line = line.replace("from orm.", "from test.orm.")
-    replaced.append(line)
-handlers.append((re.compile(r'from orm( import|\.)'), from_orm))
-    
-def assert_equals(lines, replaced, flags):
-    line = lines.pop(0)
-    line = line.replace("self.assertEquals", "eq_")
-    line = line.replace("self.assertEqual", "eq_")
-    replaced.append(line)
-    flags["needs_eq"] = True
-handlers.append((re.compile(r"\s*self\.assertEqual(s)?"), assert_equals))
-
-def assert_raises(lines, replaced, flags):
-    line = lines.pop(0)
-    line = line.replace("self.assertRaisesMessage", "assert_raises_message")
-    line = line.replace("self.assertRaises", "assert_raises")
-    replaced.append(line)
-    flags["needs_assert_raises"] = True
-handlers.append((re.compile(r"\s*self\.assertRaises(Message)?"), assert_raises))
-
-def setup_all(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def setUpAll\(self\)\:").match(line).group(1)
-    replaced.append("%s@classmethod\n" % whitespace)
-    replaced.append("%sdef setup_class(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def setUpAll\(self\)"), setup_all))
-
-def teardown_all(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def tearDownAll\(self\)\:").match(line).group(1)
-    replaced.append("%s@classmethod\n" % whitespace)
-    replaced.append("%sdef teardown_class(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def tearDownAll\(self\)"), teardown_all))
-
-def setup(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def setUp\(self\)\:").match(line).group(1)
-    replaced.append("%sdef setup(self):\n" % whitespace)
-handlers.append((re.compile(r"\s*def setUp\(self\)"), setup))
-
-def teardown(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def tearDown\(self\)\:").match(line).group(1)
-    replaced.append("%sdef teardown(self):\n" % whitespace)
-handlers.append((re.compile(r"\s*def tearDown\(self\)"), teardown))
-    
-def define_tables(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def define_tables").match(line).group(1)
-    replaced.append("%s@classmethod\n" % whitespace)
-    replaced.append("%sdef define_tables(cls, metadata):\n" % whitespace)
-handlers.append((re.compile(r"\s*def define_tables\(self, metadata\)"), define_tables))
-
-def setup_mappers(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def setup_mappers").match(line).group(1)
-    
-    i = -1
-    while re.match("\s*@testing", replaced[i]):
-        i -= 1
-        
-    replaced.insert(len(replaced) + i + 1, "%s@classmethod\n" % whitespace)
-    replaced.append("%sdef setup_mappers(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def setup_mappers\(self\)"), setup_mappers))
-
-def setup_classes(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def setup_classes").match(line).group(1)
-    
-    i = -1
-    while re.match("\s*@testing", replaced[i]):
-        i -= 1
-        
-    replaced.insert(len(replaced) + i + 1, "%s@classmethod\n" % whitespace)
-    replaced.append("%sdef setup_classes(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def setup_classes\(self\)"), setup_classes))
-
-def insert_data(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def insert_data").match(line).group(1)
-    
-    i = -1
-    while re.match("\s*@testing", replaced[i]):
-        i -= 1
-        
-    replaced.insert(len(replaced) + i + 1, "%s@classmethod\n" % whitespace)
-    replaced.append("%sdef insert_data(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def insert_data\(self\)"), insert_data))
-
-def fixtures(lines, replaced, flags):
-    line = lines.pop(0)
-    whitespace = re.compile(r"(\s*)def fixtures").match(line).group(1)
-    
-    i = -1
-    while re.match("\s*@testing", replaced[i]):
-        i -= 1
-        
-    replaced.insert(len(replaced) + i + 1, "%s@classmethod\n" % whitespace)
-    replaced.append("%sdef fixtures(cls):\n" % whitespace)
-handlers.append((re.compile(r"\s*def fixtures\(self\)"), fixtures))
-    
-    
-def call_main(lines, replaced, flags):
-    replaced.pop(-1)
-    lines.pop(0)
-handlers.append((re.compile(r"\s+testenv\.main\(\)"), call_main))
-
-def default(lines, replaced, flags):
-    replaced.append(lines.pop(0))
-handlers.append((re.compile(r".*"), default))
-
-
-if __name__ == '__main__':
-    convert("test/orm/inheritance/abc_inheritance.py")
-#    walk()

doc/build/builder/builders.py

@@ -62,7 +62,7 @@ class PyConWithSQLLexer(RegexLexer):
             ],
             'sqlpopup':[
                 (
-                    r'(.*?\n)((?:PRAGMA|BEGIN|SELECT|INSERT|DELETE|ROLLBACK|COMMIT|UPDATE|CREATE|DROP|PRAGMA|DESCRIBE).*?(?:{stop}\n*|$))',
+                    r'(.*?\n)((?:PRAGMA|BEGIN|SELECT|INSERT|DELETE|ROLLBACK|COMMIT|ALTER|UPDATE|CREATE|DROP|PRAGMA|DESCRIBE).*?(?:{stop}\n?|$))',
                     bygroups(using(PythonConsoleLexer), Token.Sql.Popup), 
                     "#pop"
                 )
@@ -91,7 +91,7 @@ class PythonWithSQLLexer(RegexLexer):
             ],
             'sqlpopup':[
                 (
-                    r'(.*?\n)((?:PRAGMA|BEGIN|SELECT|INSERT|DELETE|ROLLBACK|COMMIT|UPDATE|CREATE|DROP|PRAGMA|DESCRIBE).*?(?:{stop}\n*|$))',
+                    r'(.*?\n)((?:PRAGMA|BEGIN|SELECT|INSERT|DELETE|ROLLBACK|COMMIT|ALTER|UPDATE|CREATE|DROP|PRAGMA|DESCRIBE).*?(?:{stop}\n?|$))',
                     bygroups(using(PythonLexer), Token.Sql.Popup), 
                     "#pop"
                 )

doc/build/conf.py

@@ -17,6 +17,7 @@ import sys, os
 # is relative to the documentation root, use os.path.abspath to make it
 # absolute, like shown here.
 sys.path.insert(0, os.path.abspath('../../lib'))
+sys.path.insert(0, os.path.abspath('../../examples'))
 sys.path.insert(0, os.path.abspath('.'))
 
 import sqlalchemy
@@ -44,7 +45,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'SQLAlchemy'
-copyright = u'2007, 2008, 2009, the SQLAlchemy authors and contributors'
+copyright = u'2007, 2008, 2009, 2010, the SQLAlchemy authors and contributors'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

doc/build/copyright.rst

@@ -4,8 +4,8 @@ Appendix:  Copyright
 
 This is the MIT license: `<http://www.opensource.org/licenses/mit-license.php>`_
 
-Copyright (c) 2005, 2006, 2007, 2008 Michael Bayer and contributors. SQLAlchemy is a trademark of Michael
-Bayer.
+Copyright (c) 2005, 2006, 2007, 2008, 2009, 2010 Michael Bayer and contributors. 
+SQLAlchemy is a trademark of Michael Bayer.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software

doc/build/dbengine.rst

@@ -19,9 +19,9 @@ Where above, a :class:`~sqlalchemy.engine.Engine` references both a  :class:`~sq
 
 Creating an engine is just a matter of issuing a single call, :func:`create_engine()`::
 
-    engine = create_engine('postgres://scott:tiger@localhost:5432/mydatabase')
+    engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase')
     
-The above engine invokes the ``postgres`` dialect and a connection pool which references ``localhost:5432``.
+The above engine invokes the ``postgresql`` dialect and a connection pool which references ``localhost:5432``.
 
 The engine can be used directly to issue SQL to the database.  The most generic way is to use connections, which you get via the ``connect()`` method::
 
@@ -52,35 +52,37 @@ The ``Engine`` and ``Connection`` can do a lot more than what we illustrated abo
 
 Supported Databases 
 ====================
-Recall that the ``Dialect`` is used to describe how to talk to a specific kind of database.  Dialects are included with SQLAlchemy for many different backends; these can be seen as a Python package within the :mod:`~sqlalchemy.databases` package.  Each dialect requires the appropriate DBAPI drivers to be installed separately.
+Recall that the ``Dialect`` is used to describe how to talk to a specific kind of database.  Dialects are included with SQLAlchemy for many different backends; these can be seen as a Python package within the :mod:`~sqlalchemy.dialect` package.  Each dialect requires the appropriate DBAPI drivers to be installed separately.
 
 Dialects included with SQLAlchemy fall under one of three categories: supported, experimental, and third party.  Supported drivers are those which work against the most common databases available in the open source world, including SQLite, PostgreSQL, MySQL, and Firebird.   Very popular commercial databases which provide easy access to test platforms are also supported, these currently include MSSQL and Oracle.   These dialects are tested frequently and the level of support should be close to 100% for each.
 
-The experimental category is for drivers against less common database platforms, or commercial platforms for which no freely available and easily usable test platform is provided.   These include Access, MaxDB, Informix, and Sybase at the time of this writing.  These are partially-functioning dialects for which the SQLAlchemy project is not able to provide regular test support.  If you're interested in supporting one of these backends, contact the mailing list.
+The experimental category is for drivers against less common database platforms, or commercial platforms for which no freely available and easily usable test platform is provided.   These include Access, MaxDB, Informix, and Sybase at the time of this writing.  These are not-yet-functioning
+or partially-functioning dialects for which the SQLAlchemy project is not able to provide regular test support.  If you're interested in supporting one of these backends, contact the mailing list.
 
 There are also third-party dialects available - currently IBM offers a DB2/Informix IDS dialect for SQLAlchemy.
 
 Downloads for each DBAPI at the time of this writing are as follows:
 
 * Supported Dialects
-
- - PostgreSQL:  `psycopg2 <http://www.initd.org/tracker/psycopg/wiki/PsycopgTwo>`_ 
- - SQLite:  `sqlite3 <http://www.python.org/doc/2.5.2/lib/module-sqlite3.html>`_ (included in Python 2.5 or greater) `pysqlite <http://initd.org/tracker/pysqlite>`_
- - MySQL:   `MySQLDB (a.k.a. mysql-python) <http://sourceforge.net/projects/mysql-python>`_
- - Oracle:  `cx_Oracle <http://cx-oracle.sourceforge.net/>`_
- - Firebird:  `kinterbasdb <http://kinterbasdb.sourceforge.net/>`_
- - MS-SQL, MSAccess:  `pyodbc <http://pyodbc.sourceforge.net/>`_ (recommended) `adodbapi <http://adodbapi.sourceforge.net/>`_  `pymssql <http://pymssql.sourceforge.net/>`_
+  - PostgreSQL:  `psycopg2 <http://www.initd.org/tracker/psycopg/wiki/PsycopgTwo>`_ `pg8000 <http://pybrary.net/pg8000/>`_
+  - PostgreSQL on Jython: `PostgreSQL JDBC Driver <http://jdbc.postgresql.org/>`_
+  - SQLite:  `sqlite3 <http://www.python.org/doc/2.5.2/lib/module-sqlite3.html>`_ (included in Python 2.5 or greater) `pysqlite <http://initd.org/tracker/pysqlite>`_
+  - MySQL:   `MySQLdb (a.k.a. mysql-python) <http://sourceforge.net/projects/mysql-python>`_ `MySQL Connector/Python <https://launchpad.net/myconnpy>`_
+  - MySQL on Jython: `MySQL Connector/J JDBC driver <http://dev.mysql.com/downloads/connector/j/>`_
+  - Oracle:  `cx_Oracle <http://cx-oracle.sourceforge.net/>`_
+  - Oracle on Jython:  `Oracle JDBC Driver <http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/index.html>`_
+  - Firebird:  `kinterbasdb <http://kinterbasdb.sourceforge.net/>`_
+  - MS-SQL, MSAccess:  `pyodbc <http://pyodbc.sourceforge.net/>`_ (recommended) `adodbapi <http://adodbapi.sourceforge.net/>`_  `pymssql <http://pymssql.sourceforge.net/>`_
+  - MS-SQL on Jython:  `jTDS JDBC Driver <http://jtds.sourceforge.net/>`_
 
 * Experimental Dialects
-
- - MSAccess:  `pyodbc <http://pyodbc.sourceforge.net/>`_
- - Informix:  `informixdb <http://informixdb.sourceforge.net/>`_
- - Sybase:   TODO
- - MAXDB:    TODO
+  - MSAccess:  `pyodbc <http://pyodbc.sourceforge.net/>`_
+  - Informix:  `informixdb <http://informixdb.sourceforge.net/>`_
+  - Sybase:   TODO
+  - MAXDB:    `sapdb <http://www.sapdb.org/sapdbapi.html>`_
 
 * Third Party Dialects
-
- - DB2/Informix IDS: `ibm-db <http://code.google.com/p/ibm-db/>`_
+  - DB2/Informix IDS: `ibm-db <http://code.google.com/p/ibm-db/>`_
 
 The SQLAlchemy Wiki contains a page of database notes, describing whatever quirks and behaviors have been observed.  Its a good place to check for issues with specific databases.  `Database Notes <http://www.sqlalchemy.org/trac/wiki/DatabaseNotes>`_
 
@@ -89,31 +91,42 @@ create_engine() URL Arguments
 
 SQLAlchemy indicates the source of an Engine strictly via `RFC-1738 <http://rfc.net/rfc1738.html>`_ style URLs, combined with optional keyword arguments to specify options for the Engine.  The form of the URL is:
 
-    driver://username:password@host:port/database
+    dialect+driver://username:password@host:port/database
 
-Dialect names include the identifying name of the SQLAlchemy dialect which include ``sqlite``, ``mysql``, ``postgres``, ``oracle``, ``mssql``, and ``firebird``.  In SQLAlchemy 0.5 and earlier, the DBAPI implementation is automatically selected if more than one are available - currently this includes only MSSQL (pyodbc is the default, then adodbapi, then pymssql) and SQLite (sqlite3 is the default, or pysqlite if sqlite3 is not availble).   When using MSSQL, ``create_engine()`` accepts a ``module`` argument which specifies the name of the desired DBAPI to be used, overriding the default behavior.   
+Dialect names include the identifying name of the SQLAlchemy dialect which include ``sqlite``, ``mysql``, ``postgresql``, ``oracle``, ``mssql``, and ``firebird``.  The drivername is the name of the DBAPI to be used to connect to the database using all lowercase letters.   If not specified, a "default" DBAPI will be imported if available - this default is typically the most widely known driver available for that backend (i.e. cx_oracle, pysqlite/sqlite3, psycopg2, mysqldb).   For Jython connections, specify the `zxjdbc` driver, which is the JDBC-DBAPI bridge included with Jython.
 
-  .. sourcecode:: python+sql
-  
-    # postgresql
-    pg_db = create_engine('postgres://scott:tiger@localhost/mydatabase')
+.. sourcecode:: python+sql
 
-    # mysql
-    mysql_db = create_engine('mysql://scott:tiger@localhost/mydatabase')
-  
-    # oracle
+    # postgresql - psycopg2 is the default driver.
+    pg_db = create_engine('postgresql://scott:tiger@localhost/mydatabase')
+    pg_db = create_engine('postgresql+psycopg2://scott:tiger@localhost/mydatabase')
+    pg_db = create_engine('postgresql+pg8000://scott:tiger@localhost/mydatabase')
+
+    # postgresql on Jython
+    pg_db = create_engine('postgresql+zxjdbc://scott:tiger@localhost/mydatabase')
+    
+    # mysql - MySQLdb (mysql-python) is the default driver
+    mysql_db = create_engine('mysql://scott:tiger@localhost/foo')
+    mysql_db = create_engine('mysql+mysqldb://scott:tiger@localhost/foo')
+
+    # mysql on Jython
+    mysql_db = create_engine('mysql+zxjdbc://localhost/foo')
+
+    # mysql with pyodbc (buggy)
+    mysql_db = create_engine('mysql+pyodbc://scott:tiger@some_dsn')
+
+    # oracle - cx_oracle is the default driver
     oracle_db = create_engine('oracle://scott:tiger@127.0.0.1:1521/sidname')
-  
+
     # oracle via TNS name
-    oracle_db = create_engine('oracle://scott:tiger@tnsname')
-  
+    oracle_db = create_engine('oracle+cx_oracle://scott:tiger@tnsname')
+
     # mssql using ODBC datasource names.  PyODBC is the default driver.
     mssql_db = create_engine('mssql://mydsn')
-    mssql_db = create_engine('mssql://scott:tiger@mydsn')
-    
-    # firebird
-    firebird_db = create_engine('firebird://scott:tiger@localhost/sometest.gdm')
-  
+    mssql_db = create_engine('mssql+pyodbc://mydsn')
+    mssql_db = create_engine('mssql+adodbapi://mydsn')
+    mssql_db = create_engine('mssql+pyodbc://username:password@mydsn')
+
 SQLite connects to file based databases.   The same URL format is used, omitting the hostname, and using the "file" portion as the filename of the database.   This has the effect of four slashes being present for an absolute file path::
 
     # sqlite://<nohostname>/<path>
@@ -132,12 +145,11 @@ The :class:`~sqlalchemy.engine.base.Engine` will ask the connection pool for a c
 Custom DBAPI connect() arguments
 --------------------------------
 
-
 Custom arguments used when issuing the ``connect()`` call to the underlying DBAPI may be issued in three distinct ways.  String-based arguments can be passed directly from the URL string as query arguments:
 
 .. sourcecode:: python+sql
 
-    db = create_engine('postgres://scott:tiger@localhost/test?argument1=foo&argument2=bar')
+    db = create_engine('postgresql://scott:tiger@localhost/test?argument1=foo&argument2=bar')
 
 If SQLAlchemy's database connector is aware of a particular query argument, it may convert its type from string to its proper type.
     
@@ -145,7 +157,7 @@ If SQLAlchemy's database connector is aware of a particular query argument, it m
 
 .. sourcecode:: python+sql
 
-    db = create_engine('postgres://scott:tiger@localhost/test', connect_args = {'argument1':17, 'argument2':'bar'})
+    db = create_engine('postgresql://scott:tiger@localhost/test', connect_args = {'argument1':17, 'argument2':'bar'})
 
 The most customizable connection method of all is to pass a ``creator`` argument, which specifies a callable that returns a DBAPI connection:
 
@@ -154,7 +166,7 @@ The most customizable connection method of all is to pass a ``creator`` argument
     def connect():
         return psycopg.connect(user='scott', host='localhost')
 
-    db = create_engine('postgres://', creator=connect)
+    db = create_engine('postgresql://', creator=connect)
 
 .. _create_engine_args:
 
@@ -165,7 +177,7 @@ Keyword options can also be specified to ``create_engine()``, following the stri
 
 .. sourcecode:: python+sql
 
-    db = create_engine('postgres://...', encoding='latin1', echo=True)
+    db = create_engine('postgresql://...', encoding='latin1', echo=True)
 
 Options common to all database dialects are described at :func:`~sqlalchemy.create_engine`.
 
@@ -182,9 +194,9 @@ The ``execute()`` methods on both ``Engine`` and ``Connection`` can also receive
         print row['col1'], row['col2']
     connection.close()
 
-The above SQL construct is known as a ``select()``.  The full range of SQL constructs available are described in `sql`.
+The above SQL construct is known as a ``select()``.  The full range of SQL constructs available are described in :ref:`sqlexpression_toplevel`.
 
-Both ``Connection`` and ``Engine`` fulfill an interface known as ``Connectable`` which specifies common functionality between the two objects, namely being able to call ``connect()`` to return a ``Connection`` object (``Connection`` just returns itself), and being able to call ``execute()`` to get a result set.   Following this, most SQLAlchemy functions and objects which accept an ``Engine`` as a parameter or attribute with which to execute SQL will also accept a ``Connection``.  As of SQLAlchemy 0.3.9, this argument is named ``bind``::
+Both ``Connection`` and ``Engine`` fulfill an interface known as ``Connectable`` which specifies common functionality between the two objects, namely being able to call ``connect()`` to return a ``Connection`` object (``Connection`` just returns itself), and being able to call ``execute()`` to get a result set.   Following this, most SQLAlchemy functions and objects which accept an ``Engine`` as a parameter or attribute with which to execute SQL will also accept a ``Connection``.  This argument is named ``bind``::
 
     engine = create_engine('sqlite:///:memory:')
     
@@ -253,7 +265,7 @@ The ``Transaction`` object also handles "nested" behavior by keeping track of th
 
 Above, ``method_a`` is called first, which calls ``connection.begin()``.  Then it calls ``method_b``. When ``method_b`` calls ``connection.begin()``, it just increments a counter that is decremented when it calls ``commit()``.  If either ``method_a`` or ``method_b`` calls ``rollback()``, the whole transaction is rolled back.  The transaction is not committed until ``method_a`` calls the ``commit()`` method.  This "nesting" behavior allows the creation of functions which "guarantee" that a transaction will be used if one was not already available, but will automatically participate in an enclosing transaction if one exists.
 
-Note that SQLAlchemy's Object Relational Mapper also provides a way to control transaction scope at a higher level; this is described in `unitofwork_transaction`.
+Note that SQLAlchemy's Object Relational Mapper also provides a way to control transaction scope at a higher level; this is described in :ref:`unitofwork_transaction`.
 
 .. index::
    single: thread safety; transactions
@@ -261,7 +273,6 @@ Note that SQLAlchemy's Object Relational Mapper also provides a way to control t
 Transaction Facts:
 
 * the Transaction object, just like its parent Connection, is **not thread-safe**.
-* SQLAlchemy 0.4 will feature transactions with two-phase commit capability as well as SAVEPOINT capability.
 
 Understanding Autocommit
 ------------------------
@@ -274,12 +285,14 @@ The above transaction example illustrates how to use ``Transaction`` so that sev
     conn = engine.connect()
     conn.execute("INSERT INTO users VALUES (1, 'john')")  # autocommits
 
+.. _dbengine_implicit:
+
 Connectionless Execution, Implicit Execution 
 =============================================
 
-Recall from the first section we mentioned executing with and without a ``Connection``.  ``Connectionless`` execution refers to calling the ``execute()`` method on an object which is not a ``Connection``, which could be on the ``Engine`` itself, or could be a constructed SQL object.  When we say "implicit", we mean that we are calling the ``execute()`` method on an object which is neither a ``Connection`` nor an ``Engine`` object; this can only be used with constructed SQL objects which have their own ``execute()`` method, and can be "bound" to an ``Engine``.  A description of "constructed SQL objects" may be found in `sql`.
+Recall from the first section we mentioned executing with and without a ``Connection``.  ``Connectionless`` execution refers to calling the ``execute()`` method on an object which is not a ``Connection``, which could be on the ``Engine`` itself, or could be a constructed SQL object.  When we say "implicit", we mean that we are calling the ``execute()`` method on an object which is neither a ``Connection`` nor an ``Engine`` object; this can only be used with constructed SQL objects which have their own ``execute()`` method, and can be "bound" to an ``Engine``.  A description of "constructed SQL objects" may be found in :ref:`sqlexpression_toplevel`.
 
-A summary of all three methods follows below.  First, assume the usage of the following ``MetaData`` and ``Table`` objects; while we haven't yet introduced these concepts, for now you only need to know that we are representing a database table, and are creating an "executable" SQL construct which issues a statement to the database.  These objects are described in `metadata`.
+A summary of all three methods follows below.  First, assume the usage of the following ``MetaData`` and ``Table`` objects; while we haven't yet introduced these concepts, for now you only need to know that we are representing a database table, and are creating an "executable" SQL construct which issues a statement to the database.  These objects are described in :ref:`metadata_toplevel`.
 
 .. sourcecode:: python+sql
 
@@ -310,7 +323,7 @@ Explicit, connectionless execution delivers the expression to the ``execute()``
         # ....
     result.close()
 
-Implicit execution is also connectionless, and calls the ``execute()`` method on the expression itself, utilizing the fact that either an ``Engine`` or ``Connection`` has been *bound* to the expression object (binding is discussed further in the next section, `metadata`):
+Implicit execution is also connectionless, and calls the ``execute()`` method on the expression itself, utilizing the fact that either an ``Engine`` or ``Connection`` has been *bound* to the expression object (binding is discussed further in the next section, :ref:`metadata_toplevel`):
 
 .. sourcecode:: python+sql
 

doc/build/examples.rst

@@ -0,0 +1,121 @@
+.. _examples_toplevel:
+
+Examples
+========
+
+The SQLAlchemy distribution includes a variety of code examples illustrating a select set of patterns, some typical and some not so typical.   All are runnable and can be found in the ``/examples`` directory of the distribution.   Each example contains a README in its ``__init__.py`` file, each of which are listed below.
+
+Additional SQLAlchemy examples, some user contributed, are available on the wiki at `<http://www.sqlalchemy.org/trac/wiki/UsageRecipes>`_.
+
+Adjacency List
+--------------
+
+Location: /examples/adjacency_list/
+
+.. automodule:: adjacency_list
+
+Associations
+------------
+
+Location: /examples/association/
+
+.. automodule:: association
+
+Attribute Instrumentation
+-------------------------
+
+Location: /examples/custom_attributes/
+
+.. automodule:: custom_attributes
+
+Beaker Caching
+--------------
+
+Location: /examples/beaker_caching/
+
+.. automodule:: beaker_caching
+
+Derived Attributes
+------------------
+
+Location: /examples/derived_attributes/
+
+.. automodule:: derived_attributes
+
+
+Directed Graphs
+---------------
+
+Location: /examples/graphs/
+
+.. automodule:: graphs
+
+Dynamic Relations as Dictionaries
+----------------------------------
+
+Location: /examples/dynamic_dict/
+
+.. automodule:: dynamic_dict
+
+Horizontal Sharding
+-------------------
+
+Location: /examples/sharding
+
+.. automodule:: sharding
+
+Inheritance Mappings
+--------------------
+
+Location: /examples/inheritance/
+
+.. automodule:: inheritance
+
+Large Collections
+-----------------
+
+Location: /examples/large_collection/
+
+.. automodule:: large_collection
+
+Nested Sets
+-----------
+
+Location: /examples/nested_sets/
+
+.. automodule:: nested_sets
+
+Polymorphic Associations
+------------------------
+
+Location: /examples/poly_assoc/
+
+.. automodule:: poly_assoc
+
+PostGIS Integration
+-------------------
+
+Location: /examples/postgis
+
+.. automodule:: postgis
+
+Versioned Objects
+-----------------
+
+Location: /examples/versioning
+
+.. automodule:: versioning
+
+Vertical Attribute Mapping
+--------------------------
+
+Location: /examples/vertical
+
+.. automodule:: vertical
+
+XML Persistence
+---------------
+
+Location: /examples/elementtree/
+
+.. automodule:: elementtree

doc/build/index.rst

@@ -10,6 +10,7 @@ Table of Contents
     session
     dbengine
     metadata
+    examples
     reference/index
 
 Indices and tables

doc/build/intro.rst

@@ -8,22 +8,11 @@ Overview
 ========
 
 
-The SQLAlchemy SQL Toolkit and Object Relational Mapper is a comprehensive set of tools for working with databases and Python.  It has several distinct areas of functionality which can be used individually or combined together.  Its major API components, all public-facing, are illustrated below::
+The SQLAlchemy SQL Toolkit and Object Relational Mapper is a comprehensive set of tools for working with databases and Python.  It has several distinct areas of functionality which can be used individually or combined together.  Its major components are illustrated below.  The arrows represent the general dependencies of components:
 
-               +-----------------------------------------------------------+
-               |             Object Relational Mapper (ORM)                |
-               +-----------------------------------------------------------+
-               +---------+ +------------------------------------+ +--------+
-               |         | |       SQL Expression Language      | |        |
-               |         | +------------------------------------+ |        |
-               |         +-----------------------+ +--------------+        |
-               |        Dialect/Execution        | |    Schema Management  |
-               +---------------------------------+ +-----------------------+
-               +----------------------+ +----------------------------------+
-               |  Connection Pooling  | |              Types               |
-               +----------------------+ +----------------------------------+
+.. image:: sqla_arch_small.jpg
 
-Above, the two most significant front-facing portions of SQLAlchemy are the **Object Relational Mapper** and the **SQL Expression Language**.  These are two separate toolkits, one building off the other.  SQL Expressions can be used independently of the ORM.  When using the ORM, the SQL Expression language is used to establish object-relational configurations as well as in querying.
+Above, the two most significant front-facing portions of SQLAlchemy are the **Object Relational Mapper** and the **SQL Expression Language**.  SQL Expressions can be used independently of the ORM.  When using the ORM, the SQL Expression language remains part of the public facing API as it is used within object-relational configurations and queries.
 
 Tutorials
 =========
@@ -76,15 +65,15 @@ SQLAlchemy is designed to operate with a `DB-API <http://www.python.org/doc/peps
 Checking the Installed SQLAlchemy Version
 =========================================
 
-This documentation covers SQLAlchemy version 0.5.  If you're working on a system that already has SQLAlchemy installed, check the version from your Python prompt like this:
+This documentation covers SQLAlchemy version 0.6.  If you're working on a system that already has SQLAlchemy installed, check the version from your Python prompt like this:
 
 .. sourcecode:: python+sql
 
      >>> import sqlalchemy
      >>> sqlalchemy.__version__ # doctest: +SKIP
-     0.5.0
+     0.6.0
 
-0.4 to 0.5 Migration 
+0.5 to 0.6 Migration 
 =====================
 
-Notes on what's changed from 0.4 to 0.5 is available on the SQLAlchemy wiki at `05Migration <http://www.sqlalchemy.org/trac/wiki/05Migration>`_.
+Notes on what's changed from 0.5 to 0.6 is available on the SQLAlchemy wiki at `06Migration <http://www.sqlalchemy.org/trac/wiki/06Migration>`_.

doc/build/mappers.rst

@@ -312,7 +312,7 @@ The "equals" comparison operation by default produces an AND of all correspondin
 Controlling Ordering 
 ---------------------
 
-As of version 0.5, the ORM does not generate ordering for any query unless explicitly configured.
+The ORM does not generate ordering for any query unless explicitly configured.
 
 The "default" ordering for a collection, which applies to list-based collections, can be configured using the ``order_by`` keyword argument on ``relation()``::
 
@@ -337,6 +337,8 @@ Ordering for rows loaded through ``Query`` is usually specified using the ``orde
 
 Above, a ``Query`` issued for the ``User`` class will use the value of the mapper's ``order_by`` setting if the ``Query`` itself has no ordering specified.
 
+.. _datamapping_inheritance:
+
 Mapping Class Inheritance Hierarchies 
 --------------------------------------
 
@@ -806,7 +808,6 @@ Above, the "customers" table is joined against the "orders" table to produce a f
 Multiple Mappers for One Class 
 -------------------------------
 
-
 The first mapper created for a certain class is known as that class's "primary mapper."  Other mappers can be created as well on the "load side" - these are called **secondary mappers**.   This is a mapper that must be constructed with the keyword argument ``non_primary=True``, and represents a load-only mapper.  Objects that are loaded with a secondary mapper will have their save operation processed by the primary mapper.  It is also invalid to add new ``relation()`` objects to a non-primary mapper. To use this mapper with the Session, specify it to the ``query`` method:
 
 example:
@@ -824,8 +825,6 @@ example:
 
 The "non primary mapper" is a rarely needed feature of SQLAlchemy; in most cases, the ``Query`` object can produce any kind of query that's desired.  It's recommended that a straight ``Query`` be used in place of a non-primary mapper unless the mapper approach is absolutely needed.  Current use cases for the "non primary mapper" are when you want to map the class to a particular select statement or view to which additional query criterion can be added, and for when the particular mapped select statement or view is to be placed in a ``relation()`` of a parent mapper.
 
-Versions of SQLAlchemy previous to 0.5 included another mapper flag called "entity_name", as of version 0.5.0 this feature has been removed (it never worked very well).
-
 Constructors and Object Initialization 
 ---------------------------------------
 
@@ -881,6 +880,8 @@ Multiple extensions will be chained together and processed in order; they are sp
 
     m = mapper(User, users_table, extension=[ext1, ext2, ext3])
 
+.. _advdatamapping_relation:
+
 Relation Configuration 
 =======================
 
@@ -1466,7 +1467,7 @@ Dictionary-Based Collections
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
-A ``dict`` can be used as a collection, but a keying strategy is needed to map entities loaded by the ORM to key, value pairs.  The `sqlalchemy.orm.collections` package provides several built-in types for dictionary-based collections:
+A ``dict`` can be used as a collection, but a keying strategy is needed to map entities loaded by the ORM to key, value pairs.  The :mod:`sqlalchemy.orm.collections` package provides several built-in types for dictionary-based collections:
 
 .. sourcecode:: python+sql
 
@@ -1488,7 +1489,7 @@ A ``dict`` can be used as a collection, but a keying strategy is needed to map e
 
 These functions each provide a ``dict`` subclass with decorated ``set`` and ``remove`` methods and the keying strategy of your choice.
 
-The `sqlalchemy.orm.collections.MappedCollection` class can be used as a base class for your custom types or as a mix-in to quickly add ``dict`` collection support to other classes.  It uses a keying function to delegate to ``__setitem__`` and ``__delitem__``:
+The :class:`sqlalchemy.orm.collections.MappedCollection` class can be used as a base class for your custom types or as a mix-in to quickly add ``dict`` collection support to other classes.  It uses a keying function to delegate to ``__setitem__`` and ``__delitem__``:
 
 .. sourcecode:: python+sql
 
@@ -1521,7 +1522,7 @@ The decorations are lightweight and no-op outside of relations, but they do add
 
 The ORM uses this approach for built-ins, quietly substituting a trivial subclass when a ``list``, ``set`` or ``dict`` is used directly.
 
-The collections package provides additional decorators and support for authoring custom types.  See the `sqlalchemy.orm.collections` for more information and discussion of advanced usage and Python 2.3-compatible decoration options.
+The collections package provides additional decorators and support for authoring custom types.  See the :mod:`sqlalchemy.orm.collections` package for more information and discussion of advanced usage and Python 2.3-compatible decoration options.
 
 Configuring Loader Strategies: Lazy Loading, Eager Loading 
 -----------------------------------------------------------
@@ -1765,7 +1766,7 @@ When ``passive_deletes`` is applied, the ``children`` relation will not be loade
 Mutable Primary Keys / Update Cascades 
 ---------------------------------------
 
-As of SQLAlchemy 0.4.2, the primary key attributes of an instance can be changed freely, and will be persisted upon flush.  When the primary key of an entity changes, related items which reference the primary key must also be updated as well.  For databases which enforce referential integrity, it's required to use the database's ON UPDATE CASCADE functionality in order to propagate primary key changes.  For those which don't, the ``passive_cascades`` flag can be set to ``False`` which instructs SQLAlchemy to issue UPDATE statements individually.  The ``passive_cascades`` flag can also be ``False`` in conjunction with ON UPDATE CASCADE functionality, although in that case it issues UPDATE statements unnecessarily.
+When the primary key of an entity changes, related items which reference the primary key must also be updated as well.  For databases which enforce referential integrity, it's required to use the database's ON UPDATE CASCADE functionality in order to propagate primary key changes.  For those which don't, the ``passive_updates`` flag can be set to ``False`` which instructs SQLAlchemy to issue UPDATE statements individually.  The ``passive_updates`` flag can also be ``False`` in conjunction with ON UPDATE CASCADE functionality, although in that case it issues UPDATE statements unnecessarily.
 
 A typical mutable primary key setup might look like:
 

doc/build/metadata.rst

@@ -7,7 +7,11 @@ Database Meta Data
 Describing Databases with MetaData
 ==================================
 
-The core of SQLAlchemy's query and object mapping operations are supported by **database metadata**, which is comprised of Python objects that describe tables and other schema-level objects.  These objects can be created by explicitly naming the various components and their properties, using the Table, Column, ForeignKey, Index, and Sequence objects imported from ``sqlalchemy.schema``.  There is also support for **reflection** of some entities, which means you only specify the *name* of the entities and they are recreated from the database automatically.
+The core of SQLAlchemy's query and object mapping operations are supported by *database metadata*, which is comprised of Python objects that describe tables and other schema-level objects.  These objects are at the core of three major types of operations - issuing CREATE and DROP statements (known as *DDL*), constructing SQL queries, and expressing information about structures that already exist within the database. 
+
+Database metadata can be expressed by explicitly naming the various components and their properties, using constructs such as ``Table``, ``Column``, ``ForeignKey`` and ``Sequence``, all of which are imported from the ``sqlalchemy.schema`` package.   It can also be generated by SQLAlchemy using a process called *reflection*, which means you start with a single object such as ``Table``, assign it a name, and then instruct SQLAlchemy to load all the additional information related to that name from a particular engine source.   
+
+A key feature of SQLAlchemy's database metadata constructs is that they are designed to be used in a *declarative* style which closely resembles that of real DDL.   They are therefore most intuitive to those who have some background in creating real schema generation scripts.
 
 A collection of metadata entities is stored in an object aptly named ``MetaData``::
 
@@ -15,67 +19,44 @@ A collection of metadata entities is stored in an object aptly named ``MetaData`
     
     metadata = MetaData()
 
-To represent a Table, use the ``Table`` class::
+``MetaData`` is a container object that keeps together many different features of a database (or multiple databases) being described.   
 
-    users = Table('users', metadata, 
+To represent a table, use the ``Table`` class.  Its two primary arguments are the table name, then the ``MetaData`` object which it will be associated with.   The remaining positional arguments are mostly ``Column`` objects describing each column::
+
+    user = Table('user', metadata, 
         Column('user_id', Integer, primary_key = True),
         Column('user_name', String(16), nullable = False),
-        Column('email_address', String(60), key='email'),
+        Column('email_address', String(60)),
         Column('password', String(20), nullable = False)
     )
-    
-    user_prefs = Table('user_prefs', metadata, 
-        Column('pref_id', Integer, primary_key=True),
-        Column('user_id', Integer, ForeignKey("users.user_id"), nullable=False),
-        Column('pref_name', String(40), nullable=False),
-        Column('pref_value', String(100))
-    )
 
-The specific datatypes for each Column, such as Integer, String, etc. are described in `types`, and exist within the module ``sqlalchemy.types`` as well as the global ``sqlalchemy`` namespace.
+Above, a table called ``user`` is described, which contains four columns.   The primary key of the table consists of the ``user_id`` column.   Multiple columns may be assigned the ``primary_key=True`` flag which denotes a multi-column primary key, known as a *composite* primary key.
 
-.. _metadata_foreignkeys:
-
-Defining Foreign Keys
----------------------
-
-Foreign keys are most easily specified by the ``ForeignKey`` object within a ``Column`` object.  For a composite foreign key, i.e. a foreign key that contains multiple columns referencing multiple columns to a composite primary key, an explicit syntax is provided which allows the correct table CREATE statements to be generated::
-
-    # a table with a composite primary key
-    invoices = Table('invoices', metadata, 
-        Column('invoice_id', Integer, primary_key=True),
-        Column('ref_num', Integer, primary_key=True),
-        Column('description', String(60), nullable=False)
-    )
-    
-    # a table with a composite foreign key referencing the parent table
-    invoice_items = Table('invoice_items', metadata, 
-        Column('item_id', Integer, primary_key=True),
-        Column('item_name', String(60), nullable=False),
-        Column('invoice_id', Integer, nullable=False),
-        Column('ref_num', Integer, nullable=False),
-        ForeignKeyConstraint(['invoice_id', 'ref_num'], ['invoices.invoice_id', 'invoices.ref_num'])
-    )
-    
-Above, the ``invoice_items`` table will have ``ForeignKey`` objects automatically added to the ``invoice_id`` and ``ref_num`` ``Column`` objects as a result of the additional ``ForeignKeyConstraint`` object.
+Note also that each column describes its datatype using objects corresponding to genericized types, such as ``Integer`` and ``String``.    SQLAlchemy features dozens of types of varying levels of specificity as well as the ability to create custom types.   Documentation on the type system can be found at :ref:`types`.
 
 Accessing Tables and Columns
 ----------------------------
 
-The ``MetaData`` object supports some handy methods, such as getting a list of Tables in the order (or reverse) of their dependency::
+The ``MetaData`` object contains all of the schema constructs we've associated with it.   It supports a few methods of accessing these table objects, such as the ``sorted_tables`` accessor which returns a list of each ``Table`` object in order of foreign key dependency (that is, each table is preceded by all tables which it references)::
 
-    >>> for t in metadata.table_iterator(reverse=False):
+    >>> for t in metadata.sorted_tables:
     ...    print t.name
-    users
-    user_prefs
-        
-And ``Table`` provides an interface to the table's properties as well as that of its columns::
+    user
+    user_preference
+    invoice
+    invoice_item
+
+In most cases, individual ``Table`` objects have been explicitly declared, and these objects are typically accessed directly as module-level variables in an application. 
+Once a ``Table`` has been defined, it has a full set of accessors which allow inspection of its properties.  Given the following ``Table`` definition::
 
     employees = Table('employees', metadata, 
         Column('employee_id', Integer, primary_key=True),
-        Column('employee_name', String(60), nullable=False, key='name'),
+        Column('employee_name', String(60), nullable=False),
         Column('employee_dept', Integer, ForeignKey("departments.department_id"))
     )
-    
+
+Note the ``ForeignKey`` object used in this table - this construct defines a reference to a remote table, and is fully described in :ref:`metadata_foreignkeys`.   Methods of accessing information about this table include::
+
     # access the column "EMPLOYEE_ID":
     employees.columns.employee_id
     
@@ -122,151 +103,68 @@ And ``Table`` provides an interface to the table's properties as well as that of
 
 .. _metadata_binding:
 
-Binding MetaData to an Engine or Connection 
---------------------------------------------
 
-A ``MetaData`` object can be associated with an ``Engine`` or an individual ``Connection``; this process is called **binding**.  The term used to describe "an engine or a connection" is often referred to as a **connectable**.  Binding allows the ``MetaData`` and the elements which it contains to perform operations against the database directly, using the connection resources to which it's bound.   Common operations which are made more convenient through binding include being able to generate SQL constructs which know how to execute themselves, creating ``Table`` objects which query the database for their column and constraint information, and issuing CREATE or DROP statements.
-
-To bind ``MetaData`` to an ``Engine``, use the ``bind`` attribute::
-
-    engine = create_engine('sqlite://', **kwargs)
-    
-    # create MetaData 
-    meta = MetaData()
-
-    # bind to an engine
-    meta.bind = engine
-
-Once this is done, the ``MetaData`` and its contained ``Table`` objects can access the database directly::
-
-    meta.create_all()  # issue CREATE statements for all tables
-    
-    # describe a table called 'users', query the database for its columns
-    users_table = Table('users', meta, autoload=True)
-    
-    # generate a SELECT statement and execute
-    result = users_table.select().execute()
-
-Note that the feature of binding engines is **completely optional**.  All of the operations which take advantage of "bound" ``MetaData`` also can be given an ``Engine`` or ``Connection`` explicitly with which to perform the operation.   The equivalent "non-bound" of the above would be::
-
-    meta.create_all(engine)  # issue CREATE statements for all tables
-    
-    # describe a table called 'users',  query the database for its columns
-    users_table = Table('users', meta, autoload=True, autoload_with=engine)
-    
-    # generate a SELECT statement and execute
-    result = engine.execute(users_table.select())
-
-Reflecting Tables
------------------
-
-
-A ``Table`` object can be created without specifying any of its contained attributes, using the argument ``autoload=True`` in conjunction with the table's name and possibly its schema (if not the databases "default" schema).  (You can also specify a list or set of column names to autoload as the kwarg include_columns, if you only want to load a subset of the columns in the actual database.)  This will issue the appropriate queries to the database in order to locate all properties of the table required for SQLAlchemy to use it effectively, including its column names and datatypes, foreign and primary key constraints, and in some cases its default-value generating attributes.   To use ``autoload=True``, the table's ``MetaData`` object need be bound to an ``Engine`` or ``Connection``, or alternatively the ``autoload_with=<some connectable>`` argument can be passed.  Below we illustrate autoloading a table and then iterating through the names of its columns::
-
-    >>> messages = Table('messages', meta, autoload=True)
-    >>> [c.name for c in messages.columns]
-    ['message_id', 'message_name', 'date']
-
-Note that if a reflected table has a foreign key referencing another table, the related ``Table`` object  will be automatically created within the ``MetaData`` object if it does not exist already.  Below, suppose table ``shopping_cart_items`` references a table ``shopping_carts``.  After reflecting, the ``shopping carts`` table is present:
-
-.. sourcecode:: pycon+sql
-
-    >>> shopping_cart_items = Table('shopping_cart_items', meta, autoload=True)
-    >>> 'shopping_carts' in meta.tables:
-    True
-        
-To get direct access to 'shopping_carts', simply instantiate it via the ``Table`` constructor.  ``Table`` uses a special constructor that will return the already created ``Table`` instance if it's already present:
-
-.. sourcecode:: python+sql
-
-    shopping_carts = Table('shopping_carts', meta)
-
-Of course, it's a good idea to use ``autoload=True`` with the above table regardless.  This is so that if it hadn't been loaded already, the operation will load the table.  The autoload operation only occurs for the table if it hasn't already been loaded; once loaded, new calls to ``Table`` will not re-issue any reflection queries.
-
-Overriding Reflected Columns 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-Individual columns can be overridden with explicit values when reflecting tables; this is handy for specifying custom datatypes, constraints such as primary keys that may not be configured within the database, etc.::
-
-    >>> mytable = Table('mytable', meta,
-    ... Column('id', Integer, primary_key=True),   # override reflected 'id' to have primary key
-    ... Column('mydata', Unicode(50)),    # override reflected 'mydata' to be Unicode
-    ... autoload=True)
-
-Reflecting All Tables at Once 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-The ``MetaData`` object can also get a listing of tables and reflect the full set.  This is achieved by using the ``reflect()`` method.  After calling it, all located tables are present within the ``MetaData`` object's dictionary of tables::
-
-    meta = MetaData()
-    meta.reflect(bind=someengine)
-    users_table = meta.tables['users']
-    addresses_table = meta.tables['addresses']
-    
-``metadata.reflect()`` is also a handy way to clear or drop all tables in a database::
-
-    meta = MetaData()
-    meta.reflect(bind=someengine)
-    for table in reversed(meta.sorted_tables):
-        someengine.execute(table.delete())
-
-Specifying the Schema Name 
----------------------------
-
-
-Some databases support the concept of multiple schemas.  A ``Table`` can reference this by specifying the ``schema`` keyword argument::
-
-    financial_info = Table('financial_info', meta,
-        Column('id', Integer, primary_key=True),
-        Column('value', String(100), nullable=False),
-        schema='remote_banks'
-    )
-
-Within the ``MetaData`` collection, this table will be identified by the combination of ``financial_info`` and ``remote_banks``.  If another table called ``financial_info`` is referenced without the ``remote_banks`` schema, it will refer to a different ``Table``.  ``ForeignKey`` objects can reference columns in this table using the form ``remote_banks.financial_info.id``.
-
-ON UPDATE and ON DELETE 
-------------------------
-
-
-``ON UPDATE`` and ``ON DELETE`` clauses to a table create are specified within the ``ForeignKeyConstraint`` object, using the ``onupdate`` and ``ondelete`` keyword arguments::
-
-    foobar = Table('foobar', meta,
-        Column('id', Integer, primary_key=True),
-        Column('lala', String(40)),
-        ForeignKeyConstraint(['lala'],['hoho.lala'], onupdate="CASCADE", ondelete="CASCADE"))
-
-Note that these clauses are not supported on SQLite, and require ``InnoDB`` tables when used with MySQL.  They may also not be supported on other databases.
-
-Other Options 
---------------
-
-``Tables`` may support database-specific options, such as MySQL's ``engine`` option that can specify "MyISAM", "InnoDB", and other backends for the table::
-
-    addresses = Table('engine_email_addresses', meta,
-        Column('address_id', Integer, primary_key = True),
-        Column('remote_user_id', Integer, ForeignKey(users.c.user_id)),
-        Column('email_address', String(20)),
-        mysql_engine='InnoDB'
-    )
-    
 Creating and Dropping Database Tables 
-======================================
+-------------------------------------
 
-Creating and dropping individual tables can be done via the ``create()`` and ``drop()`` methods of ``Table``; these methods take an optional ``bind`` parameter which references an ``Engine`` or a ``Connection``.  If not supplied, the ``Engine`` bound to the ``MetaData`` will be used, else an error is raised:
+Once you've defined some ``Table`` objects, assuming you're working with a brand new database one thing you might want to do is issue CREATE statements for those tables and their related constructs (as an aside, it's also quite possible that you *don't* want to do this, if you already have some preferred methodology such as tools included with your database or an existing scripting system - if that's the case, feel free to skip this section - SQLAlchemy has no requirement that it be used to create your tables).
+
+The usual way to issue CREATE is to use ``create_all()`` on the ``MetaData`` object.  This method will issue queries that first check for the existence of each individual table, and if not found will issue the CREATE statements:
+
+    .. sourcecode:: python+sql
+
+        engine = create_engine('sqlite:///:memory:')
+    
+        metadata = MetaData()
+    
+        user = Table('user', metadata, 
+            Column('user_id', Integer, primary_key = True),
+            Column('user_name', String(16), nullable = False),
+            Column('email_address', String(60), key='email'),
+            Column('password', String(20), nullable = False)
+        )
+    
+        user_prefs = Table('user_prefs', metadata, 
+            Column('pref_id', Integer, primary_key=True),
+            Column('user_id', Integer, ForeignKey("user.user_id"), nullable=False),
+            Column('pref_name', String(40), nullable=False),
+            Column('pref_value', String(100))
+        )
+    
+        {sql}metadata.create_all(engine)
+        PRAGMA table_info(user){}
+        CREATE TABLE user(
+                user_id INTEGER NOT NULL PRIMARY KEY, 
+                user_name VARCHAR(16) NOT NULL, 
+                email_address VARCHAR(60), 
+                password VARCHAR(20) NOT NULL
+        )
+        PRAGMA table_info(user_prefs){}
+        CREATE TABLE user_prefs(
+                pref_id INTEGER NOT NULL PRIMARY KEY, 
+                user_id INTEGER NOT NULL REFERENCES user(user_id), 
+                pref_name VARCHAR(40) NOT NULL, 
+                pref_value VARCHAR(100)
+        )
+
+``create_all()`` creates foreign key constraints between tables usually inline with the table definition itself, and for this reason it also generates the tables in order of their dependency.   There are options to change this behavior such that ``ALTER TABLE`` is used instead.
+
+Dropping all tables is similarly achieved using the ``drop_all()`` method.  This method does the exact opposite of ``create_all()`` - the presence of each table is checked first, and tables are dropped in reverse order of dependency.
+
+Creating and dropping individual tables can be done via the ``create()`` and ``drop()`` methods of ``Table``.  These methods by default issue the CREATE or DROP regardless of the table being present:
 
 .. sourcecode:: python+sql
 
+    engine = create_engine('sqlite:///:memory:')
+
     meta = MetaData()
-    meta.bind = 'sqlite:///:memory:'
 
     employees = Table('employees', meta, 
         Column('employee_id', Integer, primary_key=True),
         Column('employee_name', String(60), nullable=False, key='name'),
         Column('employee_dept', Integer, ForeignKey("departments.department_id"))
     )
-    {sql}employees.create()
+    {sql}employees.create(engine)
     CREATE TABLE employees(
     employee_id SERIAL NOT NULL PRIMARY KEY,
     employee_name VARCHAR(60) NOT NULL,
@@ -278,64 +176,192 @@ Creating and dropping individual tables can be done via the ``create()`` and ``d
 
 .. sourcecode:: python+sql
 
-    {sql}employees.drop(bind=e)
+    {sql}employees.drop(engine)
     DROP TABLE employees
     {}            
 
-The ``create()`` and ``drop()`` methods also support an optional keyword argument ``checkfirst`` which will issue the database's appropriate pragma statements to check if the table exists before creating or dropping::
+To enable the "check first for the table existing" logic, add the ``checkfirst=True`` argument to ``create()`` or ``drop()``::
 
-    employees.create(bind=e, checkfirst=True)
-    employees.drop(checkfirst=False)
+    employees.create(engine, checkfirst=True)
+    employees.drop(engine, checkfirst=False)
     
-Entire groups of Tables can be created and dropped directly from the ``MetaData`` object with ``create_all()`` and ``drop_all()``.  These methods always check for the existence of each table before creating or dropping.  Each method takes an optional ``bind`` keyword argument which can reference an ``Engine`` or a ``Connection``.  If no engine is specified, the underlying bound ``Engine``,  if any, is used:
 
-.. sourcecode:: python+sql
+Binding MetaData to an Engine or Connection 
+--------------------------------------------
 
-    engine = create_engine('sqlite:///:memory:')
+Notice in the previous section the creator/dropper methods accept an argument for the database engine in use.  When a schema construct is combined with an ``Engine`` object, or an individual ``Connection`` object, we call this the *bind*.   In the above examples the bind is associated with the schema construct only for the duration of the operation.   However, the option exists to persistently associate a bind with a set of schema constructs via the ``MetaData`` object's ``bind`` attribute::
+
+    engine = create_engine('sqlite://')
     
-    metadata = MetaData()
+    # create MetaData 
+    meta = MetaData()
+
+    # bind to an engine
+    meta.bind = engine
+
+We can now call methods like ``create_all()`` without needing to pass the ``Engine``::
+
+    meta.create_all()
     
-    users = Table('users', metadata, 
-        Column('user_id', Integer, primary_key = True),
-        Column('user_name', String(16), nullable = False),
-        Column('email_address', String(60), key='email'),
-        Column('password', String(20), nullable = False)
+The MetaData's bind is used for anything that requires an active connection, such as loading the definition of a table from the database automatically (called *reflection*)::
+    
+    # describe a table called 'users', query the database for its columns
+    users_table = Table('users', meta, autoload=True)
+
+As well as for executing SQL constructs that are derived from that MetaData's table objects::
+
+    # generate a SELECT statement and execute
+    result = users_table.select().execute()
+
+Binding the MetaData to the Engine is a **completely optional** feature.   The above operations can be achieved without the persistent bind using parameters::
+
+    # describe a table called 'users', query the database for its columns
+    users_table = Table('users', meta, autoload=True, autoload_with=engine)
+
+    # generate a SELECT statement and execute
+    result = engine.execute(users_table.select())
+
+Should you use bind ?   It's probably best to start without it.   If you find yourself constantly needing to specify the same ``Engine`` object throughout the entire application, consider binding as a convenience feature which is applicable to applications that don't have multiple engines in use and don't have the need to reference connections explicitly.    It should also be noted that an application which is focused on using the SQLAlchemy ORM will not be dealing explicitly with ``Engine`` or ``Connection`` objects very much in any case, so it's probably less confusing and more "future proof" to not use the `bind` attribute.
+
+Reflecting Tables
+-----------------
+
+A ``Table`` object can be instructed to load information about itself from the corresponding database schema object already existing within the database.  This process is called *reflection*.   Most simply you need only specify the table name, a ``MetaData`` object, and the ``autoload=True`` flag.  If the ``MetaData`` is not persistently bound, also add the ``autoload_with`` argument::
+
+    >>> messages = Table('messages', meta, autoload=True, autoload_with=engine)
+    >>> [c.name for c in messages.columns]
+    ['message_id', 'message_name', 'date']
+
+The above operation will use the given engine to query the database for information about the ``messages`` table, and will then generate ``Column``, ``ForeignKey``, and other objects corresponding to this information as though the ``Table`` object were hand-constructed in Python.
+
+When tables are reflected, if a given table references another one via foreign key, a second ``Table`` object is created within the ``MetaData`` object representing the connection.   Below, assume the table ``shopping_cart_items`` references a table named ``shopping_carts``.   Reflecting the ``shopping_cart_items`` table has the effect such that the ``shopping_carts`` table will also be loaded::
+
+    >>> shopping_cart_items = Table('shopping_cart_items', meta, autoload=True, autoload_with=engine)
+    >>> 'shopping_carts' in meta.tables:
+    True
+        
+The ``MetaData`` has an interesting "singleton-like" behavior such that if you requested both tables individually, ``MetaData`` will ensure that exactly one ``Table`` object is created for each distinct table name.  The ``Table`` constructor actually returns to you the already-existing ``Table`` object if one already exists with the given name.  Such as below, we can access the already generated ``shopping_carts`` table just by naming it::
+
+    shopping_carts = Table('shopping_carts', meta)
+
+Of course, it's a good idea to use ``autoload=True`` with the above table regardless.  This is so that the table's attributes will be loaded if they have not been already.  The autoload operation only occurs for the table if it hasn't already been loaded; once loaded, new calls to ``Table`` with the same name will not re-issue any reflection queries.
+
+Overriding Reflected Columns 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Individual columns can be overridden with explicit values when reflecting tables; this is handy for specifying custom datatypes, constraints such as primary keys that may not be configured within the database, etc.::
+
+    >>> mytable = Table('mytable', meta,
+    ... Column('id', Integer, primary_key=True),   # override reflected 'id' to have primary key
+    ... Column('mydata', Unicode(50)),    # override reflected 'mydata' to be Unicode
+    ... autoload=True)
+
+Reflecting Views
+~~~~~~~~~~~~~~~~
+
+The reflection system can also reflect views.   Basic usage is the same as that of a table::
+
+    my_view = Table("some_view", metadata, autoload=True)
+    
+Above, ``my_view`` is a ``Table`` object with ``Column`` objects representing the names and types
+of each column within the view "some_view".
+
+Usually, it's desired to have at least a primary key constraint when reflecting a view, if not
+foreign keys as well.  View reflection doesn't extrapolate these constraints.
+
+Use the "override" technique for this, specifying explicitly those columns
+which are part of the primary key or have foreign key constraints::
+
+    my_view = Table("some_view", metadata, 
+                    Column("view_id", Integer, primary_key=True),
+                    Column("related_thing", Integer, ForeignKey("othertable.thing_id")),
+                    autoload=True
+    )
+
+Reflecting All Tables at Once 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``MetaData`` object can also get a listing of tables and reflect the full set.  This is achieved by using the ``reflect()`` method.  After calling it, all located tables are present within the ``MetaData`` object's dictionary of tables::
+
+    meta = MetaData()
+    meta.reflect(bind=someengine)
+    users_table = meta.tables['users']
+    addresses_table = meta.tables['addresses']
+    
+``metadata.reflect()`` also provides a handy way to clear or delete all the rows in a database::
+
+    meta = MetaData()
+    meta.reflect(bind=someengine)
+    for table in reversed(meta.sorted_tables):
+        someengine.execute(table.delete())
+
+Specifying the Schema Name 
+---------------------------
+
+Some databases support the concept of multiple schemas.  A ``Table`` can reference this by specifying the ``schema`` keyword argument::
+
+    financial_info = Table('financial_info', meta,
+        Column('id', Integer, primary_key=True),
+        Column('value', String(100), nullable=False),
+        schema='remote_banks'
+    )
+
+Within the ``MetaData`` collection, this table will be identified by the combination of ``financial_info`` and ``remote_banks``.  If another table called ``financial_info`` is referenced without the ``remote_banks`` schema, it will refer to a different ``Table``.  ``ForeignKey`` objects can specify references to columns in this table using the form ``remote_banks.financial_info.id``.
+
+The ``schema`` argument should be used for any name qualifiers required, including Oracle's "owner" attribute and similar.  It also can accommodate a dotted name for longer schemes::
+
+    schema="dbo.scott"
+
+Backend-Specific Options 
+------------------------
+
+``Table`` supports database-specific options.   For example, MySQL has different table backend types, including "MyISAM" and "InnoDB".   This can be expressed with ``Table`` using ``mysql_engine``::
+
+    addresses = Table('engine_email_addresses', meta,
+        Column('address_id', Integer, primary_key = True),
+        Column('remote_user_id', Integer, ForeignKey(users.c.user_id)),
+        Column('email_address', String(20)),
+        mysql_engine='InnoDB'
     )
     
-    user_prefs = Table('user_prefs', metadata, 
-        Column('pref_id', Integer, primary_key=True),
-        Column('user_id', Integer, ForeignKey("users.user_id"), nullable=False),
-        Column('pref_name', String(40), nullable=False),
-        Column('pref_value', String(100))
-    )
-    
-    {sql}metadata.create_all(bind=engine)
-    PRAGMA table_info(users){}
-    CREATE TABLE users(
-            user_id INTEGER NOT NULL PRIMARY KEY, 
-            user_name VARCHAR(16) NOT NULL, 
-            email_address VARCHAR(60), 
-            password VARCHAR(20) NOT NULL
-    )
-    PRAGMA table_info(user_prefs){}
-    CREATE TABLE user_prefs(
-            pref_id INTEGER NOT NULL PRIMARY KEY, 
-            user_id INTEGER NOT NULL REFERENCES users(user_id), 
-            pref_name VARCHAR(40) NOT NULL, 
-            pref_value VARCHAR(100)
-    )
+Other backends may support table-level options as well.  See the API documentation for each backend for further details.
 
 Column Insert/Update Defaults 
 ==============================
+
+SQLAlchemy provides a very rich featureset regarding column level events which take place during INSERT and UPDATE statements.  Options include:
+ 
+* Scalar values used as defaults during INSERT and UPDATE operations
+* Python functions which execute upon INSERT and UPDATE operations
+* SQL expressions which are embedded in INSERT statements (or in some cases execute beforehand)
+* SQL expressions which are embedded in UPDATE statements
+* Server side default values used during INSERT
+* Markers for server-side triggers used during UPDATE
+ 
+The general rule for all insert/update defaults is that they only take effect if no value for a particular column is passed as an ``execute()`` parameter; otherwise, the given value is used.
+
+Scalar Defaults
+---------------
+
+The simplest kind of default is a scalar value used as the default value of a column::
+
+    Table("mytable", meta,
+        Column("somecolumn", Integer, default=12)
+    )
     
+Above, the value "12" will be bound as the column value during an INSERT if no other value is supplied.
 
-SQLAlchemy includes several constructs which provide default values provided during INSERT and UPDATE statements.  The defaults may be provided as Python constants, Python functions, or SQL expressions, and the SQL expressions themselves may be "pre-executed", executed inline within the insert/update statement itself, or can be created as a SQL level "default" placed on the table definition itself.  A "default" value by definition is only invoked if no explicit value is passed into the INSERT or UPDATE statement.
+A scalar value may also be associated with an UPDATE statement, though this is not very common (as UPDATE statements are usually looking for dynamic defaults)::
 
-Pre-Executed Python Functions 
-------------------------------
+    Table("mytable", meta,
+        Column("somecolumn", Integer, onupdate=25)
+    )
 
 
-The "default" keyword argument on Column can reference a Python value or callable which is invoked at the time of an insert::
+Python-Executed Functions 
+-------------------------
+
+The ``default`` and ``onupdate`` keyword arguments also accept Python functions.   These functions are invoked at the time of insert or update if no other value for that column is supplied, and the value returned is used for the column's value.  Below illustrates a crude "sequence" that assigns an incrementing counter to a primary key column::
 
     # a function which counts upwards
     i = 0
@@ -345,16 +371,12 @@ The "default" keyword argument on Column can reference a Python value or callabl
         return i
 
     t = Table("mytable", meta, 
-        # function-based default
         Column('id', Integer, primary_key=True, default=mydefault),
-    
-        # a scalar default
-        Column('key', String(10), default="default")
     )
 
-Similarly, the "onupdate" keyword does the same thing for update statements:
+It should be noted that for real "incrementing sequence" behavior, the built-in capabilities of the database should normally be used, which may include sequence objects or other autoincrementing capabilities.  For primary key columns, SQLAlchemy will in most cases use these capabilities automatically.   See the API documentation for ``Column`` including the ``autoincrement`` flag, as well as the section on ``Sequence`` later in this chapter for background on standard primary key generation techniques.
 
-.. sourcecode:: python+sql
+To illustrate onupdate, we assign the Python ``datetime`` function ``now`` to the ``onupdate`` attribute::
 
     import datetime
     
@@ -365,13 +387,29 @@ Similarly, the "onupdate" keyword does the same thing for update statements:
         Column('last_updated', DateTime, onupdate=datetime.datetime.now),
     )
 
-Pre-executed and Inline SQL Expressions 
-----------------------------------------
+When an update statement executes and no value is passed for ``last_updated``, the ``datetime.datetime.now()`` Python function is executed and its return value used as the value for ``last_updated``.   Notice that we provide ``now`` as the function itself without calling it (i.e. there are no parenthesis following) - SQLAlchemy will execute the function at the time the statement executes.
 
+Context-Sensitive Default Functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The "default" and "onupdate" keywords may also be passed SQL expressions, including select statements or direct function calls:
+The Python functions used by ``default`` and ``onupdate`` may also make use of the current statement's context in order to determine a value.  The `context` of a statement is an internal SQLAlchemy object which contains all information about the statement being executed, including its source expression, the parameters associated with it and the cursor.  The typical use case for this context with regards to default generation is to have access to the other values being inserted or updated on the row.   To access the context, provide a function that accepts a single ``context`` argument::
 
-.. sourcecode:: python+sql
+    def mydefault(context):
+        return context.current_parameters['counter'] + 12
+        
+    t = Table('mytable', meta,
+        Column('counter', Integer),
+        Column('counter_plus_twelve', Integer, default=mydefault, onupdate=mydefault)
+    )
+
+Above we illustrate a default function which will execute for all INSERT and UPDATE statements where a value for ``counter_plus_twelve`` was otherwise not provided, and the value will be that of whatever value is present in the execution for the ``counter`` column, plus the number 12.
+
+While the context object passed to the default function has many attributes, the ``current_parameters`` member is a special member provided only during the execution of a default function for the purposes of deriving defaults from its existing values.  For a single statement that is executing many sets of bind parameters, the user-defined function is called for each set of parameters, and ``current_parameters`` will be provided with each individual parameter set for each execution.
+
+SQL Expressions 
+---------------
+
+The "default" and "onupdate" keywords may also be passed SQL expressions, including select statements or direct function calls::
 
     t = Table("mytable", meta, 
         Column('id', Integer, primary_key=True),
@@ -380,29 +418,36 @@ The "default" and "onupdate" keywords may also be passed SQL expressions, includ
         Column('create_date', DateTime, default=func.now()),
     
         # define 'key' to pull its default from the 'keyvalues' table
-        Column('key', String(20), default=keyvalues.select(keyvalues.c.type='type1', limit=1))
+        Column('key', String(20), default=keyvalues.select(keyvalues.c.type='type1', limit=1)),
 
         # define 'last_modified' to use the current_timestamp SQL function on update
-        Column('last_modified', DateTime, onupdate=func.current_timestamp())
+        Column('last_modified', DateTime, onupdate=func.utc_timestamp())
         )
 
-The above SQL functions are usually executed "inline" with the INSERT or UPDATE statement being executed.  In some cases, the function is "pre-executed" and its result pre-fetched explicitly.  This happens under the following circumstances:
+Above, the ``create_date`` column will be populated with the result of the ``now()`` SQL function (which, depending on backend, compiles into ``NOW()`` or ``CURRENT_TIMESTAMP`` in most cases) during an INSERT statement, and the ``key`` column with the result of a SELECT subquery from another table.   The ``last_modified`` column will be populated with the value of ``UTC_TIMESTAMP()``, a function specific to MySQL, when an UPDATE statement is emitted for this table.
+
+Note that when using ``func`` functions, unlike when using Python `datetime` functions we *do* call the function, i.e. with parenthesis "()" - this is because what we want in this case is the return value of the function, which is the SQL expression construct that will be rendered into the INSERT or UPDATE statement.
+
+The above SQL functions are usually executed "inline" with the INSERT or UPDATE statement being executed, meaning, a single statement is executed which embeds the given expressions or subqueries within the VALUES or SET clause of the statement.  Although in some cases, the function is "pre-executed" in a SELECT statement of its own beforehand.  This happens when all of the following is true:
 
 * the column is a primary key column
 
-* the database dialect does not support a usable ``cursor.lastrowid`` accessor (or equivalent); this currently includes PostgreSQL, Oracle, and Firebird.
+* the database dialect does not support a usable ``cursor.lastrowid`` accessor (or equivalent); this currently includes PostgreSQL, Oracle, and Firebird, as well as some MySQL dialects.
+
+* the dialect does not support the "RETURNING" clause or similar, or the ``implicit_returning`` flag is set to ``False`` for the dialect.  Dialects which support RETURNING currently include Postgresql, Oracle, Firebird, and MS-SQL.
 
 * the statement is a single execution, i.e. only supplies one set of parameters and doesn't use "executemany" behavior
 
-* the ``inline=True`` flag is not set on the ``Insert()`` or ``Update()`` construct.
+* the ``inline=True`` flag is not set on the ``Insert()`` or ``Update()`` construct, and the statement has not defined an explicit `returning()` clause.
 
-For a statement execution which is not an executemany, the returned ``ResultProxy`` will contain a collection accessible via ``result.postfetch_cols()`` which contains a list of all ``Column`` objects which had an inline-executed default.  Similarly, all parameters which were bound to the statement, including all Python and SQL expressions which were pre-executed, are present in the ``last_inserted_params()`` or ``last_updated_params()`` collections on ``ResultProxy``.  The ``last_inserted_ids()`` collection contains a list of primary key values for the row inserted.  
+Whether or not the default generation clause "pre-executes" is not something that normally needs to be considered, unless it is being addressed for performance reasons. 
 
-DDL-Level Defaults 
--------------------
+When the statement is executed with a single set of parameters (that is, it is not an "executemany" style execution), the returned ``ResultProxy`` will contain a collection accessible via ``result.postfetch_cols()`` which contains a list of all ``Column`` objects which had an inline-executed default.  Similarly, all parameters which were bound to the statement, including all Python and SQL expressions which were pre-executed, are present in the ``last_inserted_params()`` or ``last_updated_params()`` collections on ``ResultProxy``.  The ``inserted_primary_key`` collection contains a list of primary key values for the row inserted (a list so that single-column and composite-column primary keys are represented in the same format).  
 
+Server Side Defaults 
+--------------------
 
-A variant on a SQL expression default is the ``server_default``, which gets placed in the CREATE TABLE statement during a ``create()`` operation:
+A variant on the SQL expression default is the ``server_default``, which gets placed in the CREATE TABLE statement during a ``create()`` operation:
 
 .. sourcecode:: python+sql
 
@@ -426,19 +471,18 @@ Triggered Columns
 Columns with values set by a database trigger or other external process may be called out with a marker::
 
     t = Table('test', meta,
-        Column('abc', String(20), server_default=FetchedValue())
+        Column('abc', String(20), server_default=FetchedValue()),
         Column('def', String(20), server_onupdate=FetchedValue())
     )
 
-These markers do not emit a ````default```` clause when the table is created, however they do set the same internal flags as a static ``server_default`` clause, providing hints to higher-level tools that a "post-fetch" of these rows should be performed after an insert or update.
+These markers do not emit a "default" clause when the table is created, however they do set the same internal flags as a static ``server_default`` clause, providing hints to higher-level tools that a "post-fetch" of these rows should be performed after an insert or update.
 
 Defining Sequences 
 -------------------
 
+SQLAlchemy represents database sequences using the ``Sequence`` object, which is considered to be a special case of "column default".   It only has an effect on databases which have explicit support for sequences, which currently includes Postgresql, Oracle, and Firebird.  The ``Sequence`` object is otherwise ignored.
 
-A table with a sequence looks like:
-
-.. sourcecode:: python+sql
+The ``Sequence`` may be placed on any column as a "default" generator to be used during INSERT operations, and can also be configured to fire off during UPDATE operations if desired.  It is most commonly used in conjunction with a single integer primary key column::
 
     table = Table("cartitems", meta, 
         Column("cart_id", Integer, Sequence('cart_id_seq'), primary_key=True),
@@ -446,15 +490,13 @@ A table with a sequence looks like:
         Column("createdate", DateTime())
     )
 
-The ``Sequence`` object works a lot like the ``default`` keyword on ``Column``, except that it only takes effect on a database which supports sequences.  When used with a database that does not support sequences, the ``Sequence`` object has no effect; therefore it's safe to place on a table which is used against multiple database backends.  The same rules for pre- and inline execution apply.
+Where above, the table "cartitems" is associated with a sequence named "cart_id_seq".   When INSERT statements take place for "cartitems", and no value is passed for the "cart_id" column, the "cart_id_seq" sequence will be used to generate a value.
 
 When the ``Sequence`` is associated with a table, CREATE and DROP statements issued for that table will also issue CREATE/DROP for the sequence object as well, thus "bundling" the sequence object with its parent table.
 
-The flag ``optional=True`` on ``Sequence`` will produce a sequence that is only used on databases which have no "autoincrementing" capability.  For example, PostgreSQL supports primary key generation using the SERIAL keyword, whereas Oracle has no such capability.  Therefore, a ``Sequence`` placed on a primary key column with ``optional=True`` will only be used with an Oracle backend but not PostgreSQL.
+The ``Sequence`` object also implements special functionality to accommodate Postgresql's SERIAL datatype.   The SERIAL type in PG automatically generates a sequence that is used implicitly during inserts.  This means that if a ``Table`` object defines a ``Sequence`` on its primary key column so that it works with Oracle and Firebird, the ``Sequence`` would get in the way of the "implicit" sequence that PG would normally use.  For this use case, add the flag ``optional=True`` to the ``Sequence`` object - this indicates that the ``Sequence`` should only be used if the database provides no other option for generating primary key identifiers.
 
-A sequence can also be executed standalone, using an ``Engine`` or ``Connection``, returning its next value in a database-independent fashion:
-
-.. sourcecode:: python+sql
+The ``Sequence`` object also has the ability to be executed standalone like a SQL expression, which has the effect of calling its "next value" function::
 
     seq = Sequence('some_sequence')
     nextid = connection.execute(seq)
@@ -462,11 +504,101 @@ A sequence can also be executed standalone, using an ``Engine`` or ``Connection`
 Defining Constraints and Indexes 
 =================================
 
+.. _metadata_foreignkeys:
+
+Defining Foreign Keys
+---------------------
+
+A *foreign key* in SQL is a table-level construct that constrains one or more columns in that table to only allow values that are present in a different set of columns, typically but not always located on a different table.  We call the columns which are constrained the *foreign key* columns and the columns which they are constrained towards the *referenced* columns.  The referenced columns almost always define the primary key for their owning table, though there are exceptions to this.  The foreign key is the "joint" that connects together pairs of rows which have a relationship with each other, and SQLAlchemy assigns very deep importance to this concept in virtually every area of its operation.
+
+In SQLAlchemy as well as in DDL, foreign key constraints can be defined as additional attributes within the table clause, or for single-column foreign keys they may optionally be specified within the definition of a single column.   The single column foreign key is more common, and at the column level is specified by constructing a ``ForeignKey`` object as an argument to a ``Column`` object::
+
+    user_preference = Table('user_preference', metadata, 
+        Column('pref_id', Integer, primary_key=True),
+        Column('user_id', Integer, ForeignKey("user.user_id"), nullable=False),
+        Column('pref_name', String(40), nullable=False),
+        Column('pref_value', String(100))
+    )
+
+Above, we define a new table ``user_preference`` for which each row must contain a value in the ``user_id`` column that also exists in the ``user`` table's ``user_id`` column.
+
+The argument to ``ForeignKey`` is most commonly a string of the form *<tablename>.<columnname>*, or for a table in a remote schema or "owner" of the form *<schemaname>.<tablename>.<columnname>*.   It may also be an actual ``Column`` object, which as we'll see later is accessed from an existing ``Table`` object via its ``c`` collection::
+
+    ForeignKey(user.c.user_id)
+
+The advantage to using a string is that the in-python linkage between ``user`` and ``user_preference`` is resolved only when first needed, so that table objects can be easily spread across multiple modules and defined in any order.
+
+Foreign keys may also be defined at the table level, using the ``ForeignKeyConstraint`` object.   This object can describe a single- or multi-column foreign key.  A multi-column foreign key is known as a *composite* foreign key, and almost always references a table that has a composite primary key.  Below we define a table ``invoice`` which has a composite primary key::
+
+    invoice = Table('invoice', metadata, 
+        Column('invoice_id', Integer, primary_key=True),
+        Column('ref_num', Integer, primary_key=True),
+        Column('description', String(60), nullable=False)
+    )
+
+And then a table ``invoice_item`` with a composite foreign key referencing ``invoice``::
+
+    invoice_item = Table('invoice_item', metadata, 
+        Column('item_id', Integer, primary_key=True),
+        Column('item_name', String(60), nullable=False),
+        Column('invoice_id', Integer, nullable=False),
+        Column('ref_num', Integer, nullable=False),
+        ForeignKeyConstraint(['invoice_id', 'ref_num'], ['invoice.invoice_id', 'invoice.ref_num'])
+    )
+    
+It's important to note that the ``ForeignKeyConstraint`` is the only way to define a composite foreign key.   While we could also have placed individual ``ForeignKey`` objects on both the ``invoice_item.invoice_id`` and ``invoice_item.ref_num`` columns, SQLAlchemy would not be aware that these two values should be paired together - it would be two individual foreign key constraints instead of a single composite foreign key referencing two columns.
+
+Creating/Dropping Foreign Key Constraints via ALTER
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In all the above examples, the ``ForeignKey`` object causes the "REFERENCES" keyword to be added inline to a column definition within a "CREATE TABLE" statement when ``create_all()`` is issued, and ``ForeignKeyConstraint`` invokes the "CONSTRAINT" keyword inline with "CREATE TABLE".    There are some cases where this is undesireable, particularly when two tables reference each other mutually, each with a foreign key referencing the other.   In such a situation at least one of the foreign key constraints must be generated after both tables have been built.  To support such a scheme, ``ForeignKey`` and ``ForeignKeyConstraint`` offer the flag ``use_alter=True``.  When using this flag, the constraint will be generated using a definition similar to "ALTER TABLE <tablename> ADD CONSTRAINT <name> ...".   Since a name is required, the ``name`` attribute must also be specified.  For example::
+
+    node = Table('node', meta,
+        Column('node_id', Integer, primary_key=True),
+        Column('primary_element', Integer, 
+            ForeignKey('element.element_id', use_alter=True, name='fk_node_element_id')
+        )
+    )
+    
+    element = Table('element', meta,
+        Column('element_id', Integer, primary_key=True),
+        Column('parent_node_id', Integer),
+        ForeignKeyConstraint(
+            ['parent_node_id'], 
+            ['node.node_id'], 
+            use_alter=True, 
+            name='fk_element_parent_node_id'
+        )
+    )
+
+ON UPDATE and ON DELETE 
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Most databases support *cascading* of foreign key values, that is the when a parent row is updated the new value is placed in child rows, or when the parent row is deleted all corresponding child rows are set to null or deleted.  In data definition language these are specified using phrases like "ON UPDATE CASCADE", "ON DELETE CASCADE", and "ON DELETE SET NULL", corresponding to foreign key constraints.  The phrase after "ON UPDATE" or "ON DELETE" may also other allow other phrases that are specific to the database in use.  The ``ForeignKey`` and ``ForeignKeyConstraint`` objects support the generation of this clause via the ``onupdate`` and ``ondelete`` keyword arguments.  The value is any string which will be output after the appropriate "ON UPDATE" or "ON DELETE" phrase::
+
+    child = Table('child', meta,
+        Column('id', Integer, 
+                ForeignKey('parent.id', onupdate="CASCADE", ondelete="CASCADE"), 
+                primary_key=True
+        )
+    )
+    
+    composite = Table('composite', meta,
+        Column('id', Integer, primary_key=True),
+        Column('rev_id', Integer),
+        Column('note_id', Integer),
+        ForeignKeyConstraint(
+                    ['rev_id', 'note_id'],
+                    ['revisions.id', 'revisions.note_id'], 
+                    onupdate="CASCADE", ondelete="SET NULL"
+        )
+    )
+
+Note that these clauses are not supported on SQLite, and require ``InnoDB`` tables when used with MySQL.  They may also not be supported on other databases.
 
 UNIQUE Constraint
 -----------------
 
-
 Unique constraints can be created anonymously on a single column using the ``unique`` keyword on ``Column``.  Explicitly named unique constraints and/or those with multiple columns are created via the ``UniqueConstraint`` table-level construct.
 
 .. sourcecode:: python+sql
@@ -487,7 +619,6 @@ Unique constraints can be created anonymously on a single column using the ``uni
 CHECK Constraint
 ----------------
 
-
 Check constraints can be named or unnamed and can be created at the Column or Table level, using the ``CheckConstraint`` construct.  The text of the check constraint is passed directly through to the database, so there is limited "database independent" behavior.  Column level check constraints generally should only refer to the column to which they are placed, while table level constraints can refer to any columns in the table.
 
 Note that some databases do not actively support check constraints such as MySQL and SQLite.
@@ -507,14 +638,23 @@ Note that some databases do not actively support check constraints such as MySQL
         CheckConstraint('col2 > col3 + 5', name='check1')
         )
     
+    {sql}mytable.create(engine)
+    CREATE TABLE mytable (
+        col1 INTEGER  CHECK (col1>5), 
+        col2 INTEGER, 
+        col3 INTEGER, 
+        CONSTRAINT check1  CHECK (col2 > col3 + 5)
+    ){stop}
+    
 Indexes
 -------
 
-
-Indexes can be created anonymously (using an auto-generated name "ix_&lt;column label&gt;") for a single column using the inline ``index`` keyword on ``Column``, which also modifies the usage of ``unique`` to apply the uniqueness to the index itself, instead of adding a separate UNIQUE constraint.  For indexes with specific names or which encompass more than one column, use the ``Index`` construct, which requires a name.  
+Indexes can be created anonymously (using an auto-generated name ``ix_<column label>``) for a single column using the inline ``index`` keyword on ``Column``, which also modifies the usage of ``unique`` to apply the uniqueness to the index itself, instead of adding a separate UNIQUE constraint.  For indexes with specific names or which encompass more than one column, use the ``Index`` construct, which requires a name.  
 
 Note that the ``Index`` construct is created **externally** to the table which it corresponds, using ``Column`` objects and not strings.
 
+Below we illustrate a ``Table`` with several ``Index`` objects associated.  The DDL for "CREATE INDEX" is issued right after the create statements for the table:
+
 .. sourcecode:: python+sql
 
     meta = MetaData()
@@ -538,18 +678,150 @@ Note that the ``Index`` construct is created **externally** to the table which i
     # place a unique index on col5, col6
     Index('myindex', mytable.c.col5, mytable.c.col6, unique=True)
 
-The ``Index`` objects will be created along with the CREATE statements for the table itself.  An index can also be created on its own independently of the table:
+    {sql}mytable.create(engine)
+    CREATE TABLE mytable (
+        col1 INTEGER, 
+        col2 INTEGER, 
+        col3 INTEGER, 
+        col4 INTEGER, 
+        col5 INTEGER, 
+        col6 INTEGER
+    )
+    CREATE INDEX ix_mytable_col1 ON mytable (col1)
+    CREATE UNIQUE INDEX ix_mytable_col2 ON mytable (col2)
+    CREATE UNIQUE INDEX myindex ON mytable (col5, col6)
+    CREATE INDEX idx_col34 ON mytable (col3, col4){stop}
+
+The ``Index`` object also supports its own ``create()`` method:
 
 .. sourcecode:: python+sql
 
-    # create a table
-    sometable.create()
+    i = Index('someindex', mytable.c.col5)
+    {sql}i.create(engine)
+    CREATE INDEX someindex ON mytable (col5){stop}
+    
+Customizing DDL
+===============
 
-    # define an index
-    i = Index('someindex', sometable.c.col5)
+In the preceding sections we've discussed a variety of schema constructs including ``Table``, ``ForeignKeyConstraint``, ``CheckConstraint``, and ``Sequence``.   Throughout, we've relied upon the ``create()`` and ``create_all()`` methods of ``Table`` and ``MetaData`` in order to issue data definition language (DDL) for all constructs.   When issued, a pre-determined order of operations is invoked, and DDL to create each table is created unconditionally including all constraints and other objects associated with it.   For more complex scenarios where database-specific DDL is required, SQLAlchemy offers two techniques which can be used to add any DDL based on any condition, either accompanying the standard generation of tables or by itself.
 
-    # create the index, will use the table's bound connectable if the ``bind`` keyword argument not specified
-    i.create()
+Controlling DDL Sequences
+-------------------------
+
+The ``sqlalchemy.schema`` package contains SQL expression constructs that provide DDL expressions.   For example, to produce a ``CREATE TABLE`` statement:
+
+.. sourcecode:: python+sql
+
+    from sqlalchemy.schema import CreateTable
+    {sql}engine.execute(CreateTable(mytable))
+    CREATE TABLE mytable (
+        col1 INTEGER, 
+        col2 INTEGER, 
+        col3 INTEGER, 
+        col4 INTEGER, 
+        col5 INTEGER, 
+        col6 INTEGER
+    ){stop}
+    
+Above, the ``CreateTable`` construct works like any other expression construct (such as ``select()``, ``table.insert()``, etc.).  A full reference of available constructs is in :ref:`schema_api_ddl`.
+
+The DDL constructs all extend a common base class which provides the capability to be associated with an individual ``Table`` or ``MetaData`` object, to be invoked upon create/drop events.   Consider the example of a table which contains a CHECK constraint:
+
+.. sourcecode:: python+sql
+
+    users = Table('users', metadata,
+                   Column('user_id', Integer, primary_key=True),
+                   Column('user_name', String(40), nullable=False),
+                   CheckConstraint('length(user_name) >= 8',name="cst_user_name_length")
+                   )
+
+    {sql}users.create(engine)
+    CREATE TABLE users (
+        user_id SERIAL NOT NULL, 
+        user_name VARCHAR(40) NOT NULL, 
+        PRIMARY KEY (user_id), 
+        CONSTRAINT cst_user_name_length  CHECK (length(user_name) >= 8)
+    ){stop}
+
+The above table contains a column "user_name" which is subject to a CHECK constraint that validates that the length of the string is at least eight characters.   When a ``create()`` is issued for this table, DDL for the ``CheckConstraint`` will also be issued inline within the table definition.
+
+The ``CheckConstraint`` construct can also be constructed externally and associated with the ``Table`` afterwards::
+
+    constraint = CheckConstraint('length(user_name) >= 8',name="cst_user_name_length")
+    users.append_constraint(constraint)
+    
+So far, the effect is the same.  However, if we create DDL elements corresponding to the creation and removal of this constraint, and associate them with the ``Table`` as events, these new events will take over the job of issuing DDL for the constraint.  Additionally, the constraint will be added via ALTER:
+
+.. sourcecode:: python+sql
+
+    AddConstraint(constraint).execute_at("after-create", users)
+    DropConstraint(constraint).execute_at("before-drop", users)
+
+    {sql}users.create(engine)
+    CREATE TABLE users (
+        user_id SERIAL NOT NULL, 
+        user_name VARCHAR(40) NOT NULL, 
+        PRIMARY KEY (user_id)
+    )
+
+    ALTER TABLE users ADD CONSTRAINT cst_user_name_length  CHECK (length(user_name) >= 8){stop}
+
+    {sql}users.drop(engine)
+    ALTER TABLE users DROP CONSTRAINT cst_user_name_length
+    DROP TABLE users{stop}
+    
+The real usefulness of the above becomes clearer once we illustrate the ``on`` attribute of a DDL event.  The ``on`` parameter is part of the constructor, and may be a string name of a database dialect name, a tuple containing dialect names, or a Python callable.   This will limit the execution of the item to just those dialects, or when the return value of the callable is ``True``.  So if our ``CheckConstraint`` was only supported by Postgresql and not other databases, we could limit it to just that dialect::
+
+    AddConstraint(constraint, on='postgresql').execute_at("after-create", users)
+    DropConstraint(constraint, on='postgresql').execute_at("before-drop", users)
+    
+Or to any set of dialects::
+
+    AddConstraint(constraint, on=('postgresql', 'mysql')).execute_at("after-create", users)
+    DropConstraint(constraint, on=('postgresql', 'mysql')).execute_at("before-drop", users)
+    
+When using a callable, the callable is passed the ddl element, event name, the ``Table`` or ``MetaData`` object whose "create" or "drop" event is in progress, and the ``Connection`` object being used for the operation, as well as additional information as keyword arguments.  The callable can perform checks, such as whether or not a given item already exists.  Below we define ``should_create()`` and ``should_drop()`` callables that check for the presence of our named constraint:
+
+.. sourcecode:: python+sql
+
+    def should_create(ddl, event, target, connection, **kw):
+        row = connection.execute("select conname from pg_constraint where conname='%s'" % ddl.element.name).scalar()
+        return not bool(row)
+
+    def should_drop(ddl, event, target, connection, **kw):
+        return not should_create(ddl, event, target, connection, **kw)
+        
+    AddConstraint(constraint, on=should_create).execute_at("after-create", users)
+    DropConstraint(constraint, on=should_drop).execute_at("before-drop", users)
+
+    {sql}users.create(engine)
+    CREATE TABLE users (
+        user_id SERIAL NOT NULL, 
+        user_name VARCHAR(40) NOT NULL, 
+        PRIMARY KEY (user_id)
+    )
+
+    select conname from pg_constraint where conname='cst_user_name_length'
+    ALTER TABLE users ADD CONSTRAINT cst_user_name_length  CHECK (length(user_name) >= 8){stop}
+    
+    {sql}users.drop(engine)
+    select conname from pg_constraint where conname='cst_user_name_length'
+    ALTER TABLE users DROP CONSTRAINT cst_user_name_length
+    DROP TABLE users{stop}
+
+Custom DDL
+----------
+
+Custom DDL phrases are most easily achieved using the :class:`~sqlalchemy.schema.DDL` construct.  This construct works like all the other DDL elements except it accepts a string which is the
+text to be emitted:
+
+.. sourcecode:: python+sql
+
+    DDL("ALTER TABLE users ADD CONSTRAINT "
+        "cst_user_name_length "
+        " CHECK (length(user_name) >= 8)").execute_at("after-create", metadata)
+
+A more comprehensive method of creating libraries of DDL constructs is to use the :ref:`sqlalchemy.ext.compiler_toplevel` extension.   See that chapter for full details.
 
 Adapting Tables to Alternate Metadata 
 ======================================

doc/build/ormtutorial.rst

@@ -8,11 +8,11 @@ In this tutorial we will cover a basic SQLAlchemy object-relational mapping scen
 Version Check
 =============
 
-A quick check to verify that we are on at least **version 0.5** of SQLAlchemy::
+A quick check to verify that we are on at least **version 0.6** of SQLAlchemy::
 
     >>> import sqlalchemy
     >>> sqlalchemy.__version__ # doctest:+SKIP
-    0.5.0
+    0.6.0
 
 Connecting
 ==========
@@ -316,6 +316,8 @@ issuing a SELECT illustrates the changes made to the database:
     ['ed', 'fakeuser']
     {stop}[<User('ed','Ed Jones', 'f8s7ccs')>]
 
+.. _ormtutorial_querying:
+
 Querying
 ========
 
@@ -372,7 +374,7 @@ You can control the names using the ``label()`` construct for scalar attributes
     ...    print row.user_alias, row.name_label
     SELECT users_1.id AS users_1_id, users_1.name AS users_1_name, users_1.fullname AS users_1_fullname, users_1.password AS users_1_password, users_1.name AS name_label
     FROM users AS users_1
-    []
+    []{stop}
     <User('ed','Ed Jones', 'f8s7ccs')> ed
     <User('wendy','Wendy Williams', 'foobar')> wendy
     <User('mary','Mary Contrary', 'xxg527')> mary
@@ -387,8 +389,8 @@ Basic operations with ``Query`` include issuing LIMIT and OFFSET, most convenien
     SELECT users.id AS users_id, users.name AS users_name, users.fullname AS users_fullname, users.password AS users_password
     FROM users ORDER BY users.id
     LIMIT 2 OFFSET 1
-    []
-    {stop}<User('wendy','Wendy Williams', 'foobar')>
+    []{stop}
+    <User('wendy','Wendy Williams', 'foobar')>
     <User('mary','Mary Contrary', 'xxg527')>
 
 and filtering results, which is accomplished either with ``filter_by()``, which uses keyword arguments:
@@ -447,10 +449,22 @@ Here's a rundown of some of the most common operators used in ``filter()``:
 
     query.filter(User.name.in_(['ed', 'wendy', 'jack']))
 
+    # works with query objects too:
+    
+    query.filter(User.name.in_(session.query(User.name).filter(User.name.like('%ed%'))))
+    
+* NOT IN::
+
+    query.filter(~User.name.in_(['ed', 'wendy', 'jack']))
+
 * IS NULL::
 
     filter(User.name == None)
 
+* IS NOT NULL::
+
+    filter(User.name != None)
+
 * AND::
 
     from sqlalchemy import and_
@@ -567,6 +581,15 @@ To use an entirely string-based statement, using ``from_statement()``; just ensu
     ['ed']
     {stop}[<User('ed','Ed Jones', 'f8s7ccs')>]
 
+You can use ``from_statement()`` to go completely "raw", using string names to identify desired columns:
+
+.. sourcecode:: python+sql
+
+    {sql}>>> session.query("id", "name", "thenumber12").from_statement("SELECT id, name, 12 as thenumber12 FROM users where name=:name").params(name='ed').all()
+    SELECT id, name, 12 as thenumber12 FROM users where name=?
+    ['ed']
+    {stop}[(1, u'ed', 12)]
+
 Counting
 --------
 
@@ -772,7 +795,7 @@ If you want to reduce the number of queries (dramatically, in many cases), we ca
     >>> jack.addresses
     [<Address('jack@google.com')>, <Address('j25@yahoo.com')>]
 
-SQLAlchemy has the ability to control exactly which attributes and how many levels deep should be joined together in a single SQL query.  More information on this feature is available in `advdatamapping_relation`.
+SQLAlchemy has the ability to control exactly which attributes and how many levels deep should be joined together in a single SQL query.  More information on this feature is available in :ref:`advdatamapping_relation`.
 
 Querying with Joins
 ====================

doc/build/reference/dialects/access.rst

@@ -1,4 +1,4 @@
-Access
-======
+Microsoft Access
+================
 
-.. automodule:: sqlalchemy.databases.access
+.. automodule:: sqlalchemy.dialects.access.base

doc/build/reference/dialects/firebird.rst

@@ -1,4 +1,11 @@
 Firebird
 ========
 
-.. automodule:: sqlalchemy.databases.firebird
+.. automodule:: sqlalchemy.dialects.firebird.base
+
+.. _kinterbasdb:
+
+kinterbasdb
+-----------
+
+.. automodule:: sqlalchemy.dialects.firebird.kinterbasdb

doc/build/reference/dialects/index.rst

@@ -1,19 +1,35 @@
-.. _sqlalchemy.databases:
+.. _sqlalchemy.dialects:
 
-sqlalchemy.databases
+sqlalchemy.dialects
 ====================
 
+Supported Databases
+-------------------
+
+These backends are fully operational with 
+current versions of SQLAlchemy.
+
+.. toctree::
+    :glob:
+
+    firebird
+    mssql
+    mysql
+    oracle
+    postgresql
+    sqlite
+
+Unsupported Databases
+---------------------
+
+These backends are untested and may not be completely
+ported to current versions of SQLAlchemy.
+
 .. toctree::
     :glob:
 
     access
-    firebird
     informix
     maxdb
-    mssql
-    mysql
-    oracle
-    postgres
-    sqlite
     sybase
 

doc/build/reference/dialects/informix.rst

@@ -1,4 +1,4 @@
 Informix
 ========
 
-.. automodule:: sqlalchemy.databases.informix
+.. automodule:: sqlalchemy.dialects.informix.base

doc/build/reference/dialects/maxdb.rst

@@ -1,4 +1,4 @@
 MaxDB
 =====
 
-.. automodule:: sqlalchemy.databases.maxdb
+.. automodule:: sqlalchemy.dialects.maxdb.base

doc/build/reference/dialects/mssql.rst

@@ -1,4 +1,21 @@
-SQL Server
-==========
+Microsoft SQL Server
+====================
 
-.. automodule:: sqlalchemy.databases.mssql
+.. automodule:: sqlalchemy.dialects.mssql.base
+
+PyODBC
+------
+.. automodule:: sqlalchemy.dialects.mssql.pyodbc
+
+AdoDBAPI
+--------
+.. automodule:: sqlalchemy.dialects.mssql.adodbapi
+
+pymssql
+-------
+.. automodule:: sqlalchemy.dialects.mssql.pymssql
+
+zxjdbc Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.mssql.zxjdbc

doc/build/reference/dialects/mysql.rst

@@ -1,140 +1,164 @@
 MySQL
 =====
 
-.. automodule:: sqlalchemy.databases.mysql
+.. automodule:: sqlalchemy.dialects.mysql.base
 
 MySQL Column Types
 ------------------
 
-.. autoclass:: MSNumeric
+.. autoclass:: NUMERIC
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSDecimal
+.. autoclass:: DECIMAL
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSDouble
+.. autoclass:: DOUBLE
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSReal
+.. autoclass:: REAL
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSFloat
+.. autoclass:: FLOAT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSInteger
+.. autoclass:: INTEGER
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSBigInteger
+.. autoclass:: BIGINT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSMediumInteger
+.. autoclass:: MEDIUMINT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSTinyInteger
+.. autoclass:: TINYINT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSSmallInteger
+.. autoclass:: SMALLINT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSBit
+.. autoclass:: BIT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSDateTime
+.. autoclass:: DATETIME
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSDate
+.. autoclass:: DATE
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSTime
+.. autoclass:: TIME
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSTimeStamp
+.. autoclass:: TIMESTAMP
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSYear
+.. autoclass:: YEAR
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSText
+.. autoclass:: TEXT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSTinyText
+.. autoclass:: TINYTEXT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSMediumText
+.. autoclass:: MEDIUMTEXT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSLongText
+.. autoclass:: LONGTEXT
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSString
+.. autoclass:: VARCHAR
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSChar
+.. autoclass:: CHAR
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSNVarChar
+.. autoclass:: NVARCHAR
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSNChar
+.. autoclass:: NCHAR
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSVarBinary
+.. autoclass:: VARBINARY
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSBinary
+.. autoclass:: BINARY
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSBlob
+.. autoclass:: BLOB
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSTinyBlob
+.. autoclass:: TINYBLOB
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSMediumBlob
+.. autoclass:: MEDIUMBLOB
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSLongBlob
+.. autoclass:: LONGBLOB
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSEnum
+.. autoclass:: ENUM
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSSet
+.. autoclass:: SET
    :members: __init__
    :show-inheritance:
 
-.. autoclass:: MSBoolean
+.. autoclass:: BOOLEAN
    :members: __init__
    :show-inheritance:
 
+MySQL-Python Notes
+--------------------
+
+.. automodule:: sqlalchemy.dialects.mysql.mysqldb
+
+OurSQL Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.mysql.oursql
+
+MySQL-Connector Notes
+----------------------
+
+.. automodule:: sqlalchemy.dialects.mysql.mysqlconnector
+
+pyodbc Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.mysql.pyodbc
+
+zxjdbc Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.mysql.zxjdbc

doc/build/reference/dialects/oracle.rst

@@ -1,4 +1,14 @@
 Oracle
 ======
 
-.. automodule:: sqlalchemy.databases.oracle
+.. automodule:: sqlalchemy.dialects.oracle.base
+
+cx_Oracle Notes
+---------------
+
+.. automodule:: sqlalchemy.dialects.oracle.cx_oracle
+
+zxjdbc Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.oracle.zxjdbc

doc/build/reference/dialects/postgres.rst

@@ -1,4 +0,0 @@
-PostgreSQL
-==========
-
-.. automodule:: sqlalchemy.databases.postgres

doc/build/reference/dialects/postgresql.rst

@@ -0,0 +1,68 @@
+PostgreSQL
+==========
+
+.. automodule:: sqlalchemy.dialects.postgresql.base
+
+PostgresSQL Column Types
+------------------------
+
+.. autoclass:: ARRAY
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: BIT
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: BYTEA
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: CIDR
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: DOUBLE_PRECISION
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: ENUM
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: INET
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: INTERVAL
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: MACADDR
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: REAL
+    :members: __init__
+    :show-inheritance:
+
+.. autoclass:: UUID
+    :members: __init__
+    :show-inheritance:
+
+
+psycopg2 Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.postgresql.psycopg2
+
+
+pg8000 Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.postgresql.pg8000
+
+zxjdbc Notes
+--------------
+
+.. automodule:: sqlalchemy.dialects.postgresql.zxjdbc

doc/build/reference/dialects/sqlite.rst

@@ -1,5 +1,9 @@
 SQLite
 ======
 
-.. automodule:: sqlalchemy.databases.sqlite
+.. automodule:: sqlalchemy.dialects.sqlite.base
 
+Pysqlite
+--------
+
+.. automodule:: sqlalchemy.dialects.sqlite.pysqlite

doc/build/reference/dialects/sybase.rst

@@ -1,4 +1,4 @@
 Sybase
 ======
 
-.. automodule:: sqlalchemy.databases.sybase
+.. automodule:: sqlalchemy.dialects.sybase.base

doc/build/reference/ext/compiler.rst

@@ -1,3 +1,5 @@
+.. _sqlalchemy.ext.compiler_toplevel:
+
 compiler
 ========
 

doc/build/reference/ext/sqlsoup.rst

@@ -1,21 +1,6 @@
 SqlSoup
 =======
 
-:author: Jonathan Ellis
-
-SqlSoup creates mapped classes on the fly from tables, which are automatically reflected from the database based on name.  It is essentially a nicer version of the "row data gateway" pattern.
-
-.. sourcecode:: python+sql
-
-    >>> from sqlalchemy.ext.sqlsoup import SqlSoup
-    >>> soup = SqlSoup('sqlite:///')
-
-    >>> db.users.select(order_by=[db.users.c.name])
-    [MappedUsers(name='Bhargan Basepair',email='basepair@example.edu',password='basepair',classname=None,admin=1),
-     MappedUsers(name='Joe Student',email='student@example.edu',password='student',classname=None,admin=0)]
-
-Full SqlSoup documentation is on the `SQLAlchemy Wiki <http://www.sqlalchemy.org/trac/wiki/SqlSoup>`_.
-
 .. automodule:: sqlalchemy.ext.sqlsoup
-   :members:
-   :undoc-members:
+    :members:
+    

doc/build/reference/orm/collections.rst

@@ -9,6 +9,9 @@ This is an in-depth discussion of collection mechanics.  For simple examples, se
 
 .. autoclass:: collection
 
+.. autoclass:: sqlalchemy.orm.collections.MappedCollection
+   :members:
+
 .. autofunction:: collection_adapter
 
 .. autofunction:: column_mapped_collection

doc/build/reference/sqlalchemy/connections.rst

@@ -58,14 +58,6 @@ Internals
     :members:
     :show-inheritance:
 
-.. autoclass:: DefaultRunner
-    :members:
-    :show-inheritance:
-    
 .. autoclass:: ExecutionContext
     :members:
 
-.. autoclass:: SchemaIterator
-    :members:
-    :show-inheritance:
-    

doc/build/reference/sqlalchemy/expressions.rst

@@ -99,6 +99,8 @@ The expression package uses functions to construct SQL expressions.  The return
 
 .. autofunction:: text
 
+.. autofunction:: tuple_
+
 .. autofunction:: union
 
 .. autofunction:: union_all
@@ -112,6 +114,10 @@ Classes
    :members:
    :show-inheritance:
 
+.. autoclass:: _BindParamClause
+   :members:
+   :show-inheritance:
+
 .. autoclass:: ClauseElement
    :members:
    :show-inheritance:
@@ -146,6 +152,14 @@ Classes
    :members: where
    :show-inheritance:
 
+.. autoclass:: FunctionElement
+   :members:
+   :show-inheritance:
+
+.. autoclass:: Function
+   :members:
+   :show-inheritance:
+   
 .. autoclass:: FromClause
    :members:
    :show-inheritance:
@@ -166,6 +180,10 @@ Classes
    :members:
    :show-inheritance:
 
+.. autoclass:: _SelectBaseMixin
+   :members:
+   :show-inheritance:
+
 .. autoclass:: TableClause
    :members:
    :show-inheritance:

doc/build/reference/sqlalchemy/pooling.rst

@@ -32,7 +32,7 @@ directly to :func:`~sqlalchemy.create_engine` as keyword arguments:
 ``pool_size``, ``max_overflow``, ``pool_recycle`` and
 ``pool_timeout``.  For example::
 
-  engine = create_engine('postgres://me@localhost/mydb',
+  engine = create_engine('postgresql://me@localhost/mydb',
                          pool_size=20, max_overflow=0)
 
 In the case of SQLite, a :class:`SingletonThreadPool` is provided instead,

doc/build/reference/sqlalchemy/schema.rst

@@ -1,4 +1,4 @@
-.. _schema:
+.. _schema_api_toplevel:
 
 Database Schema
 ===============
@@ -12,7 +12,6 @@ Tables and Columns
 
 .. autoclass:: Column
     :members:
-    :inherited-members:
     :undoc-members:
     :show-inheritance:
 
@@ -23,7 +22,6 @@ Tables and Columns
 
 .. autoclass:: Table
     :members:
-    :inherited-members:
     :undoc-members:
     :show-inheritance:
 
@@ -102,14 +100,61 @@ Default Generators and Markers
     :undoc-members:
     :show-inheritance:
 
-DDL
----
+.. _schema_api_ddl:
+
+DDL Generation
+--------------
+
+.. autoclass:: DDLElement
+    :members:
+    :undoc-members:
+    :show-inheritance:
 
 .. autoclass:: DDL
     :members:
     :undoc-members:
     :show-inheritance:
 
+.. autoclass:: CreateTable
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: DropTable
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: CreateSequence
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: DropSequence
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: CreateIndex
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: DropIndex
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: AddConstraint
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: DropConstraint
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 Internals
 ---------
 

doc/build/reference/sqlalchemy/types.rst

@@ -38,52 +38,62 @@ database column type available on the target database when issuing a
 type is emitted in ``CREATE TABLE``, such as ``VARCHAR`` see `SQL
 Standard Types`_ and the other sections of this chapter.
 
-.. autoclass:: String
+.. autoclass:: Boolean
+  :show-inheritance:
+
+.. autoclass:: Date
+ :show-inheritance:
+
+.. autoclass:: DateTime
    :show-inheritance:
 
-.. autoclass:: Unicode
+.. autoclass:: Enum
+  :show-inheritance:
+  :members:
+
+.. autoclass:: Float
+  :show-inheritance:
+  :members:
+
+.. autoclass:: Integer
+  :show-inheritance:
+
+.. autoclass:: Interval
+ :show-inheritance:
+
+.. autoclass:: LargeBinary
+ :show-inheritance:
+
+.. autoclass:: Numeric
+  :show-inheritance:
+  :members:
+
+.. autoclass:: PickleType
+ :show-inheritance:
+
+.. autoclass:: SchemaType
+  :show-inheritance:
+  :members:
+  :undoc-members:
+
+.. autoclass:: SmallInteger
+ :show-inheritance:
+
+.. autoclass:: String
    :show-inheritance:
 
 .. autoclass:: Text
    :show-inheritance:
 
+.. autoclass:: Time
+  :show-inheritance:
+
+.. autoclass:: Unicode
+  :show-inheritance:
+
 .. autoclass:: UnicodeText
    :show-inheritance:
 
-.. autoclass:: Integer
-   :show-inheritance:
-
-.. autoclass:: SmallInteger
-   :show-inheritance:
-
-.. autoclass:: Numeric
-   :show-inheritance:
-
-.. autoclass:: Float
-   :show-inheritance:
-
-.. autoclass:: DateTime
-   :show-inheritance:
-
-.. autoclass:: Date
-   :show-inheritance:
-
-.. autoclass:: Time
-   :show-inheritance:
-
-.. autoclass:: Interval
-   :show-inheritance:
-
-.. autoclass:: Boolean
-   :show-inheritance:
-
-.. autoclass:: Binary
-   :show-inheritance:
-
-.. autoclass:: PickleType
-   :show-inheritance:
-
-
 SQL Standard Types
 ------------------
 
@@ -91,84 +101,117 @@ The SQL standard types always create database column types of the same
 name when ``CREATE TABLE`` is issued.  Some types may not be supported
 on all databases.
 
-.. autoclass:: INT
-   :show-inheritance:
-
-.. autoclass:: sqlalchemy.types.INTEGER
-   :show-inheritance:
-
-.. autoclass:: CHAR
-   :show-inheritance:
-
-.. autoclass:: VARCHAR
-   :show-inheritance:
-
-.. autoclass:: NCHAR
-   :show-inheritance:
-
-.. autoclass:: TEXT
-   :show-inheritance:
-
-.. autoclass:: FLOAT
-   :show-inheritance:
-
-.. autoclass:: NUMERIC
-   :show-inheritance:
-
-.. autoclass:: DECIMAL
-   :show-inheritance:
-
-.. autoclass:: TIMESTAMP
-   :show-inheritance:
-
-.. autoclass:: DATETIME
-   :show-inheritance:
-
-.. autoclass:: CLOB
-   :show-inheritance:
+.. autoclass:: BINARY
+  :show-inheritance:
 
 .. autoclass:: BLOB
-   :show-inheritance:
+  :show-inheritance:
 
 .. autoclass:: BOOLEAN
-   :show-inheritance:
+  :show-inheritance:
 
-.. autoclass:: SMALLINT
-   :show-inheritance:
+.. autoclass:: CHAR
+  :show-inheritance:
+
+.. autoclass:: CLOB
+  :show-inheritance:
 
 .. autoclass:: DATE
-   :show-inheritance:
+  :show-inheritance:
+
+.. autoclass:: DATETIME
+  :show-inheritance:
+
+.. autoclass:: DECIMAL
+  :show-inheritance:
+
+.. autoclass:: FLOAT
+  :show-inheritance:
+
+.. autoclass:: INT
+  :show-inheritance:
+
+.. autoclass:: sqlalchemy.types.INTEGER
+  :show-inheritance:
+
+.. autoclass:: NCHAR
+  :show-inheritance:
+
+.. autoclass:: NUMERIC
+  :show-inheritance:
+
+.. autoclass:: SMALLINT
+  :show-inheritance:
+
+.. autoclass:: TEXT
+  :show-inheritance:
 
 .. autoclass:: TIME
-   :show-inheritance:
+  :show-inheritance:
+
+.. autoclass:: TIMESTAMP
+  :show-inheritance:
+
+.. autoclass:: VARBINARY
+  :show-inheritance:
+
+.. autoclass:: VARCHAR
+  :show-inheritance:
 
 
 Vendor-Specific Types
 ---------------------
 
 Database-specific types are also available for import from each
-database's dialect module. See the :ref:`sqlalchemy.databases`
+database's dialect module. See the :ref:`sqlalchemy.dialects`
 reference for the database you're interested in.
 
 For example, MySQL has a ``BIGINTEGER`` type and PostgreSQL has an
 ``INET`` type.  To use these, import them from the module explicitly::
 
-    from sqlalchemy.databases.mysql import MSBigInteger, MSEnum
+    from sqlalchemy.dialects import mysql
 
     table = Table('foo', meta,
-        Column('id', MSBigInteger),
-        Column('enumerates', MSEnum('a', 'b', 'c'))
+        Column('id', mysql.BIGINTEGER),
+        Column('enumerates', mysql.ENUM('a', 'b', 'c'))
     )
 
 Or some PostgreSQL types::
 
-    from sqlalchemy.databases.postgres import PGInet, PGArray
+    from sqlalchemy.dialects import postgresql
 
     table = Table('foo', meta,
-        Column('ipaddress', PGInet),
-        Column('elements', PGArray(str))
+        Column('ipaddress', postgresql.INET),
+        Column('elements', postgresql.ARRAY(str))
         )
 
+Each dialect provides the full set of typenames supported by
+that backend within its `__all__` collection, so that a simple
+`import *` or similar will import all supported types as 
+implemented for that backend::
+
+    from sqlalchemy.dialects.postgresql import *
+
+    t = Table('mytable', metadata,
+               Column('id', INTEGER, primary_key=True),
+               Column('name', VARCHAR(300)),
+               Column('inetaddr', INET)
+    )
+
+Where above, the INTEGER and VARCHAR types are ultimately from 
+sqlalchemy.types, and INET is specific to the Postgresql dialect.
+
+Some dialect level types have the same name as the SQL standard type,
+but also provide additional arguments.  For example, MySQL implements
+the full range of character and string types including additional arguments
+such as `collation` and `charset`::
+
+    from sqlalchemy.dialects.mysql import VARCHAR, TEXT
+
+    table = Table('foo', meta,
+        Column('col1', VARCHAR(200, collation='binary')),
+        Column('col2', TEXT(charset='latin1'))
+    )
 
 Custom Types
 ------------
@@ -181,7 +224,7 @@ The simplest method is implementing a :class:`TypeDecorator`, a helper
 class that makes it easy to augment the bind parameter and result
 processing capabilities of one of the built in types.
 
-To build a type object from scratch, subclass `:class:TypeEngine`.
+To build a type object from scratch, subclass `:class:UserDefinedType`.
 
 .. autoclass:: TypeDecorator
    :members:
@@ -189,6 +232,12 @@ To build a type object from scratch, subclass `:class:TypeEngine`.
    :inherited-members:
    :show-inheritance:
 
+.. autoclass:: UserDefinedType
+   :members:
+   :undoc-members:
+   :inherited-members:
+   :show-inheritance:
+
 .. autoclass:: TypeEngine
    :members:
    :undoc-members:

doc/build/session.rst

@@ -54,7 +54,7 @@ In our previous example regarding ``sessionmaker()``, we specified a ``bind`` fo
     Session = sessionmaker()
 
     # later, we create the engine
-    engine = create_engine('postgres://...')
+    engine = create_engine('postgresql://...')
     
     # associate it with our custom Session class
     Session.configure(bind=engine)
@@ -74,7 +74,7 @@ The ``Session`` can also be explicitly bound to an individual database ``Connect
     # global application scope.  create Session class, engine
     Session = sessionmaker()
 
-    engine = create_engine('postgres://...')
+    engine = create_engine('postgresql://...')
     
     ...
     
@@ -135,7 +135,7 @@ Frequently Asked Questions
 
     You typically invoke ``Session()`` when you first need to talk to your database, and want to save some objects or load some existing ones.  Then, you work with it, save your changes, and then dispose of it....or at the very least ``close()`` it.  It's not a "global" kind of object, and should be handled more like a "local variable", as it's generally **not** safe to use with concurrent threads.  Sessions are very inexpensive to make, and don't use any resources whatsoever until they are first used...so create some !
 
-    There is also a pattern whereby you're using a **contextual session**, this is described later in `unitofwork_contextual`.  In this pattern, a helper object is maintaining a ``Session`` for you, most commonly one that is local to the current thread (and sometimes also local to an application instance).  SQLAlchemy has worked this pattern out such that it still *looks* like you're creating a new session as you need one...so in that case, it's still a guaranteed win to just say ``Session()`` whenever you want a session.  
+    There is also a pattern whereby you're using a **contextual session**, this is described later in :ref:`unitofwork_contextual`.  In this pattern, a helper object is maintaining a ``Session`` for you, most commonly one that is local to the current thread (and sometimes also local to an application instance).  SQLAlchemy has worked this pattern out such that it still *looks* like you're creating a new session as you need one...so in that case, it's still a guaranteed win to just say ``Session()`` whenever you want a session.  
 
 * Is the Session a cache ? 
 
@@ -179,7 +179,7 @@ The ``query()`` function takes one or more *entities* and returns a new ``Query`
     user_mapper = class_mapper(User)
     session.query(user_mapper)
 
-When ``Query`` returns results, each object instantiated is stored within the identity map.   When a row matches an object which is already present, the same object is returned.  In the latter case, whether or not the row is populated onto an existing object depends upon whether the attributes of the instance have been *expired* or not.  As of 0.5, a default-configured ``Session`` automatically expires all instances along transaction boundaries, so that with a normally isolated transaction, there shouldn't be any issue of instances representing data which is stale with regards to the current transaction.
+When ``Query`` returns results, each object instantiated is stored within the identity map.   When a row matches an object which is already present, the same object is returned.  In the latter case, whether or not the row is populated onto an existing object depends upon whether the attributes of the instance have been *expired* or not.  A default-configured ``Session`` automatically expires all instances along transaction boundaries, so that with a normally isolated transaction, there shouldn't be any issue of instances representing data which is stale with regards to the current transaction.
 
 Adding New or Existing Items
 ----------------------------
@@ -197,7 +197,7 @@ To add a list of items to the session at once, use ``add_all()``::
 
     session.add_all([item1, item2, item3])
 
-The ``add()`` operation **cascades** along the ``save-update`` cascade.  For more details see the section `unitofwork_cascades`.
+The ``add()`` operation **cascades** along the ``save-update`` cascade.  For more details see the section :ref:`unitofwork_cascades`.
 
 Merging
 -------
@@ -219,7 +219,7 @@ With ``merge()``, the given instance is not placed within the session, and can b
   * An application which reads an object structure from a file and wishes to save it to the database might parse the file, build up the structure, and then use ``merge()`` to save it to the database, ensuring that the data within the file is used to formulate the primary key of each element of the structure.  Later, when the file has changed, the same process can be re-run, producing a slightly different object structure, which can then be ``merged()`` in again, and the ``Session`` will automatically update the database to reflect those changes.
   * A web application stores mapped entities within an HTTP session object.  When each request starts up, the serialized data can be merged into the session, so that the original entity may be safely shared among requests and threads.
 
-``merge()`` is frequently used by applications which implement their own second level caches.  This refers to an application which uses an in memory dictionary, or an tool like Memcached to store objects over long running spans of time.  When such an object needs to exist within a ``Session``, ``merge()`` is a good choice since it leaves the original cached object untouched.  For this use case, merge provides a keyword option called ``dont_load=True``.  When this boolean flag is set to ``True``, ``merge()`` will not issue any SQL to reconcile the given object against the current state of the database, thereby reducing query overhead.   The limitation is that the given object and all of its children may not contain any pending changes, and it's also of course possible that newer information in the database will not be present on the merged object, since no load is issued.
+``merge()`` is frequently used by applications which implement their own second level caches.  This refers to an application which uses an in memory dictionary, or an tool like Memcached to store objects over long running spans of time.  When such an object needs to exist within a ``Session``, ``merge()`` is a good choice since it leaves the original cached object untouched.  For this use case, merge provides a keyword option called ``load=False``.  When this boolean flag is set to ``False``, ``merge()`` will not issue any SQL to reconcile the given object against the current state of the database, thereby reducing query overhead.   The limitation is that the given object and all of its children may not contain any pending changes, and it's also of course possible that newer information in the database will not be present on the merged object, since no load is issued.
 
 Deleting
 --------
@@ -249,6 +249,17 @@ The solution is to use proper cascading::
     del user.addresses[1]
     session.flush()
 
+Deleting based on Filter Criterion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The caveat with ``Session.delete()`` is that you need to have an object handy already in order to delete.   The Query includes a ``delete()`` method which deletes based on filtering criteria::
+
+    session.query(User).filter(User.id==7).delete()
+    
+The ``Query.delete()`` method includes functionality to "expire" objects already in the session which 
+match the criteria.   However it does have some caveats, including that "delete" and "delete-orphan" 
+cascades won't be fully expressed for collections which are already loaded.  See the API docs for :meth:`~sqlalchemy.orm.query.Query.delete` for more details.
+
 Flushing
 --------
 
@@ -290,7 +301,7 @@ Rolling Back
   * Objects which were marked as *deleted* within the lifespan of the transaction are promoted back to the *persistent* state, corresponding to their DELETE statement being rolled back.  Note that if those objects were first *pending* within the transaction, that operation takes precedence instead.
   * All objects not expunged are fully expired.  
 
-With that state understood, the ``Session`` may safely continue usage after a rollback occurs (note that this is a new feature as of version 0.5).
+With that state understood, the ``Session`` may safely continue usage after a rollback occurs.
 
 When a ``flush()`` fails, typically for reasons like primary key, foreign key, or "not nullable" constraint violations, a ``rollback()`` is issued automatically (it's currently not possible for a flush to continue after a partial failure).  However, the flush process always uses its own transactional demarcator called a *subtransaction*, which is described more fully in the docstrings for ``Session``.  What it means here is that even though the database transaction has been rolled back, the end user must still issue ``rollback()`` to fully reset the state of the ``Session``.
 
@@ -368,6 +379,8 @@ The session is also keeping track of all newly created (i.e. pending) objects, a
 
 Note that objects within the session are by default *weakly referenced*.  This means that when they are dereferenced in the outside application, they fall out of scope from within the ``Session`` as well and are subject to garbage collection by the Python interpreter.  The exceptions to this include objects which are pending, objects which are marked as deleted, or persistent objects which have pending changes on them.  After a full flush, these collections are all empty, and all objects are again weakly referenced.  To disable the weak referencing behavior and force all objects within the session to remain until explicitly expunged, configure ``sessionmaker()`` with the ``weak_identity_map=False`` setting.
 
+.. _unitofwork_cascades:
+
 Cascades
 ========
 
@@ -382,12 +395,14 @@ Cascading is configured by setting the ``cascade`` keyword argument on a ``relat
 
 The above mapper specifies two relations, ``items`` and ``customer``.  The ``items`` relationship specifies "all, delete-orphan" as its ``cascade`` value, indicating that all  ``add``, ``merge``, ``expunge``, ``refresh`` ``delete`` and ``expire`` operations performed on a parent ``Order`` instance should also be performed on the child ``Item`` instances attached to it.  The ``delete-orphan`` cascade value additionally indicates that if an ``Item`` instance is no longer associated with an ``Order``, it should also be deleted.  The "all, delete-orphan" cascade argument allows a so-called *lifecycle* relationship between an ``Order`` and an ``Item`` object.
 
-The ``customer`` relationship specifies only the "save-update" cascade value, indicating most operations will not be cascaded from a parent ``Order`` instance to a child ``User`` instance except for the ``add()`` operation.  "save-update" cascade indicates that an ``add()`` on the parent will cascade to all child items, and also that items added to a parent which is already present in the session will also be added.
+The ``customer`` relationship specifies only the "save-update" cascade value, indicating most operations will not be cascaded from a parent ``Order`` instance to a child ``User`` instance except for the ``add()`` operation.  "save-update" cascade indicates that an ``add()`` on the parent will cascade to all child items, and also that items added to a parent which is already present in the session will also be added.  "save-update" cascade also cascades the *pending history* of a relation()-based attribute, meaning that objects which were removed from a scalar or collection attribute whose changes have not yet been flushed are also placed into the new session - this so that foreign key clear operations and deletions will take place (new in 0.6).
 
 Note that the ``delete-orphan`` cascade only functions for relationships where the target object can have a single parent at a time, meaning it is only appropriate for one-to-one or one-to-many relationships.  For a :func:`~sqlalchemy.orm.relation` which establishes one-to-one via a local foreign key, i.e. a many-to-one that stores only a single parent, or one-to-one/one-to-many via a "secondary" (association) table, a warning will be issued if ``delete-orphan`` is configured.  To disable this warning, also specify the ``single_parent=True`` flag on the relationship, which constrains objects to allow attachment to only one parent at a time.
 
 The default value for ``cascade`` on :func:`~sqlalchemy.orm.relation()` is ``save-update, merge``.
 
+.. _unitofwork_transaction:
+
 Managing Transactions
 =====================
 
@@ -459,8 +474,8 @@ Enabling Two-Phase Commit
 
 Finally, for MySQL, PostgreSQL, and soon Oracle as well, the session can be instructed to use two-phase commit semantics. This will coordinate the committing of transactions across databases so that the transaction is either committed or rolled back in all databases. You can also ``prepare()`` the session for interacting with transactions not managed by SQLAlchemy. To use two phase transactions set the flag ``twophase=True`` on the session::
 
-    engine1 = create_engine('postgres://db1')
-    engine2 = create_engine('postgres://db2')
+    engine1 = create_engine('postgresql://db1')
+    engine2 = create_engine('postgresql://db2')
     
     Session = sessionmaker(twophase=True)
 
@@ -549,7 +564,7 @@ Note that above, we issue a ``commit()`` both on the ``Session`` as well as the
 
 When using the ``threadlocal`` engine context, the process above is simplified; the ``Session`` uses the same connection/transaction as everyone else in the current thread, whether or not you explicitly bind it::
 
-    engine = create_engine('postgres://mydb', strategy="threadlocal")
+    engine = create_engine('postgresql://mydb', strategy="threadlocal")
     engine.begin()
     
     session = Session()  # session takes place in the transaction like everyone else
@@ -558,6 +573,8 @@ When using the ``threadlocal`` engine context, the process above is simplified;
     
     engine.commit() # commit the transaction
 
+.. _unitofwork_contextual:
+
 Contextual/Thread-local Sessions 
 =================================
 
@@ -606,6 +623,8 @@ The contextual session may be disposed of by calling ``Session.remove()``::
 
 After ``remove()`` is called, the next operation with the contextual session will start a new ``Session`` for the current thread.
 
+.. _session_lifespan:
+
 Lifespan of a Contextual Session 
 --------------------------------
 
@@ -632,7 +651,7 @@ A (really, really) common question is when does the contextual session get creat
                         Session.remove() <-
     web response   <-  
 
-The above example illustrates an explicit call to ``Session.remove()``.  This has the effect such that each web request starts fresh with a brand new session.   When integrating with a web framework, there's actually many options on how to proceed for this step, particularly as of version 0.5:
+The above example illustrates an explicit call to ``Session.remove()``.  This has the effect such that each web request starts fresh with a brand new session.   When integrating with a web framework, there's actually many options on how to proceed for this step:
 
 * Session.remove() - this is the most cut and dry approach; the ``Session`` is thrown away, all of its transactional/connection resources are closed out, everything within it is explicitly gone.  A new ``Session`` will be used on the next request.
 * Session.close() - Similar to calling ``remove()``, in that all objects are explicitly expunged and all transactional/connection resources closed, except the actual ``Session`` object hangs around.  It doesn't make too much difference here unless the start of the web request would like to pass specific options to the initial construction of ``Session()``, such as a specific ``Engine`` to bind to.
@@ -652,8 +671,8 @@ Vertical Partitioning
 
 Vertical partitioning places different kinds of objects, or different tables, across multiple databases::
 
-    engine1 = create_engine('postgres://db1')
-    engine2 = create_engine('postgres://db2')
+    engine1 = create_engine('postgresql://db1')
+    engine2 = create_engine('postgresql://db2')
 
     Session = sessionmaker(twophase=True)
 

doc/build/sqlexpression.rst

@@ -10,13 +10,13 @@ Version Check
 =============
 
 
-A quick check to verify that we are on at least **version 0.5** of SQLAlchemy:
+A quick check to verify that we are on at least **version 0.6** of SQLAlchemy:
 
 .. sourcecode:: pycon+sql
 
     >>> import sqlalchemy
     >>> sqlalchemy.__version__ # doctest:+SKIP
-    0.5.0
+    0.6.0
     
 Connecting
 ==========
@@ -63,25 +63,25 @@ Next, to tell the ``MetaData`` we'd actually like to create our selection of tab
 
     {sql}>>> metadata.create_all(engine) #doctest: +NORMALIZE_WHITESPACE
     PRAGMA table_info("users")
-    {}
+    ()
     PRAGMA table_info("addresses")
-    {}
+    ()
     CREATE TABLE users (
         id INTEGER NOT NULL, 
         name VARCHAR, 
         fullname VARCHAR, 
         PRIMARY KEY (id)
     )
-    {}
+    ()
     COMMIT
     CREATE TABLE addresses (
         id INTEGER NOT NULL, 
         user_id INTEGER, 
         email_address VARCHAR NOT NULL, 
         PRIMARY KEY (id), 
-         FOREIGN KEY(user_id) REFERENCES users (id)
+        FOREIGN KEY(user_id) REFERENCES users (id)
     )
-    {}
+    ()
     COMMIT
 
 Users familiar with the syntax of CREATE TABLE may notice that the VARCHAR columns were generated without a length; on SQLite, this is a valid datatype, but on most databases it's not allowed.  So if running this tutorial on a database such as PostgreSQL or MySQL, and you wish to use SQLAlchemy to generate the tables, a "length" may be provided to the ``String`` type as below::
@@ -143,10 +143,10 @@ What about the ``result`` variable we got when we called ``execute()`` ?  As the
 
 .. sourcecode:: pycon+sql
 
-    >>> result.last_inserted_ids()
+    >>> result.inserted_primary_key
     [1]
     
-The value of ``1`` was automatically generated by SQLite, but only because we did not specify the ``id`` column in our ``Insert`` statement; otherwise, our explicit value would have been used.   In either case, SQLAlchemy always knows how to get at a newly generated primary key value, even though the method of generating them is different across different databases; each databases' ``Dialect`` knows the specific steps needed to determine the correct value (or values; note that ``last_inserted_ids()`` returns a list so that it supports composite primary keys).
+The value of ``1`` was automatically generated by SQLite, but only because we did not specify the ``id`` column in our ``Insert`` statement; otherwise, our explicit value would have been used.   In either case, SQLAlchemy always knows how to get at a newly generated primary key value, even though the method of generating them is different across different databases; each databases' ``Dialect`` knows the specific steps needed to determine the correct value (or values; note that ``inserted_primary_key`` returns a list so that it supports composite primary keys).
 
 Executing Multiple Statements 
 ==============================
@@ -213,7 +213,7 @@ When the ``MetaData`` is bound, statements will also compile against the engine'
 
     >>> metadata.bind = None
 
-Detailed examples of connectionless and implicit execution are available in the "Engines" chapter: `dbengine_implicit`.
+Detailed examples of connectionless and implicit execution are available in the "Engines" chapter: :ref:`dbengine_implicit`.
     
 Selecting 
 ==========
@@ -426,6 +426,12 @@ If you have come across an operator which really isn't available, you can always
     >>> print users.c.name.op('tiddlywinks')('foo')
     users.name tiddlywinks :name_1
     
+This function can also be used to make bitwise operators explicit. For example::
+
+    somecolumn.op('&')(0xff)
+
+is a bitwise AND of the value in `somecolumn`.
+    
 Conjunctions 
 =============
 
@@ -477,10 +483,11 @@ So with all of this vocabulary, let's select all users who have an email address
 
 Once again, SQLAlchemy figured out the FROM clause for our statement.  In fact it will determine the FROM clause based on all of its other bits; the columns clause, the where clause, and also some other elements which we haven't covered yet, which include ORDER BY, GROUP BY, and HAVING. 
 
+.. _sqlexpression_text:
+
 Using Text 
 ===========
 
-
 Our last example really became a handful to type.  Going from what one understands to be a textual SQL expression into a Python construct which groups components together in a programmatic style can be hard.  That's why SQLAlchemy lets you just use strings too.  The ``text()`` construct represents any textual statement.  To use bind parameters with ``text()``, always use the named colon format.  Such as below, we create a ``text()`` and execute it, feeding in the bind parameters to the ``execute()`` method:
 
 .. sourcecode:: pycon+sql
@@ -499,11 +506,11 @@ Our last example really became a handful to type.  Going from what one understan
     ['m', 'z', '%@aol.com', '%@msn.com']
     {stop}[(u'Wendy Williams, wendy@aol.com',)]
 
-To gain a "hybrid" approach, any of SA's SQL constructs can have text freely intermingled wherever you like - the ``text()`` construct can be placed within any other ``ClauseElement`` construct, and when used in a non-operator context, a direct string may be placed which converts to ``text()`` automatically.  Below we combine the usage of ``text()`` and strings with our constructed ``select()`` object, by using the ``select()`` object to structure the statement, and the ``text()``/strings to provide all the content within the structure.  For this example, SQLAlchemy is not given any ``Column`` or ``Table`` objects in any of its expressions, so it cannot generate a FROM clause.  So we also give it the ``from_obj`` keyword argument, which is a list of ``ClauseElements`` (or strings) to be placed within the FROM clause:
+To gain a "hybrid" approach, the `select()` construct accepts strings for most of its arguments.  Below we combine the usage of strings with our constructed ``select()`` object, by using the ``select()`` object to structure the statement, and strings to provide all the content within the structure.  For this example, SQLAlchemy is not given any ``Column`` or ``Table`` objects in any of its expressions, so it cannot generate a FROM clause.  So we also give it the ``from_obj`` keyword argument, which is a list of ``ClauseElements`` (or strings) to be placed within the FROM clause:
 
 .. sourcecode:: pycon+sql
 
-    >>> s = select([text("users.fullname || ', ' || addresses.email_address AS title")], 
+    >>> s = select(["users.fullname || ', ' || addresses.email_address AS title"], 
     ...        and_( 
     ...            "users.id = addresses.user_id", 
     ...             "users.name BETWEEN 'm' AND 'z'",
@@ -523,7 +530,6 @@ Going from constructed SQL to text, we lose some capabilities.  We lose the capa
 Using Aliases 
 ==============
 
-
 The alias corresponds to a "renamed" version of a table or arbitrary relation, which occurs anytime you say "SELECT  .. FROM sometable AS someothername".  The ``AS`` creates a new name for the table.  Aliases are super important in SQL as they allow you to reference the same table more than once.  Scenarios where you need to do this include when you self-join a table to itself, or more commonly when you need to join from a parent table to a child table multiple times.  For example, we know that our user ``jack`` has two email addresses.  How can we locate jack based on the combination of those two addresses?  We need to join twice to it.  Let's construct two distinct aliases for the ``addresses`` table and join:
 
 .. sourcecode:: pycon+sql
@@ -625,7 +631,7 @@ That's the output ``outerjoin()`` produces, unless, of course, you're stuck in a
 
 .. sourcecode:: pycon+sql
 
-    >>> from sqlalchemy.databases.oracle import OracleDialect
+    >>> from sqlalchemy.dialects.oracle import dialect as OracleDialect
     >>> print s.compile(dialect=OracleDialect(use_ansi=False))
     SELECT users.fullname 
     FROM users, addresses 
@@ -786,7 +792,7 @@ SQL functions are created using the ``func`` keyword, which generates functions
     
 By "generates", we mean that **any** SQL function is created based on the word you choose::
 
-    >>> print func.xyz_my_goofy_function()
+    >>> print func.xyz_my_goofy_function() # doctest: +NORMALIZE_WHITESPACE
     xyz_my_goofy_function()    
     
 Certain function names are known by SQLAlchemy, allowing special behavioral rules to be applied.   Some for example are "ANSI" functions, which mean they don't get the parenthesis added after them, such as CURRENT_TIMESTAMP:
@@ -845,7 +851,6 @@ See also :attr:`sqlalchemy.sql.expression.func`.
 Unions and Other Set Operations 
 -------------------------------
 
-
 Unions come in two flavors, UNION and UNION ALL, which are available via module level functions:
 
 .. sourcecode:: pycon+sql
@@ -884,6 +889,33 @@ Also available, though not supported on all databases, are ``intersect()``, ``in
     ['%@%.com', '%@msn.com']
     {stop}[(1, 1, u'jack@yahoo.com'), (4, 2, u'wendy@aol.com')]
 
+A common issue with so-called "compound" selectables arises due to the fact that they nest with parenthesis.  SQLite in particular doesn't like a statement that starts with parenthesis.  So when nesting a "compound" inside a "compound", it's often necessary to apply
+``.alias().select()`` to the first element of the outermost compound, if that element is also a compound.  For example, to nest a "union" and a "select" inside of "except\_", SQLite will want
+the "union" to be stated as a subquery:
+
+.. sourcecode:: pycon+sql
+
+    >>> u = except_(
+    ...    union(
+    ...         addresses.select(addresses.c.email_address.like('%@yahoo.com')),
+    ...         addresses.select(addresses.c.email_address.like('%@msn.com'))
+    ...     ).alias().select(),   # apply subquery here
+    ...    addresses.select(addresses.c.email_address.like('%@msn.com'))
+    ... )
+    {sql}>>> print conn.execute(u).fetchall()   # doctest: +NORMALIZE_WHITESPACE
+    SELECT anon_1.id, anon_1.user_id, anon_1.email_address 
+    FROM (SELECT addresses.id AS id, addresses.user_id AS user_id, 
+    addresses.email_address AS email_address FROM addresses 
+    WHERE addresses.email_address LIKE ? UNION SELECT addresses.id AS id, 
+    addresses.user_id AS user_id, addresses.email_address AS email_address 
+    FROM addresses WHERE addresses.email_address LIKE ?) AS anon_1 EXCEPT 
+    SELECT addresses.id, addresses.user_id, addresses.email_address 
+    FROM addresses 
+    WHERE addresses.email_address LIKE ?
+    ['%@yahoo.com', '%@msn.com', '%@msn.com']
+    {stop}[(1, 1, u'jack@yahoo.com')]
+
+
 Scalar Selects 
 --------------
 
@@ -996,8 +1028,8 @@ Finally, we're back to UPDATE.  Updates work a lot like INSERTS, except there is
     COMMIT
     {stop}<sqlalchemy.engine.base.ResultProxy object at 0x...>
 
-    >>> # update a column to an expression.  Send a dictionary to values():
-    {sql}>>> conn.execute(users.update().values({users.c.fullname:"Fullname: " + users.c.name})) #doctest: +ELLIPSIS
+    >>> # update a column to an expression.:
+    {sql}>>> conn.execute(users.update().values(fullname="Fullname: " + users.c.name)) #doctest: +ELLIPSIS
     UPDATE users SET fullname=(? || users.name)
     ['Fullname: ']
     COMMIT
@@ -1011,7 +1043,7 @@ A correlated update lets you update a table using selection from another table,
 .. sourcecode:: pycon+sql
 
     >>> s = select([addresses.c.email_address], addresses.c.user_id==users.c.id).limit(1)
-    {sql}>>> conn.execute(users.update().values({users.c.fullname:s})) #doctest: +ELLIPSIS,+NORMALIZE_WHITESPACE
+    {sql}>>> conn.execute(users.update().values(fullname=s)) #doctest: +ELLIPSIS,+NORMALIZE_WHITESPACE
     UPDATE users SET fullname=(SELECT addresses.email_address 
     FROM addresses 
     WHERE addresses.user_id = users.id 

doc/build/static/docs.css

@@ -19,6 +19,7 @@ p {
 	margin-bottom:10px;
 }
 
+
 a {font-weight:normal; text-decoration:underline;}
 a:link {color:#0000FF;}
 a:visited {color:#0000FF;}
@@ -68,6 +69,11 @@ h1, h2, h3, h4, h5 {
     font-size: 1.4em;
 }
 
+.document img {
+    display:block;
+    margin: 0 auto;
+}
+
 .document h1 {
     display:none;
 }

doc/build/templates/site_base.mako

@@ -7,7 +7,7 @@
 </%text>
 
 <div style="text-align:right">
-<b>Quick Select:</b> <a href="/docs/05/">0.5</a> | <a href="/docs/04/">0.4</a> | <a href="/docs/03/">0.3</a><br/>
+<b>Quick Select:</b> <a href="/docs/06/">0.6</a> | <a href="/docs/05/">0.5</a> | <a href="/docs/04/">0.4</a><br/>
 <b>PDF Download:</b> <a href="${pathto('sqlalchemy_' + release.replace('.', '_') + '.pdf', 1)}">download</a>
 </div>
 

doc/build/testdocs.py

@@ -55,7 +55,7 @@ def teststring(s, name, globs=None, verbose=None, report=True,
     return runner.failures, runner.tries
 
 def replace_file(s, newfile):
-    engine = r"'(sqlite|postgres|mysql):///.*'"
+    engine = r"'(sqlite|postgresql|mysql):///.*'"
     engine = re.compile(engine, re.MULTILINE)
     s, n = re.subn(engine, "'sqlite:///" + newfile + "'", s)
     if not n:

examples/README

@@ -1,2 +0,0 @@
-placeholder
-

examples/adjacency_list/__init__.py

@@ -0,0 +1,16 @@
+"""
+An example of a dictionary-of-dictionaries structure mapped using
+an adjacency list model.
+
+E.g.::
+
+    node = TreeNode('rootnode')
+    node.append('node1')
+    node.append('node3')
+    session.add(node)
+    session.commit()
+
+    dump_tree(node)
+
+"""
+

examples/adjacency_list/adjacency_list.py

@@ -0,0 +1,133 @@
+from sqlalchemy import MetaData, Table, Column, Sequence, ForeignKey,\
+                        Integer, String, create_engine
+                        
+from sqlalchemy.orm import sessionmaker, mapper, relation, backref,\
+                                eagerload_all
+                                
+from sqlalchemy.orm.collections import attribute_mapped_collection
+
+metadata = MetaData()
+
+tree_table = Table('tree', metadata,
+    Column('id', Integer, primary_key=True),
+    Column('parent_id', Integer, ForeignKey('tree.id')),
+    Column('name', String(50), nullable=False)
+)
+
+class TreeNode(object):
+    def __init__(self, name, parent=None):
+        self.name = name
+        self.parent = parent
+        
+    def append(self, nodename):
+        self.children[nodename] = TreeNode(nodename, parent=self)
+    
+    def __repr__(self):
+        return "TreeNode(name=%r, id=%r, parent_id=%r)" % (
+                    self.name,
+                    self.id,
+                    self.parent_id
+                )
+                
+def dump_tree(node, indent=0):
+    
+    return "   " * indent + repr(node) + \
+                "\n" + \
+                "".join([
+                    dump_tree(c, indent +1) 
+                    for c in node.children.values()]
+                )
+                    
+
+mapper(TreeNode, tree_table, properties={
+    'children': relation(TreeNode, 
+
+                        # cascade deletions
+                        cascade="all",
+    
+                        # many to one + adjacency list - remote_side
+                        # is required to reference the 'remote' 
+                        # column in the join condition.
+                        backref=backref("parent", remote_side=tree_table.c.id),
+                         
+                        # children will be represented as a dictionary
+                        # on the "name" attribute.
+                        collection_class=attribute_mapped_collection('name'),
+                    )
+    })
+
+if __name__ == '__main__':
+    engine = create_engine('sqlite://', echo=True)
+
+    def msg(msg):
+        print "\n\n\n" + "-" * len(msg)
+        print msg
+        print "-" * len(msg)
+
+    msg("Creating Tree Table:")
+
+    metadata.create_all(engine)
+
+    # session.  using expire_on_commit=False
+    # so that the session's contents are not expired
+    # after each transaction commit.
+    session = sessionmaker(engine, expire_on_commit=False)()
+
+    node = TreeNode('rootnode')
+    node.append('node1')
+    node.append('node3')
+
+    node2 = TreeNode('node2')
+    node2.append('subnode1')
+    node.children['node2'] = node2
+    node.children['node2'].append('subnode2')
+
+    msg("Created new tree structure:")
+
+    print dump_tree(node)
+
+    msg("flush + commit:")
+
+    session.add(node)
+    session.commit()
+
+    msg("Tree After Save:")
+
+    print dump_tree(node)
+
+    node.append('node4')
+    node.children['node4'].append('subnode3')
+    node.children['node4'].append('subnode4')
+    node.children['node4'].children['subnode3'].append('subsubnode1')
+
+    # mark node1 as deleted and remove
+    session.delete(node.children['node1'])
+
+    msg("Removed node1.  flush + commit:")
+    session.commit()
+
+    print "\n\n\n----------------------------"
+    print "Tree After Save:"
+    print "----------------------------"
+
+    # expire the "children" collection so that
+    # it reflects the deletion of "node1".
+    session.expire(node, ['children'])
+    print dump_tree(node)
+
+    msg("Emptying out the session entirely, "
+        "selecting tree on root, using eager loading to join four levels deep.")
+    session.expunge_all()
+    node = session.query(TreeNode).\
+                        options(eagerload_all("children", "children", 
+                                                "children", "children")).\
+                        filter(TreeNode.name=="rootnode").\
+                        first()
+
+    msg("Full Tree:")
+    print dump_tree(node)
+
+    msg( "Marking root node as deleted, flush + commit:" )
+
+    session.delete(node)
+    session.commit()

examples/adjacencytree/basic_tree.py

@@ -1,129 +0,0 @@
-"""A basic Adjacency List model tree."""
-
-from sqlalchemy import MetaData, Table, Column, Sequence, ForeignKey
-from sqlalchemy import Integer, String
-from sqlalchemy.orm import create_session, mapper, relation, backref
-from sqlalchemy.orm.collections import attribute_mapped_collection
-
-metadata = MetaData('sqlite:///')
-metadata.bind.echo = True
-
-trees = Table('treenodes', metadata,
-    Column('id', Integer, Sequence('treenode_id_seq', optional=True),
-           primary_key=True),
-    Column('parent_id', Integer, ForeignKey('treenodes.id'), nullable=True),
-    Column('name', String(50), nullable=False))
-
-
-class TreeNode(object):
-    """a rich Tree class which includes path-based operations"""
-    def __init__(self, name):
-        self.name = name
-        self.parent = None
-        self.id = None
-        self.parent_id = None
-    def append(self, node):
-        if isinstance(node, str):
-            node = TreeNode(node)
-        node.parent = self
-        self.children[node.name] = node
-    def __repr__(self):
-        return self._getstring(0, False)
-    def __str__(self):
-        return self._getstring(0, False)
-    def _getstring(self, level, expand = False):
-        s = ('  ' * level) + "%s (%s,%s, %d)" % (
-            self.name, self.id,self.parent_id,id(self)) + '\n'
-        if expand:
-            s += ''.join([n._getstring(level+1, True)
-                          for n in self.children.values()])
-        return s
-    def print_nodes(self):
-        return self._getstring(0, True)
-
-mapper(TreeNode, trees, properties={
-    'children': relation(TreeNode, cascade="all",
-                         backref=backref("parent", remote_side=[trees.c.id]),
-                         collection_class=attribute_mapped_collection('name'),
-                         lazy=False, join_depth=3)})
-
-print "\n\n\n----------------------------"
-print "Creating Tree Table:"
-print "----------------------------"
-
-trees.create()
-
-node2 = TreeNode('node2')
-node2.append('subnode1')
-node = TreeNode('rootnode')
-node.append('node1')
-node.append(node2)
-node.append('node3')
-node.children['node2'].append('subnode2')
-
-print "\n\n\n----------------------------"
-print "Created new tree structure:"
-print "----------------------------"
-
-print node.print_nodes()
-
-print "\n\n\n----------------------------"
-print "Flushing:"
-print "----------------------------"
-
-session = create_session()
-session.add(node)
-session.flush()
-
-print "\n\n\n----------------------------"
-print "Tree After Save:"
-print "----------------------------"
-
-print node.print_nodes()
-
-node.append('node4')
-node.children['node4'].append('subnode3')
-node.children['node4'].append('subnode4')
-node.children['node4'].children['subnode3'].append('subsubnode1')
-del node.children['node1']
-
-print "\n\n\n----------------------------"
-print "Modified the tree"
-print "(added node4, node4/subnode3, node4/subnode4,"
-print "node4/subnode3/subsubnode1, deleted node1):"
-print "----------------------------"
-
-print node.print_nodes()
-
-print "\n\n\n----------------------------"
-print "Flushing:"
-print "----------------------------"
-session.flush()
-
-print "\n\n\n----------------------------"
-print "Tree After Save:"
-print "----------------------------"
-
-print node.print_nodes()
-
-nodeid = node.id
-
-print "\n\n\n----------------------------"
-print "Clearing session, selecting "
-print "tree new where node_id=%d:" % nodeid
-print "----------------------------"
-
-session.expunge_all()
-t = session.query(TreeNode).filter(TreeNode.id==nodeid)[0]
-
-print "\n\n\n----------------------------"
-print "Full Tree:"
-print "----------------------------"
-print t.print_nodes()
-
-print "\n\n\n----------------------------"
-print "Marking root node as deleted"
-print "and flushing:"
-print "----------------------------"
-session.delete(t)
-session.flush()

examples/association/__init__.py

@@ -0,0 +1,25 @@
+"""
+
+Examples illustrating the usage of the "association object" pattern,
+where an intermediary object associates two endpoint objects together.
+
+The first example illustrates a basic association from a User object
+to a collection or Order objects, each which references a collection of Item objects.
+
+The second example builds upon the first to add the Association Proxy extension.
+
+E.g.::
+
+    # create an order
+    order = Order('john smith')
+
+    # append an OrderItem association via the "itemassociations"
+    # collection with a custom price.
+    order.itemassociations.append(OrderItem(item('MySQL Crowbar'), 10.99))
+
+    # append two more Items via the transparent "items" proxy, which
+    # will create OrderItems automatically using the default price.
+    order.items.append(item('SA Mug'))
+    order.items.append(item('SA Hat'))
+
+"""

examples/beaker_caching/__init__.py

@@ -0,0 +1,77 @@
+"""
+Illustrates how to embed Beaker cache functionality within
+the Query object, allowing full cache control as well as the
+ability to pull "lazy loaded" attributes from long term cache 
+as well.
+
+In this demo, the following techniques are illustrated:
+
+* Using custom subclasses of Query
+* Basic technique of circumventing Query to pull from a 
+  custom cache source instead of the database.
+* Rudimental caching with Beaker, using "regions" which allow
+  global control over a fixed set of configurations.
+* Using custom MapperOption objects to configure options on 
+  a Query, including the ability to invoke the options 
+  deep within an object graph when lazy loads occur.
+
+E.g.::
+    
+    # query for Person objects, specifying cache
+    q = Session.query(Person).options(FromCache("default", "all_people"))
+    
+    # specify that each Person's "addresses" collection comes from
+    # cache too
+    q = q.options(FromCache("default", "by_person", Person.addresses))
+    
+    # query
+    print q.all()
+    
+To run, both SQLAlchemy and Beaker (1.4 or greater) must be
+installed or on the current PYTHONPATH. The demo will create a local
+directory for datafiles, insert initial data, and run. Running the
+demo a second time will utilize the cache files already present, and
+exactly one SQL statement against two tables will be emitted - the
+displayed result however will utilize dozens of lazyloads that all
+pull from cache.
+
+Three endpoint scripts, in order of complexity, are run as follows::
+
+   python examples/beaker_caching/helloworld.py
+
+   python examples/beaker_caching/relation_caching.py
+
+   python examples/beaker_caching/advanced.py
+
+   python examples/beaker_caching/local_session_caching.py
+
+
+Listing of files:
+
+    environment.py - Establish data / cache file paths, and configurations, 
+    bootstrap fixture data if necessary.
+
+    meta.py - Represent persistence structures which allow the usage of
+    Beaker caching with SQLAlchemy.  Introduces a query option called
+    FromCache.
+
+    model.py - The datamodel, which represents Person that has multiple
+    Address objects, each with PostalCode, City, Country
+
+    fixture_data.py - creates demo PostalCode, Address, Person objects
+    in the database.
+
+    helloworld.py - the basic idea.
+
+    relation_caching.py - Illustrates how to add cache options on
+    relation endpoints, so that lazyloads load from cache.
+
+    advanced.py - Further examples of how to use FromCache.  Combines
+    techniques from the first two scripts.
+
+    local_session_caching.py - Grok everything so far ?   This example
+    creates a new Beaker container that will persist data in a dictionary
+    which is local to the current session.   remove() the session
+    and the cache is gone.
+
+"""

examples/beaker_caching/advanced.py

@@ -0,0 +1,79 @@
+"""advanced.py
+
+Illustrate usage of Query combined with the FromCache option, 
+including front-end loading, cache invalidation, namespace techniques
+and collection caching.
+
+"""
+
+import environment
+from model import Person, Address, cache_address_bits
+from meta import Session, FromCache
+from sqlalchemy.orm import eagerload
+
+def load_name_range(start, end, invalidate=False):
+    """Load Person objects on a range of names.
+    
+    start/end are integers, range is then
+    "person <start>" - "person <end>".
+    
+    The cache option we set up is called "name_range", indicating 
+    a range of names for the Person class.
+    
+    The `Person.addresses` collections are also cached.  Its basically
+    another level of tuning here, as that particular cache option
+    can be transparently replaced with eagerload(Person.addresses). 
+    The effect is that each Person and his/her Address collection
+    is cached either together or separately, affecting the kind of
+    SQL that emits for unloaded Person objects as well as the distribution
+    of data within the cache.
+    """
+    q = Session.query(Person).\
+            filter(Person.name.between("person %.2d" % start, "person %.2d" % end)).\
+            options(cache_address_bits).\
+            options(FromCache("default", "name_range"))
+
+    # have the "addresses" collection cached separately
+    # each lazyload of Person.addresses loads from cache.
+    q = q.options(FromCache("default", "by_person", Person.addresses))
+    
+    # alternatively, eagerly load the "addresses" collection, so that they'd
+    # be cached together.   This issues a bigger SQL statement and caches
+    # a single, larger value in the cache per person rather than two
+    # separate ones.
+    #q = q.options(eagerload(Person.addresses))
+    
+    # if requested, invalidate the cache on current criterion.
+    if invalidate:
+        q.invalidate()
+        
+    return q.all()
+
+print "two through twelve, possibly from cache:\n"
+print ", ".join([p.name for p in load_name_range(2, 12)])
+
+print "\ntwenty five through forty, possibly from cache:\n"
+print ", ".join([p.name for p in load_name_range(25, 40)])
+
+# loading them again, no SQL is emitted
+print "\ntwo through twelve, from the cache:\n"
+print ", ".join([p.name for p in load_name_range(2, 12)])
+
+# but with invalidate, they are
+print "\ntwenty five through forty, invalidate first:\n"
+print ", ".join([p.name for p in load_name_range(25, 40, True)])
+
+# illustrate the address loading from either cache/already 
+# on the Person
+print "\n\nPeople plus addresses, two through twelve, addresses possibly from cache"
+for p in load_name_range(2, 12):
+    print p.format_full()
+
+# illustrate the address loading from either cache/already 
+# on the Person
+print "\n\nPeople plus addresses, two through twelve, addresses from cache"
+for p in load_name_range(2, 12):
+    print p.format_full()
+
+print "\n\nIf this was the first run of advanced.py, try "\
+        "a second run.  Only one SQL statement will be emitted."

examples/beaker_caching/environment.py

@@ -0,0 +1,44 @@
+"""environment.py
+
+Establish data / cache file paths, and configurations, 
+bootstrap fixture data if necessary.
+
+"""
+import meta, model, fixture_data
+from sqlalchemy import create_engine
+import os
+
+root = "./beaker_data/"
+
+if not os.path.exists(root):
+    raw_input("Will create datafiles in %r.\n"
+                "To reset the cache + database, delete this directory.\n"
+                "Press enter to continue.\n" % root
+                )
+    os.makedirs(root)
+    
+dbfile = os.path.join(root, "beaker_demo.db")
+engine = create_engine('sqlite:///%s' % dbfile, echo=True)
+meta.Session.configure(bind=engine)
+
+# configure the "default" cache region.
+meta.cache_manager.regions['default'] ={
+
+        # using type 'file' to illustrate
+        # serialized persistence.  In reality,
+        # use memcached.   Other backends
+        # are much, much slower.
+        'type':'file',
+        'data_dir':root,
+        'expire':3600,
+        
+        # set start_time to current time
+        # to re-cache everything
+        # upon application startup
+        #'start_time':time.time()
+    }
+
+installed = False
+if not os.path.exists(dbfile):
+    fixture_data.install()
+    installed = True

examples/beaker_caching/fixture_data.py

@@ -0,0 +1,49 @@
+"""fixture_data.py
+
+Installs some sample data.   Here we have a handful of postal codes for a few US/
+Canadian cities.   Then, 100 Person records are installed, each with a
+randomly selected postal code.
+
+"""
+from meta import Session, Base
+from model import City, Country, PostalCode, Person, Address
+import random
+
+def install():
+    Base.metadata.create_all(Session().bind)
+
+    data = [
+        ('Chicago', 'United States', ('60601', '60602', '60603', '60604')),
+        ('Montreal', 'Canada', ('H2S 3K9', 'H2B 1V4', 'H7G 2T8')),
+        ('Edmonton', 'Canada', ('T5J 1R9', 'T5J 1Z4', 'T5H 1P6')),
+        ('New York', 'United States', ('10001', '10002', '10003', '10004', '10005', '10006')),
+        ('San Francisco', 'United States', ('94102', '94103', '94104', '94105', '94107', '94108'))
+    ]
+    
+    countries = {}
+    all_post_codes = []
+    for city, country, postcodes in data:
+        try:
+            country = countries[country]
+        except KeyError:
+            countries[country] = country = Country(country)
+            
+        city = City(city, country)
+        pc = [PostalCode(code, city) for code in postcodes]
+        Session.add_all(pc)
+        all_post_codes.extend(pc)
+    
+    for i in xrange(1, 51):
+        person = Person(
+                    "person %.2d" % i,
+                    Address(
+                        street="street %.2d" % i, 
+                        postal_code=all_post_codes[random.randint(0, len(all_post_codes) - 1)]
+                    )
+                )
+        Session.add(person)
+
+    Session.commit()
+    
+    # start the demo fresh
+    Session.remove()

examples/beaker_caching/helloworld.py

@@ -0,0 +1,62 @@
+"""helloworld.py
+
+Illustrate how to load some data, and cache the results.
+
+"""
+
+import environment
+from model import Person
+from meta import Session, FromCache
+
+# load Person objects.  cache the result under the namespace "all_people".
+print "loading people...."
+people = Session.query(Person).options(FromCache("default", "all_people")).all()
+
+# remove the Session.  next query starts from scratch.
+Session.remove()
+
+# load again, using the same FromCache option. now they're cached 
+# under "all_people", no SQL is emitted.
+print "loading people....again!"
+people = Session.query(Person).options(FromCache("default", "all_people")).all()
+
+# want to load on some different kind of query ?  change the namespace 
+# you send to FromCache
+print "loading people two through twelve"
+people_two_through_twelve = Session.query(Person).\
+                            options(FromCache("default", "people_on_range")).\
+                            filter(Person.name.between("person 02", "person 12")).\
+                            all()
+
+# the data is cached under the "namespace" you send to FromCache, *plus*
+# the bind parameters of the query.    So this query, having
+# different literal parameters under "Person.name.between()" than the 
+# previous one, issues new SQL...
+print "loading people five through fifteen"
+people_five_through_fifteen = Session.query(Person).\
+                            options(FromCache("default", "people_on_range")).\
+                            filter(Person.name.between("person 05", "person 15")).\
+                            all()
+
+
+# ... but using the same params as are already cached, no SQL
+print "loading people two through twelve...again!"
+people_two_through_twelve = Session.query(Person).\
+                            options(FromCache("default", "people_on_range")).\
+                            filter(Person.name.between("person 02", "person 12")).\
+                            all()
+
+
+# invalidate the cache for the three queries we've done.  Recreate
+# each Query, which includes at the very least the same FromCache, 
+# same list of objects to be loaded, and the same parameters in the 
+# same order, then call invalidate().
+print "invalidating everything"
+Session.query(Person).options(FromCache("default", "all_people")).invalidate()
+Session.query(Person).\
+            options(FromCache("default", "people_on_range")).\
+            filter(Person.name.between("person 02", "person 12")).invalidate()
+Session.query(Person).\
+            options(FromCache("default", "people_on_range")).\
+            filter(Person.name.between("person 05", "person 15")).invalidate()
+

examples/beaker_caching/local_session_caching.py

@@ -0,0 +1,95 @@
+"""local_session_caching.py
+
+Create a new Beaker cache type + a local region that will store
+cached data local to the current Session.
+
+This is an advanced example which assumes familiarity
+with the basic operation of CachingQuery.
+
+"""
+
+from beaker import cache, container
+import collections
+
+class ScopedSessionNamespace(container.MemoryNamespaceManager):
+    """A Beaker cache type which will cache objects locally on 
+    the current session.
+
+    When used with the query_cache system, the effect is that the objects
+    in the cache are the same as that within the session - the merge()
+    is a formality that doesn't actually create a second instance.  
+    This makes it safe to use for updates of data from an identity
+    perspective (still not ideal for deletes though).
+
+    When the session is removed, the cache is gone too, so the cache
+    is automatically disposed upon session.remove().
+
+    """
+    
+    def __init__(self, namespace, scoped_session, **kwargs):
+        """__init__ is called by Beaker itself."""
+        
+        container.NamespaceManager.__init__(self, namespace)
+        self.scoped_session = scoped_session
+    
+    @classmethod
+    def create_session_container(cls, beaker_name, scoped_session):
+        """Create a new session container for a given scoped_session."""
+        
+        def create_namespace(namespace, **kw):
+            return cls(namespace, scoped_session, **kw)
+        cache.clsmap[beaker_name] = create_namespace
+        
+    @property
+    def dictionary(self):
+        """Return the cache dictionary used by this MemoryNamespaceManager."""
+        
+        sess = self.scoped_session()
+        try:
+            nscache = sess._beaker_cache
+        except AttributeError:
+            sess._beaker_cache = nscache = collections.defaultdict(dict)
+        return nscache[self.namespace]
+
+
+if __name__ == '__main__':
+    import environment
+    import meta
+    
+    # create a Beaker container type called "ext:local_session".
+    # it will reference the ScopedSession in meta.
+    ScopedSessionNamespace.create_session_container("ext:local_session", meta.Session)
+    
+    # set up a region based on this new container type.
+    meta.cache_manager.regions['local_session'] ={'type':'ext:local_session'}
+    
+    from model import Person
+    
+    # query to load Person by name, with criterion
+    # of "person 10"
+    q = meta.Session.query(Person).\
+                    options(meta.FromCache("local_session", "by_name")).\
+                    filter(Person.name=="person 10")
+                    
+    # load from DB
+    person10 = q.one()
+    
+    # next call, the query is cached.
+    person10 = q.one()
+
+    # clear out the Session.  The "_beaker_cache" dictionary
+    # disappears with it.
+    meta.Session.remove()
+    
+    # query calls from DB again
+    person10 = q.one()
+    
+    # identity is preserved - person10 is the *same* object that's
+    # ultimately inside the cache.   So it is safe to manipulate
+    # the not-queried-for attributes of objects when using such a 
+    # cache without the need to invalidate - however, any change 
+    # that would change the results of a cached query, such as 
+    # inserts, deletes, or modification to attributes that are 
+    # part of query criterion, still require careful invalidation.
+    cache, key = q._get_cache_plus_key()
+    assert person10 is cache.get(key)[0]

examples/beaker_caching/meta.py

@@ -0,0 +1,221 @@
+"""meta.py
+
+Represent persistence structures which allow the usage of
+Beaker caching with SQLAlchemy.
+
+The three new concepts introduced here are:
+
+ * CachingQuery - a Query subclass that caches and
+   retrieves results in/from Beaker.
+ * FromCache - a query option that establishes caching
+   parameters on a Query
+ * _params_from_query - extracts value parameters from 
+   a Query.
+
+The rest of what's here are standard SQLAlchemy and
+Beaker constructs.
+   
+"""
+from sqlalchemy.orm import scoped_session, sessionmaker
+from sqlalchemy.orm.interfaces import MapperOption
+from sqlalchemy.orm.query import Query
+from sqlalchemy.sql import visitors
+from sqlalchemy.ext.declarative import declarative_base
+from beaker import cache
+
+class CachingQuery(Query):
+    """A Query subclass which optionally loads full results from a Beaker 
+    cache region.
+    
+    The CachingQuery is instructed to load from cache based on two optional
+    attributes configured on the instance, called 'cache_region' and 'cache_namespace'.
+    
+    When these attributes are present, any iteration of the Query will configure
+    a Beaker cache against this region and a generated namespace, which takes
+    into account the 'cache_namespace' name as well as the entities this query
+    is created against (i.e. the columns and classes sent to the constructor).
+    The 'cache_namespace' is a string name that represents a particular structure
+    of query.  E.g. a query that filters on a name might use the name "by_name",
+    a query that filters on a date range to a joined table might use the name
+    "related_date_range".
+    
+    The Query then attempts to retrieved a cached value using a key, which
+    is generated from all the parameterized values present in the Query.  In
+    this way, the combination of "cache_namespace" and embedded parameter values
+    correspond exactly to the lexical structure of a SQL statement combined
+    with its bind parameters.   If no such key exists then the ultimate SQL
+    is emitted and the objects loaded.
+    
+    The returned objects, if loaded from cache, are merged into the Query's
+    session using Session.merge(load=False), which is a fast performing
+    method to ensure state is present.
+
+    The FromCache mapper option below represents the "public" method of 
+    configuring the "cache_region" and "cache_namespace" attributes,
+    and includes the ability to be invoked upon lazy loaders embedded
+    in an object graph.
+    
+    """
+    
+    def _get_cache_plus_key(self):
+        """For a query with cache_region and cache_namespace configured,
+        return the correspoinding Cache instance and cache key, based
+        on this query's current criterion and parameter values.
+        
+        """
+        if not hasattr(self, 'cache_region'):
+            raise ValueError("This Query does not have caching parameters configured.")
+            
+        # cache namespace - the token handed in by the 
+        # option + class we're querying against
+        namespace = " ".join([self.cache_namespace] + [str(x) for x in self._entities])
+        
+        # memcached wants this
+        namespace = namespace.replace(' ', '_')
+        
+        if hasattr(self, 'cache_key'):
+            # if a hardcoded cache_key was attached, use that
+            cache_key = self.cache_key
+        else:
+            # cache key - the value arguments from this query's parameters.
+            args = _params_from_query(self)
+            cache_key = " ".join([str(x) for x in args])
+        
+        # get cache
+        cache = cache_manager.get_cache_region(namespace, self.cache_region)
+        
+        # optional - hash the cache_key too for consistent length
+        # import uuid
+        # cache_key= str(uuid.uuid5(uuid.NAMESPACE_DNS, cache_key))
+        
+        return cache, cache_key
+        
+    def __iter__(self):
+        """override __iter__ to pull results from Beaker
+           if particular attributes have been configured.
+        """
+        if hasattr(self, 'cache_region'):
+            cache, cache_key = self._get_cache_plus_key()
+            ret = cache.get_value(cache_key, createfunc=lambda: list(Query.__iter__(self)))
+            return self.merge_result(ret, load=False)
+        else:
+            return Query.__iter__(self)
+
+    def invalidate(self):
+        """Invalidate the cache represented in this Query."""
+
+        cache, cache_key = self._get_cache_plus_key()
+        cache.remove(cache_key)
+
+    def set_value(self, value):
+        """Set the value in the cache for this query."""
+
+        cache, cache_key = self._get_cache_plus_key()
+        cache.put(cache_key, value)        
+
+class FromCache(MapperOption):
+    """A MapperOption which configures a Query to use a particular 
+    cache namespace and region.
+    
+    Can optionally be configured to be invoked for a specific 
+    lazy loader.
+    
+    """
+    def __init__(self, region, namespace, key=None, cache_key=None):
+        """Construct a new FromCache.
+        
+        :param region: the cache region.  Should be a
+        region configured in the Beaker CacheManager.
+        
+        :param namespace: the cache namespace.  Should
+        be a name uniquely describing the target Query's
+        lexical structure.
+        
+        :param key: optional.  A Class.attrname which
+        indicates a particular class relation() whose
+        lazy loader should be pulled from the cache.
+        
+        :param cache_key: optional.  A string cache key 
+        that will serve as the key to the query.   Use this
+        if your query has a huge amount of parameters (such
+        as when using in_()) which correspond more simply to 
+        some other identifier.
+
+        """
+        self.region = region
+        self.namespace = namespace
+        self.cache_key = cache_key
+        if key:
+            self.cls_ = key.property.parent.class_
+            self.propname = key.property.key
+            self.propagate_to_loaders = True
+        else:
+            self.cls_ = self.propname = None
+            self.propagate_to_loaders = False
+    
+    def _set_query_cache(self, query):
+        """Configure this FromCache's region and namespace on a query."""
+        
+        if hasattr(query, 'cache_region'):
+            raise ValueError("This query is already configured "
+                            "for region %r namespace %r" % 
+                            (query.cache_region, query.cache_namespace)
+                        )
+        query.cache_region = self.region
+        query.cache_namespace = self.namespace
+        if self.cache_key:
+            query.cache_key = self.cache_key
+        
+    def process_query_conditionally(self, query):
+        """Process a Query that is used within a lazy loader.
+        
+        (the process_query_conditionally() method is a SQLAlchemy
+        hook invoked only within lazyload.)
+        
+        """
+        if self.cls_ is not None and query._current_path:
+            mapper, key = query._current_path[-2:]
+            if issubclass(mapper.class_, self.cls_) and key == self.propname:
+                self._set_query_cache(query)
+
+    def process_query(self, query):
+        """Process a Query during normal loading operation."""
+        
+        if self.cls_ is None:
+            self._set_query_cache(query)
+
+def _params_from_query(query):
+    """Pull the bind parameter values from a query.
+    
+    This takes into account any scalar attribute bindparam set up.
+    
+    E.g. params_from_query(query.filter(Cls.foo==5).filter(Cls.bar==7)))
+    would return [5, 7].
+    
+    """
+    v = []
+    def visit_bindparam(bind):
+        value = query._params.get(bind.key, bind.value)
+        
+        # lazyloader may dig a callable in here, intended
+        # to late-evaluate params after autoflush is called.
+        # convert to a scalar value.
+        if callable(value):
+            value = value()
+            
+        v.append(value)
+    if query._criterion is not None:
+        visitors.traverse(query._criterion, {}, {'bindparam':visit_bindparam})
+    return v
+
+# Beaker CacheManager.  A home base for cache configurations.
+# Configured at startup in __init__.py
+cache_manager = cache.CacheManager()
+
+# global application session.
+# configured at startup in __init__.py
+Session = scoped_session(sessionmaker(query_cls=CachingQuery))
+
+# global declarative base class.
+Base = declarative_base()
+

examples/beaker_caching/model.py

@@ -0,0 +1,103 @@
+"""Model.   We are modeling Person objects with a collection
+of Address objects.  Each Address has a PostalCode, which 
+in turn references a City and then a Country:
+
+Person --(1..n)--> Address
+Address --(has a)--> PostalCode
+PostalCode --(has a)--> City
+City --(has a)--> Country
+
+"""
+from sqlalchemy import Column, Integer, String, ForeignKey
+from sqlalchemy.orm import relation
+from meta import Base, FromCache, Session
+
+class Country(Base):
+    __tablename__ = 'country'
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String(100), nullable=False)
+    
+    def __init__(self, name):
+        self.name = name
+
+class City(Base):
+    __tablename__ = 'city'
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String(100), nullable=False)
+    country_id = Column(Integer, ForeignKey('country.id'), nullable=False)
+    country = relation(Country)
+
+    def __init__(self, name, country):
+        self.name = name
+        self.country = country
+
+class PostalCode(Base):
+    __tablename__ = 'postal_code'
+
+    id = Column(Integer, primary_key=True)
+    code = Column(String(10), nullable=False)
+    city_id = Column(Integer, ForeignKey('city.id'), nullable=False)
+    city = relation(City)
+    
+    @property
+    def country(self):
+        return self.city.country
+        
+    def __init__(self, code, city):
+        self.code = code
+        self.city = city
+    
+class Address(Base):
+    __tablename__ = 'address'
+
+    id = Column(Integer, primary_key=True)
+    person_id = Column(Integer, ForeignKey('person.id'), nullable=False)
+    street = Column(String(200), nullable=False)
+    postal_code_id = Column(Integer, ForeignKey('postal_code.id'))
+    postal_code = relation(PostalCode)
+    
+    @property
+    def city(self):
+        return self.postal_code.city
+        
+    @property
+    def country(self):
+        return self.postal_code.country
+    
+    def __str__(self):
+        return "%s\t"\
+              "%s, %s\t"\
+              "%s" % (self.street, self.city.name, 
+                self.postal_code.code, self.country.name)
+        
+class Person(Base):
+    __tablename__ = 'person'
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String(100), nullable=False)
+    addresses = relation(Address, collection_class=set)
+
+    def __init__(self, name, *addresses):
+        self.name = name
+        self.addresses = set(addresses)
+
+    def __str__(self):
+        return self.name
+    
+    def __repr__(self):
+        return "Person(name=%r)" % self.name
+        
+    def format_full(self):
+        return "\t".join([str(x) for x in [self] + list(self.addresses)])
+        
+# Caching options.   A set of three FromCache options
+# which can be applied to Query(), causing the "lazy load"
+# of these attributes to be loaded from cache.
+cache_address_bits = (
+                FromCache("default", "byid", PostalCode.city), 
+                FromCache("default", "byid", City.country), 
+                FromCache("default", "byid", Address.postal_code),
+            )
+

examples/beaker_caching/relation_caching.py

@@ -0,0 +1,25 @@
+"""relation_caching.py
+
+Load a set of Person and Address objects, specifying that 
+related PostalCode, City, Country objects should be pulled from long 
+term cache.
+
+"""
+import environment
+from model import Person, Address, cache_address_bits
+from meta import Session
+from sqlalchemy.orm import eagerload
+import os
+
+for p in Session.query(Person).options(eagerload(Person.addresses), cache_address_bits):
+    print p.format_full()
+
+
+print "\n\nIf this was the first run of relation_caching.py, SQL was likely emitted to "\
+        "load postal codes, cities, countries.\n"\
+        "If run a second time, only a single SQL statement will run - all "\
+        "related data is pulled from cache.\n"\
+        "To clear the cache, delete the directory %r.  \n"\
+        "This will cause a re-load of cities, postal codes and countries on "\
+        "the next run.\n"\
+        % os.path.join(environment.root, 'container_file')

examples/collections/large_collection.py

@@ -1,93 +0,0 @@
-"""illlustrates techniques for dealing with very large collections.
-
-Also see the docs regarding the new "dynamic" relation option, which 
-presents a more refined version of some of these patterns.
-"""
-
-from sqlalchemy import MetaData, Table, Column, Integer, String, ForeignKey
-from sqlalchemy.orm import (mapper, relation, create_session, MapperExtension,
-    object_session)
-
-meta = MetaData('sqlite://')
-meta.bind.echo = True
-
-org_table = Table('organizations', meta, 
-    Column('org_id', Integer, primary_key=True),
-    Column('org_name', String(50), nullable=False, key='name'),
-    mysql_engine='InnoDB')
-    
-member_table = Table('members', meta,
-    Column('member_id', Integer, primary_key=True),
-    Column('member_name', String(50), nullable=False, key='name'),
-    Column('org_id', Integer, ForeignKey('organizations.org_id')), 
-    mysql_engine='InnoDB')
-meta.create_all()    
-    
-class Organization(object):
-    def __init__(self, name):
-        self.name = name
-    member_query = property(lambda self:object_session(self).query(Member).with_parent(self), 
-        doc="""locate a subset of the members associated with this Organization""")
-    
-class Member(object):
-    def __init__(self, name):
-        self.name = name
-
-# note that we can also place "ON DELETE CASCADE" on the tables themselves,
-# instead of using this extension
-class DeleteMemberExt(MapperExtension):
-    """will delete child Member objects in one pass when Organizations are deleted"""
-    def before_delete(self, mapper, connection, instance):
-        connection.execute(member_table.delete(member_table.c.org_id==instance.org_id))
-
-mapper(Organization, org_table, extension=DeleteMemberExt(), properties = {
-    # set up the relationship with "lazy=None" so no loading occurs (even lazily),
-    # "cascade='all, delete-orphan'" to declare Member objects as local to their parent Organization,
-    # "passive_deletes=True" so that the "delete, delete-orphan" cascades do not load in the child objects
-    # upon deletion
-    'members' : relation(Member, lazy=None, passive_deletes=True, cascade="all, delete-orphan")
-})
-
-mapper(Member, member_table)
-
-sess = create_session()
-
-# create org with some members
-org = Organization('org one')
-org.members.append(Member('member one'))
-org.members.append(Member('member two'))
-org.members.append(Member('member three'))
-
-sess.add(org)
-
-print "-------------------------\nflush one - save org + 3 members"
-sess.flush()
-sess.expunge_all()
-
-# reload. load the org and some child members
-print "-------------------------\nload subset of members"
-org = sess.query(Organization).get(org.org_id)
-members = org.member_query.filter(member_table.c.name.like('%member t%')).all()
-print members
-
-sess.expunge_all()
-
-
-# reload.  create some more members and flush, without loading any of the original members
-org = sess.query(Organization).get(org.org_id)
-org.members.append(Member('member four'))
-org.members.append(Member('member five'))
-org.members.append(Member('member six'))
-
-print "-------------------------\nflush two - save 3 more members"
-sess.flush()
-
-sess.expunge_all()
-org = sess.query(Organization).get(org.org_id)
-
-# now delete.  note that this will explictily delete members four, five and six because they are in the session,
-# but will not issue individual deletes for members one, two and three, nor will it load them.
-sess.delete(org)
-print "-------------------------\nflush three - delete org, delete members in one statement"
-sess.flush()
-

examples/custom_attributes/__init__.py

@@ -0,0 +1,8 @@
+"""
+Two examples illustrating modifications to SQLAlchemy's attribute management system.
+
+``listen_for_events.py`` illustrates the usage of :class:`~sqlalchemy.orm.interfaces.AttributeExtension` to intercept attribute events.  It additionally illustrates a way to automatically attach these listeners to all class attributes using a :class:`~sqlalchemy.orm.interfaces.InstrumentationManager`.
+
+``custom_management.py`` illustrates much deeper usage of :class:`~sqlalchemy.orm.interfaces.InstrumentationManager` as well as collection adaptation, to completely change the underlying method used to store state on an object.   This example was developed to illustrate techniques which would be used by other third party object instrumentation systems to interact with SQLAlchemy's event system and is only intended for very intricate framework integrations.
+
+"""

examples/custom_attributes/custom_management.py

@@ -82,6 +82,7 @@ class MyClass(object):
 
 class MyCollectionAdapter(object):
     """An wholly alternative instrumentation implementation."""
+    
     def __init__(self, key, state, collection):
         self.key = key
         self.state = state

examples/derived_attributes/__init__.py

@@ -0,0 +1,10 @@
+"""Illustrates a clever technique using Python descriptors to create custom attributes representing SQL expressions when used at the class level, and Python expressions when used at the instance level.   In some cases this technique replaces the need to configure the attribute in the mapping, instead relying upon ordinary Python behavior to create custom expression components.
+
+E.g.::
+
+    class BaseInterval(object):
+        @hybrid
+        def contains(self,point):
+            return (self.start <= point) & (point < self.end)
+
+"""

examples/derived_attributes/attributes.py

@@ -1,6 +1,3 @@
-"""A couple of helper descriptors to allow to use the same code as query
-criterion creators and as instance code. As this doesn't do advanced
-magic recompiling, you can only use basic expression-like code."""
 
 import new
 

examples/dynamic_dict/__init__.py

@@ -0,0 +1,5 @@
+"""Illustrates how to place a dictionary-like facade on top of a "dynamic" relation, so
+that dictionary operations (assuming simple string keys) can operate upon a large 
+collection without loading the full collection at once.
+
+"""

examples/dynamic_dict/dynamic_dict.py

@@ -1,24 +1,13 @@
-"""Illustrates how to place a dictionary-like facade on top of a dynamic_loader, so
-that dictionary operations (assuming simple string keys) can operate upon a large 
-collection without loading the full collection at once.
-
-This is something that may eventually be added as a feature to dynamic_loader() itself.
-
-Similar approaches could be taken towards sets and dictionaries with non-string keys 
-although the hash policy of the members would need to be distilled into a filter() criterion.
-
-"""
-
-class MyProxyDict(object):
+class ProxyDict(object):
     def __init__(self, parent, collection_name, childclass, keyname):
         self.parent = parent
         self.collection_name = collection_name
         self.childclass = childclass
         self.keyname = keyname
-        
+    
+    @property
     def collection(self):
         return getattr(self.parent, self.collection_name)
-    collection = property(collection)
     
     def keys(self):
         descriptor = getattr(self.childclass, self.keyname)
@@ -41,43 +30,58 @@ class MyProxyDict(object):
 
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy import create_engine, Column, Integer, String, ForeignKey
-from sqlalchemy.orm import sessionmaker, dynamic_loader
+from sqlalchemy.orm import sessionmaker, relation
 
-Base = declarative_base(engine=create_engine('sqlite://'))
+engine=create_engine('sqlite://', echo=True)
+Base = declarative_base(engine)
 
-class MyParent(Base):
+class Parent(Base):
     __tablename__ = 'parent'
     id = Column(Integer, primary_key=True)
     name = Column(String(50))
-    _collection = dynamic_loader("MyChild", cascade="all, delete-orphan")
+    _collection = relation("Child", lazy="dynamic", cascade="all, delete-orphan")
     
+    @property
     def child_map(self):
-        return MyProxyDict(self, '_collection', MyChild, 'key')
-    child_map = property(child_map)
+        return ProxyDict(self, '_collection', Child, 'key')
     
-class MyChild(Base):
+class Child(Base):
     __tablename__ = 'child'
     id = Column(Integer, primary_key=True)
     key = Column(String(50))
     parent_id = Column(Integer, ForeignKey('parent.id'))
 
+    def __repr__(self):
+        return "Child(key=%r)" % self.key
     
 Base.metadata.create_all()
 
 sess = sessionmaker()()
 
-p1 = MyParent(name='p1')
+p1 = Parent(name='p1')
 sess.add(p1)
 
-p1.child_map['k1'] = k1 = MyChild(key='k1')
-p1.child_map['k2'] = k2 = MyChild(key='k2')
+print "\n---------begin setting nodes, autoflush occurs\n"
+p1.child_map['k1'] = Child(key='k1')
+p1.child_map['k2'] = Child(key='k2')
 
-assert p1.child_map.keys() == ['k1', 'k2']
+# this will autoflush the current map.
+# ['k1', 'k2']
+print "\n---------print keys - flushes first\n"
+print p1.child_map.keys()
 
-assert p1.child_map['k1'] is k1
+# k1
+print "\n---------print 'k1' node\n"
+print p1.child_map['k1']
 
-p1.child_map['k2'] = k2b = MyChild(key='k2')
-assert p1.child_map['k2'] is k2b
+print "\n---------update 'k2' node - must find existing, and replace\n"
+p1.child_map['k2'] = Child(key='k2')
 
-assert sess.query(MyChild).all() == [k1, k2b]
+print "\n---------print 'k2' key - flushes first\n"
+# k2
+print p1.child_map['k2']
+
+print "\n---------print all child nodes\n"
+# [k1, k2b]
+print sess.query(Child).all()
 

examples/elementtree/__init__.py

@@ -0,0 +1,30 @@
+"""
+Illustrates three strategies for persisting and querying XML documents as represented by 
+ElementTree in a relational database.   The techniques do not apply any mappings to the ElementTree objects directly, so are compatible with the native cElementTree as well as lxml, and can be adapted to suit any kind of DOM representation system.   Querying along xpath-like strings is illustrated as well.
+
+In order of complexity:
+
+* ``pickle.py`` - Quick and dirty, serialize the whole DOM into a BLOB column.  While the example
+  is very brief, it has very limited functionality.
+* ``adjacency_list.py`` - Each DOM node is stored in an individual table row, with attributes
+  represented in a separate table.  The nodes are associated in a hierarchy using an adjacency list
+  structure.  A query function is introduced which can search for nodes along any path with a given
+  structure of attributes, basically a (very narrow) subset of xpath.
+* ``optimized_al.py`` - Uses the same strategy as ``adjacency_list.py``, but adds a
+  :class:`~sqlalchemy.orm.interfaces.MapperExtension` which optimizes how the hierarchical structure  
+  is loaded, such that the full set of DOM nodes are loaded within a single table result set, and
+  are organized hierarchically as they are received during a load.
+
+E.g.::
+    
+    # parse an XML file and persist in the database
+    doc = ElementTree.parse("test.xml")
+    session.add(Document(file, doc))
+    session.commit()
+    
+    # locate documents with a certain path/attribute structure 
+    for document in find_document('/somefile/header/field2[@attr=foo]'):
+        # dump the XML
+        print document
+    
+"""

examples/elementtree/adjacency_list.py

@@ -13,16 +13,6 @@ from sqlalchemy.orm import mapper, relation, create_session, lazyload
 
 import sys, os, StringIO, re
 
-import logging
-logging.basicConfig()
-
-# uncomment to show SQL statements
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
-
-# uncomment to show SQL statements and result sets
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG)
-
-
 from xml.etree import ElementTree
 
 meta = MetaData()

examples/elementtree/optimized_al.py

@@ -12,16 +12,6 @@ from sqlalchemy.orm import mapper, relation, create_session, lazyload
 
 import sys, os, StringIO, re
 
-import logging
-logging.basicConfig()
-
-# uncomment to show SQL statements
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
-
-# uncomment to show SQL statements and result sets
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG)
-
-
 from xml.etree import ElementTree
 
 meta = MetaData()

examples/elementtree/pickle.py

@@ -12,15 +12,6 @@ from sqlalchemy.orm import mapper, create_session
 
 import sys, os
 
-import logging
-logging.basicConfig()
-
-# uncomment to show SQL statements
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
-
-# uncomment to show SQL statements and result sets
-#logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG)
-
 from xml.etree import ElementTree
 
 engine = create_engine('sqlite://')

examples/graphs/__init__.py

@@ -0,0 +1,9 @@
+"""
+An example of persistence for a directed graph structure.   The graph is stored as a collection of edges, each referencing both a "lower" and an "upper" node in a table of nodes.  Basic persistence and querying for lower- and upper- neighbors are illustrated::
+
+    n2 = Node(2)
+    n5 = Node(5)
+    n2.add_neighbor(n5)
+    print n2.higher_neighbors()
+
+"""

examples/inheritance/__init__.py

@@ -0,0 +1,4 @@
+"""
+Working examples of single-table, joined-table, and concrete-table inheritance as described in :ref:`datamapping_inheritance`.
+
+"""

examples/inheritance/polymorph.py

@@ -56,8 +56,6 @@ class Company(object):
         return "Company %s" % self.name
 
 
-person_join = people.outerjoin(engineers).outerjoin(managers)
-
 person_mapper = mapper(Person, people, polymorphic_on=people.c.type, polymorphic_identity='person')
 mapper(Engineer, engineers, inherits=person_mapper, polymorphic_identity='engineer')
 mapper(Manager, managers, inherits=person_mapper, polymorphic_identity='manager')
@@ -75,7 +73,6 @@ c.employees.append(Engineer(name='wally', status='CGG', engineer_name='engineer2
 c.employees.append(Manager(name='jsmith', status='ABA', manager_name='manager2'))
 session.add(c)
 
-print session.new
 session.flush()
 session.expunge_all()
 
@@ -96,7 +93,7 @@ session.expunge_all()
 
 c = session.query(Company).get(1)
 for e in c.employees:
-    print e, e._sa_instance_state.key
+    print e
 
 session.delete(c)
 session.flush()

examples/inheritance/single.py

@@ -45,8 +45,6 @@ person_mapper = mapper(Person, employees_table, polymorphic_on=employees_table.c
 manager_mapper = mapper(Manager, inherits=person_mapper, polymorphic_identity='manager')
 engineer_mapper = mapper(Engineer, inherits=person_mapper, polymorphic_identity='engineer')
 
-
-
 mapper(Company, companies, properties={
     'employees': relation(Person, lazy=True, backref='company')
 })
@@ -65,7 +63,7 @@ session.expunge_all()
 
 c = session.query(Company).get(1)
 for e in c.employees:
-    print e, e._sa_instance_state.key, e.company
+    print e, e.company
 
 print "\n"
 
@@ -80,7 +78,7 @@ session.expunge_all()
 
 c = session.query(Company).get(1)
 for e in c.employees:
-    print e, e._sa_instance_state.key
+    print e
 
 session.delete(c)
 session.flush()

examples/large_collection/__init__.py

@@ -0,0 +1,8 @@
+"""Large collection example.
+
+Illustrates the options to use with :func:`~sqlalchemy.orm.relation()` when the list of related objects is very large, including:
+
+* "dynamic" relations which query slices of data as accessed
+* how to use ON DELETE CASCADE in conjunction with ``passive_deletes=True`` to greatly improve the performance of related collection deletion.
+
+"""

examples/large_collection/large_collection.py

@@ -0,0 +1,94 @@
+
+from sqlalchemy import (MetaData, Table, Column, Integer, String, ForeignKey,
+                        create_engine)
+from sqlalchemy.orm import (mapper, relation, sessionmaker)
+
+
+meta = MetaData()
+
+org_table = Table('organizations', meta, 
+    Column('org_id', Integer, primary_key=True),
+    Column('org_name', String(50), nullable=False, key='name'),
+    mysql_engine='InnoDB')
+    
+member_table = Table('members', meta,
+    Column('member_id', Integer, primary_key=True),
+    Column('member_name', String(50), nullable=False, key='name'),
+    Column('org_id', Integer, ForeignKey('organizations.org_id', ondelete="CASCADE")),
+    mysql_engine='InnoDB')
+    
+    
+class Organization(object):
+    def __init__(self, name):
+        self.name = name
+    
+class Member(object):
+    def __init__(self, name):
+        self.name = name
+
+mapper(Organization, org_table, properties = {
+    'members' : relation(Member, 
+        # Organization.members will be a Query object - no loading
+        # of the entire collection occurs unless requested
+        lazy="dynamic", 
+        
+        # Member objects "belong" to their parent, are deleted when 
+        # removed from the collection
+        cascade="all, delete-orphan",
+        
+        # "delete, delete-orphan" cascade does not load in objects on delete,
+        # allows ON DELETE CASCADE to handle it.
+        # this only works with a database that supports ON DELETE CASCADE - 
+        # *not* sqlite or MySQL with MyISAM
+        passive_deletes=True, 
+    )
+})
+
+mapper(Member, member_table)
+
+if __name__ == '__main__':
+    engine = create_engine("mysql://scott:tiger@localhost/test", echo=True)
+    meta.create_all(engine)
+    
+    # expire_on_commit=False means the session contents
+    # will not get invalidated after commit.
+    sess = sessionmaker(engine, expire_on_commit=False)()
+
+    # create org with some members
+    org = Organization('org one')
+    org.members.append(Member('member one'))
+    org.members.append(Member('member two'))
+    org.members.append(Member('member three'))
+
+    sess.add(org)
+
+    print "-------------------------\nflush one - save org + 3 members\n"
+    sess.commit()
+
+    # the 'members' collection is a Query.  it issues 
+    # SQL as needed to load subsets of the collection.
+    print "-------------------------\nload subset of members\n"
+    members = org.members.filter(member_table.c.name.like('%member t%')).all()
+    print members
+
+    # new Members can be appended without any
+    # SQL being emitted to load the full collection
+    org.members.append(Member('member four'))
+    org.members.append(Member('member five'))
+    org.members.append(Member('member six'))
+
+    print "-------------------------\nflush two - save 3 more members\n"
+    sess.commit()
+
+    # delete the object.   Using ON DELETE CASCADE 
+    # SQL is only emitted for the head row - the Member rows 
+    # disappear automatically without the need for additional SQL.
+    sess.delete(org)
+    print "-------------------------\nflush three - delete org, delete members in one statement\n"
+    sess.commit()
+    
+    print "-------------------------\nno Member rows should remain:\n"
+    print sess.query(Member).count()
+    
+    print "------------------------\ndone.  dropping tables."
+    meta.drop_all(engine)

examples/nested_sets/__init__.py

@@ -0,0 +1,4 @@
+"""
+Illustrates a rudimentary way to implement the "nested sets" pattern for hierarchical data using the SQLAlchemy ORM.
+
+"""

examples/pickle/custom_pickler.py

@@ -1,91 +0,0 @@
-"""illustrates one way to use a custom pickler that is session-aware."""
-
-from sqlalchemy import MetaData, Table, Column, Integer, String, PickleType
-from sqlalchemy.orm import (mapper, create_session, MapperExtension,
-    class_mapper, EXT_CONTINUE)
-from sqlalchemy.orm.session import object_session
-from cStringIO import StringIO
-from pickle import Pickler, Unpickler
-import threading
-
-meta = MetaData('sqlite://')
-meta.bind.echo = True
-
-class MyExt(MapperExtension):
-    def populate_instance(self, mapper, selectcontext, row, instance, **flags):
-        MyPickler.sessions.current = selectcontext.session
-        return EXT_CONTINUE
-    def before_insert(self, mapper, connection, instance):
-        MyPickler.sessions.current = object_session(instance)
-        return EXT_CONTINUE
-    def before_update(self, mapper, connection, instance):
-        MyPickler.sessions.current = object_session(instance)
-        return EXT_CONTINUE
-    
-class MyPickler(object):
-    sessions = threading.local()
-
-    def persistent_id(self, obj):
-        if getattr(obj, "id", None) is None:
-            sess = MyPickler.sessions.current
-            newsess = create_session(bind=sess.connection(class_mapper(Bar)))
-            newsess.add(obj)
-            newsess.flush()
-        key = "%s:%s" % (type(obj).__name__, obj.id)
-        return key
-
-    def persistent_load(self, key):
-        name, ident = key.split(":")
-        sess = MyPickler.sessions.current
-        return sess.query(Bar).get(ident)
-
-    def dumps(self, graph, protocol):
-        src = StringIO()
-        pickler = Pickler(src)
-        pickler.persistent_id = self.persistent_id
-        pickler.dump(graph)
-        return src.getvalue()
-
-    def loads(self, data):
-        dst = StringIO(data)
-        unpickler = Unpickler(dst)
-        unpickler.persistent_load = self.persistent_load
-        return unpickler.load()
-
-foo_table = Table('foo', meta, 
-    Column('id', Integer, primary_key=True),
-    Column('bar', PickleType(pickler=MyPickler()), nullable=False))
-
-bar_table = Table('bar', meta,
-    Column('id', Integer, primary_key=True),
-    Column('data', String(40)))
-
-meta.create_all()
-
-class Foo(object):
-    pass
-
-class Bar(object):
-    def __init__(self, value):
-        self.data = value
-
-    def __eq__(self, other):
-        if not other is None:
-            return self.data == other.data
-        return NotImplemented
-
-
-mapper(Foo, foo_table, extension=MyExt())
-mapper(Bar, bar_table)
-
-sess = create_session()
-f = Foo()
-f.bar = Bar('some bar')
-sess.add(f)
-sess.flush()
-sess.expunge_all()
-
-del MyPickler.sessions.current
-
-f = sess.query(Foo).get(f.id)
-assert f.bar.data == 'some bar'

examples/poly_assoc/__init__.py

@@ -0,0 +1,10 @@
+"""
+Illustrates polymorphic associations, a method of associating a particular child object with many different types of parent object.
+
+This example is based off the original blog post at `<http://techspot.zzzeek.org/?p=13>`_ and illustrates three techniques:
+
+* ``poly_assoc.py`` - imitates the non-foreign-key schema used by Ruby on Rails' Active Record.
+* ``poly_assoc_fk.py`` - Adds a polymorphic association table so that referential integrity can be maintained.
+* ``poly_assoc_generic.py`` - further automates the approach of ``poly_assoc_fk.py`` to also generate the association table definitions automatically.
+
+"""

examples/postgis/__init__.py

@@ -0,0 +1,34 @@
+"""A naive example illustrating techniques to help 
+embed PostGIS functionality.
+
+This example was originally developed in the hopes that it would be extrapolated into a comprehensive PostGIS integration layer.  We are pleased to announce that this has come to fruition as `GeoAlchemy <http://www.geoalchemy.org/>`_.
+
+The example illustrates:
+
+* a DDL extension which allows CREATE/DROP to work in 
+  conjunction with AddGeometryColumn/DropGeometryColumn
+  
+* a Geometry type, as well as a few subtypes, which
+  convert result row values to a GIS-aware object,
+  and also integrates with the DDL extension.
+
+* a GIS-aware object which stores a raw geometry value
+  and provides a factory for functions such as AsText().
+  
+* an ORM comparator which can override standard column
+  methods on mapped objects to produce GIS operators.
+  
+* an attribute event listener that intercepts strings
+  and converts to GeomFromText().
+  
+* a standalone operator example.
+
+The implementation is limited to only public, well known
+and simple to use extension points. 
+
+E.g.::
+
+    print session.query(Road).filter(Road.road_geom.intersects(r1.road_geom)).all()
+
+"""
+

examples/postgis/postgis.py

@@ -1,39 +1,3 @@
-"""A naive example illustrating techniques to help 
-embed PostGIS functionality.
-
-The techniques here could be used by a capable developer 
-as the basis for a comprehensive PostGIS SQLAlchemy extension.
-Please note this is an entirely incomplete proof of concept
-only, and PostGIS support is *not* a supported feature 
-of SQLAlchemy.
-
-Includes:
-
- * a DDL extension which allows CREATE/DROP to work in 
-   conjunction with AddGeometryColumn/DropGeometryColumn
-   
- * a Geometry type, as well as a few subtypes, which
-   convert result row values to a GIS-aware object,
-   and also integrates with the DDL extension.
-
- * a GIS-aware object which stores a raw geometry value
-   and provides a factory for functions such as AsText().
-   
- * an ORM comparator which can override standard column
-   methods on mapped objects to produce GIS operators.
-   
- * an attribute event listener that intercepts strings
-   and converts to GeomFromText().
-   
- * a standalone operator example.
-
-The implementation is limited to only public, well known
-and simple to use extension points. Future SQLAlchemy
-expansion points may allow more seamless integration of
-some features.
-
-"""
-
 from sqlalchemy.orm.interfaces import AttributeExtension
 from sqlalchemy.orm.properties import ColumnProperty
 from sqlalchemy.types import TypeEngine
@@ -101,7 +65,7 @@ class Geometry(TypeEngine):
                 return value
         return process
         
-    def result_processor(self, dialect):
+    def result_processor(self, dialect, coltype):
         def process(value):
             if value is not None:
                 return PersistentGisElement(value)
@@ -231,7 +195,7 @@ if __name__ == '__main__':
     from sqlalchemy.orm import sessionmaker, column_property
     from sqlalchemy.ext.declarative import declarative_base
 
-    engine = create_engine('postgres://scott:tiger@localhost/gistest', echo=True)
+    engine = create_engine('postgresql://scott:tiger@localhost/gistest', echo=True)
     metadata = MetaData(engine)
     Base = declarative_base(metadata=metadata)
 

examples/query_caching/implicit.py

@@ -1,68 +0,0 @@
-"""Example of caching objects in a per-session cache,
-including implicit usage of the statement and params as a key.
-
-"""
-from sqlalchemy.orm.query import Query
-from sqlalchemy.orm.session import Session
-
-class CachingQuery(Query):
-    
-    def __iter__(self):
-        try:
-            cache = self.session._cache
-        except AttributeError:
-            self.session._cache = cache = {}
-            
-        stmt = self.statement.compile()
-        params = stmt.params
-        params.update(self._params)
-        cachekey = str(stmt) + str(params)
-
-        try:
-            ret = cache[cachekey]
-        except KeyError:
-            ret = list(Query.__iter__(self))
-            cache[cachekey] = ret
-
-        return iter(ret)
-
-
-# example usage
-if __name__ == '__main__':
-    from sqlalchemy import Column, create_engine, Integer, String
-    from sqlalchemy.orm import sessionmaker
-    from sqlalchemy.ext.declarative import declarative_base
-    
-    Session = sessionmaker(query_cls=CachingQuery)
-    
-    Base = declarative_base(engine=create_engine('sqlite://', echo=True))
-    
-    class User(Base):
-        __tablename__ = 'users'
-        id = Column(Integer, primary_key=True)
-        name = Column(String(100))
-        
-        def __repr__(self):
-            return "User(name=%r)" % self.name
-
-    Base.metadata.create_all()
-    
-    sess = Session()
-    
-    sess.add_all(
-        [User(name='u1'), User(name='u2'), User(name='u3')]
-    )
-    sess.commit()
-    
-    # issue a query
-    print sess.query(User).filter(User.name.in_(['u2', 'u3'])).all()
-
-    # issue another
-    print sess.query(User).filter(User.name == 'u1').all()
-    
-    # pull straight from cache
-    print sess.query(User).filter(User.name.in_(['u2', 'u3'])).all()
-
-    print sess.query(User).filter(User.name == 'u1').all()
-    
-

examples/query_caching/per_session.py

@@ -1,71 +0,0 @@
-"""Example of caching objects in a per-session cache.
-
-
-This approach is faster in that objects don't need to be detached/remerged
-between sessions, but is slower in that the cache is empty at the start
-of each session's lifespan.
-
-"""
-
-from sqlalchemy.orm.query import Query, _generative
-from sqlalchemy.orm.session import Session
-
-class CachingQuery(Query):
-    
-    # generative method to set a "cache" key.  The method of "keying" the cache
-    # here can be made more sophisticated, such as caching based on the query._criterion.
-    @_generative()
-    def with_cache_key(self, cachekey):
-        self.cachekey = cachekey
-
-    def __iter__(self):
-        if hasattr(self, 'cachekey'):
-            try:
-                cache = self.session._cache
-            except AttributeError:
-                self.session._cache = cache = {}
-                
-            try:
-                ret = cache[self.cachekey]
-            except KeyError:
-                ret = list(Query.__iter__(self))
-                cache[self.cachekey] = ret
-
-            return iter(ret)
-
-        else:
-            return Query.__iter__(self)
-
-# example usage
-if __name__ == '__main__':
-    from sqlalchemy import Column, create_engine, Integer, String
-    from sqlalchemy.orm import sessionmaker
-    from sqlalchemy.ext.declarative import declarative_base
-    
-    Session = sessionmaker(query_cls=CachingQuery)
-    
-    Base = declarative_base(engine=create_engine('sqlite://', echo=True))
-    
-    class User(Base):
-        __tablename__ = 'users'
-        id = Column(Integer, primary_key=True)
-        name = Column(String(100))
-        
-        def __repr__(self):
-            return "User(name=%r)" % self.name
-
-    Base.metadata.create_all()
-    
-    sess = Session()
-    
-    sess.add_all(
-        [User(name='u1'), User(name='u2'), User(name='u3')]
-    )
-    sess.commit()
-    
-    # cache two user objects
-    sess.query(User).with_cache_key('u2andu3').filter(User.name.in_(['u2', 'u3'])).all()
-    
-    # pull straight from cache
-    print sess.query(User).with_cache_key('u2andu3').all()
-    

examples/query_caching/query_caching.py

@@ -1,71 +0,0 @@
-"""Example of caching objects in a global cache."""
-
-from sqlalchemy.orm.query import Query, _generative
-from sqlalchemy.orm.session import Session
-
-# the cache.  This would be replaced with the caching mechanism of
-# choice, i.e. LRU cache, memcached, etc.
-_cache = {}
-
-class CachingQuery(Query):
-    
-    # generative method to set a "cache" key.  The method of "keying" the cache
-    # here can be made more sophisticated, such as caching based on the query._criterion.
-    @_generative()
-    def with_cache_key(self, cachekey):
-        self.cachekey = cachekey
-
-    # single point of object loading is __iter__().  objects in the cache are not associated
-    # with a session and are never returned directly; only merged copies.
-    def __iter__(self):
-        if hasattr(self, 'cachekey'):
-            try:
-                ret = _cache[self.cachekey]
-            except KeyError:
-                ret = list(Query.__iter__(self))
-                for x in ret:
-                    self.session.expunge(x)
-                _cache[self.cachekey] = ret
-
-            return iter(self.session.merge(x, dont_load=True) for x in ret)
-
-        else:
-            return Query.__iter__(self)
-
-# example usage
-if __name__ == '__main__':
-    from sqlalchemy import Column, create_engine, Integer, String
-    from sqlalchemy.orm import sessionmaker
-    from sqlalchemy.ext.declarative import declarative_base
-    
-    Session = sessionmaker(query_cls=CachingQuery)
-    
-    Base = declarative_base(engine=create_engine('sqlite://', echo=True))
-    
-    class User(Base):
-        __tablename__ = 'users'
-        id = Column(Integer, primary_key=True)
-        name = Column(String(100))
-        
-        def __repr__(self):
-            return "User(name=%r)" % self.name
-
-    Base.metadata.create_all()
-    
-    sess = Session()
-    
-    sess.add_all(
-        [User(name='u1'), User(name='u2'), User(name='u3')]
-    )
-    sess.commit()
-    
-    # cache two user objects
-    sess.query(User).with_cache_key('u2andu3').filter(User.name.in_(['u2', 'u3'])).all()
-    
-    sess.close()
-    
-    sess = Session()
-    
-    # pull straight from cache
-    print sess.query(User).with_cache_key('u2andu3').all()
-    

examples/sharding/__init__.py

@@ -0,0 +1,20 @@
+"""a basic example of using the SQLAlchemy Sharding API.
+Sharding refers to horizontally scaling data across multiple
+databases.
+
+The basic components of a "sharded" mapping are:
+
+* multiple databases, each assigned a 'shard id'
+* a function which can return a single shard id, given an instance
+  to be saved; this is called "shard_chooser"
+* a function which can return a list of shard ids which apply to a particular
+  instance identifier; this is called "id_chooser".  If it returns all shard ids,
+  all shards will be searched.
+* a function which can return a list of shard ids to try, given a particular 
+  Query ("query_chooser").  If it returns all shard ids, all shards will be 
+  queried and the results joined together.
+
+In this example, four sqlite databases will store information about
+weather data on a database-per-continent basis.  We provide example shard_chooser, id_chooser and query_chooser functions.  The query_chooser illustrates inspection of the SQL expression element in order to attempt to determine a single shard being requested.
+
+"""

examples/sharding/attribute_shard.py

@@ -1,21 +1,3 @@
-"""a basic example of using the SQLAlchemy Sharding API.
-Sharding refers to horizontally scaling data across multiple
-databases.
-
-In this example, four sqlite databases will store information about
-weather data on a database-per-continent basis.
-
-To set up a sharding system, you need:
-    1. multiple databases, each assined a 'shard id'
-    2. a function which can return a single shard id, given an instance
-    to be saved; this is called "shard_chooser"
-    3. a function which can return a list of shard ids which apply to a particular
-    instance identifier; this is called "id_chooser".  If it returns all shard ids,
-    all shards will be searched.
-    4. a function which can return a list of shard ids to try, given a particular 
-    Query ("query_chooser").  If it returns all shard ids, all shards will be 
-    queried and the results joined together.
-"""
 
 # step 1. imports
 from sqlalchemy import (create_engine, MetaData, Table, Column, Integer,
@@ -134,7 +116,13 @@ def query_chooser(query):
     # and convert to shard ids
     class FindContinent(sql.ClauseVisitor):
         def visit_binary(self, binary):
-            if binary.left is weather_locations.c.continent:
+            # "shares_lineage()" returns True if both columns refer to the same
+            # statement column, adjusting for any annotations present.
+            # (an annotation is an internal clone of a Column object
+            # and occur when using ORM-mapped attributes like 
+            # "WeatherLocation.continent"). A simpler comparison, though less accurate, 
+            # would be "binary.left.key == 'continent'".
+            if binary.left.shares_lineage(weather_locations.c.continent):
                 if binary.operator == operators.eq:
                     ids.append(shard_lookup[binary.right.value])
                 elif binary.operator == operators.in_op:

examples/versioning/__init__.py

@@ -0,0 +1,67 @@
+"""
+Illustrates an extension which creates version tables for entities and stores records for each change.  The same idea as Elixir's versioned extension, but more efficient (uses attribute API to get history) and handles class inheritance.  The given extensions generate an anonymous "history" class which represents historical versions of the target object.   
+
+Usage is illustrated via a unit test module ``test_versioning.py``, which can be run via nose::
+
+    nosetests -w examples/versioning/
+
+A fragment of example usage, using declarative::
+
+    from history_meta import VersionedMeta, VersionedListener
+
+    Base = declarative_base(metaclass=VersionedMeta, bind=engine)
+    Session = sessionmaker(extension=VersionedListener())
+
+    class SomeClass(Base):
+        __tablename__ = 'sometable'
+    
+        id = Column(Integer, primary_key=True)
+        name = Column(String(50))
+    
+        def __eq__(self, other):
+            assert type(other) is SomeClass and other.id == self.id
+        
+    sess = Session()
+    sc = SomeClass(name='sc1')
+    sess.add(sc)
+    sess.commit()
+
+    sc.name = 'sc1modified'
+    sess.commit()
+
+    assert sc.version == 2
+
+    SomeClassHistory = SomeClass.__history_mapper__.class_
+
+    assert sess.query(SomeClassHistory).\\
+                filter(SomeClassHistory.version == 1).\\
+                all() \\
+                == [SomeClassHistory(version=1, name='sc1')]
+
+To apply ``VersionedMeta`` to a subset of classes (probably more typical), the metaclass can be applied on a per-class basis::
+
+    from history_meta import VersionedMeta, VersionedListener
+
+    Base = declarative_base(bind=engine)
+
+    class SomeClass(Base):
+        __tablename__ = 'sometable'
+
+        # ...
+
+    class SomeVersionedClass(Base):
+        __metaclass__ = VersionedMeta
+        __tablename__ = 'someothertable'
+
+        # ...
+
+The ``VersionedMeta`` is a declarative metaclass - to use the extension with plain mappers, the ``_history_mapper`` function can be applied::
+
+    from history_meta import _history_mapper
+
+    m = mapper(SomeClass, sometable)
+    _history_mapper(m)
+
+    SomeHistoryClass = SomeClass.__history_mapper__.class_
+
+"""

examples/versioning/history_meta.py

@@ -0,0 +1,164 @@
+from sqlalchemy.ext.declarative import DeclarativeMeta
+from sqlalchemy.orm import mapper, class_mapper, attributes, object_mapper
+from sqlalchemy.orm.exc import UnmappedClassError, UnmappedColumnError
+from sqlalchemy import Table, Column, ForeignKeyConstraint, Integer
+from sqlalchemy.orm.interfaces import SessionExtension
+
+def col_references_table(col, table):
+    for fk in col.foreign_keys:
+        if fk.references(table):
+            return True
+    return False
+
+def _history_mapper(local_mapper):
+    cls = local_mapper.class_
+
+    # SLIGHT SQLA HACK #1 - set the "active_history" flag
+    # on on column-mapped attributes so that the old version
+    # of the info is always loaded (currently sets it on all attributes)
+    for prop in local_mapper.iterate_properties:
+        getattr(local_mapper.class_, prop.key).impl.active_history = True
+    
+    super_mapper = local_mapper.inherits
+    super_history_mapper = getattr(cls, '__history_mapper__', None)
+    
+    polymorphic_on = None
+    super_fks = []
+    if not super_mapper or local_mapper.local_table is not super_mapper.local_table:
+        cols = []
+        for column in local_mapper.local_table.c:
+            if column.name == 'version':
+                continue
+                
+            col = column.copy()
+
+            if super_mapper and col_references_table(column, super_mapper.local_table):
+                super_fks.append((col.key, list(super_history_mapper.base_mapper.local_table.primary_key)[0]))
+
+            cols.append(col)
+            
+            if column is local_mapper.polymorphic_on:
+                polymorphic_on = col
+        
+        if super_mapper:
+            super_fks.append(('version', super_history_mapper.base_mapper.local_table.c.version))
+            cols.append(Column('version', Integer, primary_key=True))
+        else:
+            cols.append(Column('version', Integer, primary_key=True))
+        
+        if super_fks:
+            cols.append(ForeignKeyConstraint(*zip(*super_fks)))
+
+        table = Table(local_mapper.local_table.name + '_history', local_mapper.local_table.metadata,
+           *cols
+        )
+    else:
+        # single table inheritance.  take any additional columns that may have
+        # been added and add them to the history table.
+        for column in local_mapper.local_table.c:
+            if column.key not in super_history_mapper.local_table.c:
+                col = column.copy()
+                super_history_mapper.local_table.append_column(col)
+        table = None
+        
+    if super_history_mapper:
+        bases = (super_history_mapper.class_,)
+    else:
+        bases = local_mapper.base_mapper.class_.__bases__
+    versioned_cls = type.__new__(type, "%sHistory" % cls.__name__, bases, {})
+    
+    m = mapper(
+            versioned_cls, 
+            table, 
+            inherits=super_history_mapper, 
+            polymorphic_on=polymorphic_on,
+            polymorphic_identity=local_mapper.polymorphic_identity
+            )
+    cls.__history_mapper__ = m
+    
+    if not super_history_mapper:
+        cls.version = Column('version', Integer, default=1, nullable=False)
+    
+    
+class VersionedMeta(DeclarativeMeta):
+    def __init__(cls, classname, bases, dict_):
+        DeclarativeMeta.__init__(cls, classname, bases, dict_)
+
+        try:
+            mapper = class_mapper(cls)
+            _history_mapper(mapper)
+        except UnmappedClassError:
+            pass
+
+
+def versioned_objects(iter):
+    for obj in iter:
+        if hasattr(obj, '__history_mapper__'):
+            yield obj
+
+def create_version(obj, session, deleted = False):
+    obj_mapper = object_mapper(obj)
+    history_mapper = obj.__history_mapper__
+    history_cls = history_mapper.class_
+    
+    obj_state = attributes.instance_state(obj)
+    
+    attr = {}
+
+    obj_changed = False
+    
+    for om, hm in zip(obj_mapper.iterate_to_root(), history_mapper.iterate_to_root()):
+        if hm.single:
+            continue
+    
+        for hist_col in hm.local_table.c:
+            if hist_col.key == 'version':
+                continue
+                
+            obj_col = om.local_table.c[hist_col.key]
+
+            # SLIGHT SQLA HACK #3 - get the value of the
+            # attribute based on the MapperProperty related to the
+            # mapped column.  this will allow usage of MapperProperties
+            # that have a different keyname than that of the mapped column.
+            try:
+                prop = obj_mapper._get_col_to_prop(obj_col)
+            except UnmappedColumnError:
+                # in the case of single table inheritance, there may be 
+                # columns on the mapped table intended for the subclass only.
+                # the "unmapped" status of the subclass column on the 
+                # base class is a feature of the declarative module as of sqla 0.5.2.
+                continue
+                
+            # expired object attributes and also deferred cols might not be in the
+            # dict.  force it to load no matter what by using getattr().
+            if prop.key not in obj_state.dict:
+                getattr(obj, prop.key)
+
+            a, u, d = attributes.get_history(obj, prop.key)
+
+            if d:
+                attr[hist_col.key] = d[0]
+                obj_changed = True
+            elif u:
+                attr[hist_col.key] = u[0]
+            else:
+                raise Exception("TODO: what makes us arrive here ?")
+                
+    if not obj_changed and not deleted:            
+        return
+
+    attr['version'] = obj.version
+    hist = history_cls()
+    for key, value in attr.iteritems():
+        setattr(hist, key, value)
+    session.add(hist)
+    obj.version += 1
+    
+class VersionedListener(SessionExtension):
+    def before_flush(self, session, flush_context, instances):
+        for obj in versioned_objects(session.dirty):
+            create_version(obj, session)
+        for obj in versioned_objects(session.deleted):
+            create_version(obj, session, deleted = True)
+            

examples/versioning/test_versioning.py

@@ -0,0 +1,248 @@
+from sqlalchemy.ext.declarative import declarative_base
+from history_meta import VersionedMeta, VersionedListener
+from sqlalchemy import create_engine, Column, Integer, String, ForeignKey
+from sqlalchemy.orm import clear_mappers, compile_mappers, sessionmaker, deferred
+from sqlalchemy.test.testing import TestBase, eq_
+from sqlalchemy.test.entities import ComparableEntity
+
+def setup():
+    global engine
+    engine = create_engine('sqlite://', echo=True)
+    
+class TestVersioning(TestBase):
+    def setup(self):
+        global Base, Session
+        Base = declarative_base(metaclass=VersionedMeta, bind=engine)
+        Session = sessionmaker(extension=VersionedListener())
+        
+    def teardown(self):
+        clear_mappers()
+        Base.metadata.drop_all()
+    
+    def create_tables(self):
+        Base.metadata.create_all()
+        
+    def test_plain(self):
+        class SomeClass(Base, ComparableEntity):
+            __tablename__ = 'sometable'
+            
+            id = Column(Integer, primary_key=True)
+            name = Column(String(50))
+        
+        self.create_tables()
+        sess = Session()
+        sc = SomeClass(name='sc1')
+        sess.add(sc)
+        sess.commit()
+        
+        sc.name = 'sc1modified'
+        sess.commit()
+        
+        assert sc.version == 2
+        
+        SomeClassHistory = SomeClass.__history_mapper__.class_
+        
+        eq_(
+            sess.query(SomeClassHistory).filter(SomeClassHistory.version == 1).all(),
+            [SomeClassHistory(version=1, name='sc1')]
+        )
+
+        sc.name = 'sc1modified2'
+
+        eq_(
+            sess.query(SomeClassHistory).order_by(SomeClassHistory.version).all(),
+            [
+                SomeClassHistory(version=1, name='sc1'),
+                SomeClassHistory(version=2, name='sc1modified')
+            ]
+        )
+
+        assert sc.version == 3
+        
+        sess.commit()
+
+        sc.name = 'temp'
+        sc.name = 'sc1modified2'
+
+        sess.commit()
+
+        eq_(
+            sess.query(SomeClassHistory).order_by(SomeClassHistory.version).all(),
+            [
+                SomeClassHistory(version=1, name='sc1'),
+                SomeClassHistory(version=2, name='sc1modified')
+            ]
+        )
+        
+        sess.delete(sc)
+        sess.commit()
+
+        eq_(
+            sess.query(SomeClassHistory).order_by(SomeClassHistory.version).all(),
+            [
+                SomeClassHistory(version=1, name='sc1'),
+                SomeClassHistory(version=2, name='sc1modified'),
+                SomeClassHistory(version=3, name='sc1modified2')
+            ]
+        )
+
+
+
+
+    def test_deferred(self):
+        """test versioning of unloaded, deferred columns."""
+        
+        class SomeClass(Base, ComparableEntity):
+            __tablename__ = 'sometable'
+
+            id = Column(Integer, primary_key=True)
+            name = Column(String(50))
+            data = deferred(Column(String(25)))
+            
+        self.create_tables()
+        sess = Session()
+        sc = SomeClass(name='sc1', data='somedata')
+        sess.add(sc)
+        sess.commit()
+        sess.close()
+        
+        sc = sess.query(SomeClass).first()
+        assert 'data' not in sc.__dict__
+        
+        sc.name = 'sc1modified'
+        sess.commit()
+
+        assert sc.version == 2
+
+        SomeClassHistory = SomeClass.__history_mapper__.class_
+
+        eq_(
+            sess.query(SomeClassHistory).filter(SomeClassHistory.version == 1).all(),
+            [SomeClassHistory(version=1, name='sc1', data='somedata')]
+        )
+        
+        
+    def test_joined_inheritance(self):
+        class BaseClass(Base, ComparableEntity):
+            __tablename__ = 'basetable'
+
+            id = Column(Integer, primary_key=True)
+            name = Column(String(50))
+            type = Column(String(20))
+            
+            __mapper_args__ = {'polymorphic_on':type, 'polymorphic_identity':'base'}
+            
+        class SubClassSeparatePk(BaseClass):
+            __tablename__ = 'subtable1'
+
+            id = Column(Integer, primary_key=True)
+            base_id = Column(Integer, ForeignKey('basetable.id'))
+            subdata1 = Column(String(50))
+
+            __mapper_args__ = {'polymorphic_identity':'sep'}
+
+        class SubClassSamePk(BaseClass):
+            __tablename__ = 'subtable2'
+
+            id = Column(Integer, ForeignKey('basetable.id'), primary_key=True)
+            subdata2 = Column(String(50))
+
+            __mapper_args__ = {'polymorphic_identity':'same'}
+
+        self.create_tables()
+        sess = Session()
+
+        sep1 = SubClassSeparatePk(name='sep1', subdata1='sep1subdata')
+        base1 = BaseClass(name='base1')
+        same1 = SubClassSamePk(name='same1', subdata2='same1subdata')
+        sess.add_all([sep1, base1, same1])
+        sess.commit()
+        
+        base1.name = 'base1mod'
+        same1.subdata2 = 'same1subdatamod'
+        sep1.name ='sep1mod'
+        sess.commit()
+
+        BaseClassHistory = BaseClass.__history_mapper__.class_
+        SubClassSeparatePkHistory = SubClassSeparatePk.__history_mapper__.class_
+        SubClassSamePkHistory = SubClassSamePk.__history_mapper__.class_
+        eq_(
+            sess.query(BaseClassHistory).order_by(BaseClassHistory.id).all(),
+            [
+                SubClassSeparatePkHistory(id=1, name=u'sep1', type=u'sep', version=1), 
+                BaseClassHistory(id=2, name=u'base1', type=u'base', version=1), 
+                SubClassSamePkHistory(id=3, name=u'same1', type=u'same', version=1)            
+            ]
+        )
+        
+        same1.subdata2 = 'same1subdatamod2'
+
+        eq_(
+            sess.query(BaseClassHistory).order_by(BaseClassHistory.id, BaseClassHistory.version).all(),
+            [
+                SubClassSeparatePkHistory(id=1, name=u'sep1', type=u'sep', version=1), 
+                BaseClassHistory(id=2, name=u'base1', type=u'base', version=1), 
+                SubClassSamePkHistory(id=3, name=u'same1', type=u'same', version=1), 
+                SubClassSamePkHistory(id=3, name=u'same1', type=u'same', version=2)            
+            ]
+        )
+
+        base1.name = 'base1mod2'
+        eq_(
+            sess.query(BaseClassHistory).order_by(BaseClassHistory.id, BaseClassHistory.version).all(),
+            [
+                SubClassSeparatePkHistory(id=1, name=u'sep1', type=u'sep', version=1), 
+                BaseClassHistory(id=2, name=u'base1', type=u'base', version=1), 
+                BaseClassHistory(id=2, name=u'base1mod', type=u'base', version=2), 
+                SubClassSamePkHistory(id=3, name=u'same1', type=u'same', version=1), 
+                SubClassSamePkHistory(id=3, name=u'same1', type=u'same', version=2)            
+            ]
+        )
+
+    def test_single_inheritance(self):
+        class BaseClass(Base, ComparableEntity):
+            __tablename__ = 'basetable'
+
+            id = Column(Integer, primary_key=True)
+            name = Column(String(50))
+            type = Column(String(50))
+            __mapper_args__ = {'polymorphic_on':type, 'polymorphic_identity':'base'}
+            
+        class SubClass(BaseClass):
+
+            subname = Column(String(50))
+            __mapper_args__ = {'polymorphic_identity':'sub'}
+
+        self.create_tables()
+        sess = Session()
+
+        b1 = BaseClass(name='b1')
+        sc = SubClass(name='s1', subname='sc1')
+        
+        sess.add_all([b1, sc])
+        
+        sess.commit()
+        
+        b1.name='b1modified'
+
+        BaseClassHistory = BaseClass.__history_mapper__.class_
+        SubClassHistory = SubClass.__history_mapper__.class_
+        
+        eq_(
+            sess.query(BaseClassHistory).order_by(BaseClassHistory.id, BaseClassHistory.version).all(),
+            [BaseClassHistory(id=1, name=u'b1', type=u'base', version=1)]
+        )
+        
+        sc.name ='s1modified'
+        b1.name='b1modified2'
+
+        eq_(
+            sess.query(BaseClassHistory).order_by(BaseClassHistory.id, BaseClassHistory.version).all(),
+            [
+                BaseClassHistory(id=1, name=u'b1', type=u'base', version=1),
+                BaseClassHistory(id=1, name=u'b1modified', type=u'base', version=2),
+                SubClassHistory(id=2, name=u's1', type=u'sub', version=1)
+            ]
+        )
+        
+

examples/vertical/__init__.py

@@ -0,0 +1,27 @@
+"""
+Illustrates "vertical table" mappings.
+
+A "vertical table" refers to a technique where individual attributes of an object are stored as distinct rows in a table.   
+The "vertical table" technique is used to persist objects which can have a varied set of attributes, at the expense of simple query control and brevity.   It is commonly found in content/document management systems in order to represent user-created structures flexibly.
+
+Two variants on the approach are given.  In the second, each row references a "datatype" which contains information about the type of information stored in the attribute, such as integer, string, or date.
+
+
+Example::
+
+    shrew = Animal(u'shrew')
+    shrew[u'cuteness'] = 5
+    shrew[u'weasel-like'] = False
+    shrew[u'poisonous'] = True
+
+    session.add(shrew)
+    session.flush()
+
+    q = (session.query(Animal).
+         filter(Animal.facts.any(
+           and_(AnimalFact.key == u'weasel-like',
+                AnimalFact.value == True))))
+    print 'weasel-like animals', q.all()
+
+
+"""

examples/vertical/dictlike-polymorphic.py

@@ -25,8 +25,6 @@ we'll use a Python @property to build a smart '.value' attribute that wraps up
 reading and writing those various '_value' columns and keeps the '.type' up to
 date.
 
-Note: Something much like 'comparable_property' is slated for inclusion in a
-      future version of SQLAlchemy.
 """
 
 from sqlalchemy.orm.interfaces import PropComparator

examples/vertical/dictlike.py

@@ -28,6 +28,7 @@ entity, and another related table holding key/value pairs::
 Because the key/value pairs in a vertical scheme are not fixed in advance,
 accessing them like a Python dict can be very convenient.  The example below
 can be used with many common vertical schemas as-is or with minor adaptations.
+
 """
 
 class VerticalProperty(object):

examples/vertical/vertical.py

@@ -1,201 +0,0 @@
-"""this example illustrates a "vertical table".  an object is stored with individual attributes 
-represented in distinct database rows.  This allows objects to be created with dynamically changing
-fields that are all persisted in a normalized fashion."""
-
-from sqlalchemy import (create_engine, MetaData, Table, Column, Integer, String,
-    ForeignKey, PickleType, DateTime, and_)
-from sqlalchemy.orm import mapper, relation, sessionmaker, scoped_session
-from sqlalchemy.orm.collections import mapped_collection
-import datetime
-
-engine = create_engine('sqlite://', echo=False)
-meta = MetaData(engine)
-
-Session = scoped_session(sessionmaker())
-
-# represent Entity objects
-entities = Table('entities', meta, 
-    Column('entity_id', Integer, primary_key=True),
-    Column('title', String(100), nullable=False),
-    )
-
-# represent named, typed fields
-entity_fields = Table('entity_fields', meta,
-    Column('field_id', Integer, primary_key=True),
-    Column('name', String(40), nullable=False),
-    Column('datatype', String(30), nullable=False))
-    
-# associate a field row with an entity row, including a typed value
-entity_values = Table('entity_values', meta, 
-    Column('value_id', Integer, primary_key=True),
-    Column('field_id', Integer, ForeignKey('entity_fields.field_id'), nullable=False),
-    Column('entity_id', Integer, ForeignKey('entities.entity_id'), nullable=False),
-    Column('int_value', Integer), 
-    Column('string_value', String(500)),
-    Column('binary_value', PickleType),
-    Column('datetime_value', DateTime))
-
-meta.create_all()
-
-class Entity(object):
-    """a persistable dynamic object.  
-    
-    Marshalls attributes into a dictionary which is 
-    mapped to the database.
-    
-    """
-    def __init__(self, **kwargs):
-        for k in kwargs:
-            setattr(self, k, kwargs[k])
-            
-    def __getattr__(self, key):
-        """Proxy requests for attributes to the underlying _entities dictionary."""
-
-        if key[0] == '_':
-            return super(Entity, self).__getattr__(key)
-        try:
-            return self._entities[key].value
-        except KeyError:
-            raise AttributeError(key)
-
-    def __setattr__(self, key, value):
-        """Proxy requests for attribute set operations to the underlying _entities dictionary."""
-
-        if key[0] == "_" or hasattr(Entity, key):
-            object.__setattr__(self, key, value)
-            return
-            
-        try:
-            ev = self._entities[key]
-            ev.value = value
-        except KeyError:
-            ev = _EntityValue(key, value)
-            self._entities[key] = ev
-        
-class _EntityField(object):
-    """Represents a field of a particular name and datatype."""
-
-    def __init__(self, name, datatype):
-        self.name = name
-        self.datatype = datatype
-
-class _EntityValue(object):
-    """Represents an individual value."""
-
-    def __init__(self, key, value):
-        datatype = self._figure_datatype(value)
-        field = \
-            Session.query(_EntityField).filter(
-                and_(_EntityField.name==key, _EntityField.datatype==datatype)
-            ).first()
-
-        if not field:
-            field = _EntityField(key, datatype)
-            Session.add(field)
-        
-        self.field = field
-        setattr(self, self.field.datatype + "_value", value)
-    
-    def _figure_datatype(self, value):
-        typemap = {
-            int:'int',
-            str:'string',
-            datetime.datetime:'datetime',
-        }
-        for k in typemap:
-            if isinstance(value, k):
-                return typemap[k]
-        else:
-            return 'binary'
-
-    def _get_value(self):
-        return getattr(self, self.field.datatype + "_value")
-
-    def _set_value(self, value):
-        setattr(self, self.field.datatype + "_value", value)
-    value = property(_get_value, _set_value)
-    
-    def name(self):
-        return self.field.name
-    name = property(name)
-
-
-# the mappers are a straightforward eager chain of 
-# Entity--(1->many)->EntityValue-(many->1)->EntityField
-# notice that we are identifying each mapper to its connecting
-# relation by just the class itself.
-mapper(_EntityField, entity_fields)
-mapper(
-    _EntityValue, entity_values,
-    properties = {
-        'field' : relation(_EntityField, lazy=False, cascade='all')
-    }
-)
-
-mapper(Entity, entities, properties = {
-    '_entities' : relation(
-                        _EntityValue, 
-                        lazy=False, 
-                        cascade='all', 
-                        collection_class=mapped_collection(lambda entityvalue: entityvalue.field.name)
-                    )
-})
-
-session = Session()
-entity1 = Entity(
-    title = 'this is the first entity',
-    name = 'this is the name',
-    price = 43,
-    data = ('hello', 'there')
-)
-
-entity2 = Entity(
-    title = 'this is the second entity',
-    name = 'this is another name',
-    price = 50,
-    data = ('hoo', 'ha')
-)
-
-session.add_all([entity1, entity2])
-session.commit()
-
-for entity in session.query(Entity):
-    print "Entity id %d:" % entity.entity_id, entity.title, entity.name, entity.price, entity.data
-
-# perform some changes, add a new Entity
-
-entity1.price = 90
-entity1.title = 'another new title'
-entity2.data = {'oof':5,'lala':8}
-
-entity3 = Entity(
-    title = 'third entity',
-    name = 'new name',
-    price = '$1.95',    # note we change 'price' to be a string.
-                        # this creates a new _EntityField separate from the
-                        # one used by integer 'price'.
-    data = 'some data'
-)
-session.add(entity3)
-
-session.commit()
-
-print "----------------"
-for entity in session.query(Entity):
-    print "Entity id %d:" % entity.entity_id, entity.title, entity.name, entity.price, entity.data
-
-print "----------------"
-# illustrate each _EntityField that's been created and list each Entity which uses it
-for ent_id, name, datatype in session.query(_EntityField.field_id, _EntityField.name, _EntityField.datatype):
-    print name, datatype, "(Enitites:",  ",".join([
-        str(entid) for entid in session.query(Entity.entity_id).\
-            join(
-                (_EntityValue, _EntityValue.entity_id==Entity.entity_id), 
-                (_EntityField, _EntityField.field_id==_EntityValue.field_id)
-            ).filter(_EntityField.field_id==ent_id)
-    ]), ")"
-
-# delete all the Entity objects
-for entity in session.query(Entity):
-    session.delete(entity)
-session.commit()

ez_setup.py

@@ -0,0 +1,276 @@
+#!python
+"""Bootstrap setuptools installation
+
+If you want to use setuptools in your package's setup.py, just include this
+file in the same directory with it, and add this to the top of your setup.py::
+
+    from ez_setup import use_setuptools
+    use_setuptools()
+
+If you want to require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, you can do so by supplying
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import sys
+DEFAULT_VERSION = "0.6c9"
+DEFAULT_URL     = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
+
+md5_data = {
+    'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca',
+    'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb',
+    'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b',
+    'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a',
+    'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618',
+    'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac',
+    'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5',
+    'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4',
+    'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c',
+    'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b',
+    'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27',
+    'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277',
+    'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa',
+    'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e',
+    'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e',
+    'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f',
+    'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2',
+    'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc',
+    'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167',
+    'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64',
+    'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d',
+    'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20',
+    'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab',
+    'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
+    'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2',
+    'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e',
+    'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372',
+    'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902',
+    'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de',
+    'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b',
+    'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03',
+    'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a',
+    'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6',
+    'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a',
+}
+
+import sys, os
+try: from hashlib import md5
+except ImportError: from md5 import md5
+
+def _validate_md5(egg_name, data):
+    if egg_name in md5_data:
+        digest = md5(data).hexdigest()
+        if digest != md5_data[egg_name]:
+            print >>sys.stderr, (
+                "md5 validation of %s failed!  (Possible download problem?)"
+                % egg_name
+            )
+            sys.exit(2)
+    return data
+
+def use_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    download_delay=15
+):
+    """Automatically find/download setuptools and make it available on sys.path
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end with
+    a '/').  `to_dir` is the directory where setuptools will be downloaded, if
+    it is not already available.  If `download_delay` is specified, it should
+    be the number of seconds that will be paused before initiating a download,
+    should one be required.  If an older version of setuptools is installed,
+    this routine will print a message to ``sys.stderr`` and raise SystemExit in
+    an attempt to abort the calling script.
+    """
+    was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules
+    def do_download():
+        egg = download_setuptools(version, download_base, to_dir, download_delay)
+        sys.path.insert(0, egg)
+        import setuptools; setuptools.bootstrap_install_from = egg
+    try:
+        import pkg_resources
+    except ImportError:
+        return do_download()       
+    try:
+        pkg_resources.require("setuptools>="+version); return
+    except pkg_resources.VersionConflict, e:
+        if was_imported:
+            print >>sys.stderr, (
+            "The required version of setuptools (>=%s) is not available, and\n"
+            "can't be installed while this script is running. Please install\n"
+            " a more recent version first, using 'easy_install -U setuptools'."
+            "\n\n(Currently using %r)"
+            ) % (version, e.args[0])
+            sys.exit(2)
+        else:
+            del pkg_resources, sys.modules['pkg_resources']    # reload ok
+            return do_download()
+    except pkg_resources.DistributionNotFound:
+        return do_download()
+
+def download_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    delay = 15
+):
+    """Download setuptools from a specified location and return its filename
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end
+    with a '/'). `to_dir` is the directory where the egg will be downloaded.
+    `delay` is the number of seconds to pause before an actual download attempt.
+    """
+    import urllib2, shutil
+    egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3])
+    url = download_base + egg_name
+    saveto = os.path.join(to_dir, egg_name)
+    src = dst = None
+    if not os.path.exists(saveto):  # Avoid repeated downloads
+        try:
+            from distutils import log
+            if delay:
+                log.warn("""
+---------------------------------------------------------------------------
+This script requires setuptools version %s to run (even to display
+help).  I will attempt to download it for you (from
+%s), but
+you may need to enable firewall access for this script first.
+I will start the download in %d seconds.
+
+(Note: if this machine does not have network access, please obtain the file
+
+   %s
+
+and place it in this directory before rerunning this script.)
+---------------------------------------------------------------------------""",
+                    version, download_base, delay, url
+                ); from time import sleep; sleep(delay)
+            log.warn("Downloading %s", url)
+            src = urllib2.urlopen(url)
+            # Read/write all in one block, so we don't create a corrupt file
+            # if the download is interrupted.
+            data = _validate_md5(egg_name, src.read())
+            dst = open(saveto,"wb"); dst.write(data)
+        finally:
+            if src: src.close()
+            if dst: dst.close()
+    return os.path.realpath(saveto)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+def main(argv, version=DEFAULT_VERSION):
+    """Install or upgrade setuptools and EasyInstall"""
+    try:
+        import setuptools
+    except ImportError:
+        egg = None
+        try:
+            egg = download_setuptools(version, delay=0)
+            sys.path.insert(0,egg)
+            from setuptools.command.easy_install import main
+            return main(list(argv)+[egg])   # we're done here
+        finally:
+            if egg and os.path.exists(egg):
+                os.unlink(egg)
+    else:
+        if setuptools.__version__ == '0.0.1':
+            print >>sys.stderr, (
+            "You have an obsolete version of setuptools installed.  Please\n"
+            "remove it from your system entirely before rerunning this script."
+            )
+            sys.exit(2)
+
+    req = "setuptools>="+version
+    import pkg_resources
+    try:
+        pkg_resources.require(req)
+    except pkg_resources.VersionConflict:
+        try:
+            from setuptools.command.easy_install import main
+        except ImportError:
+            from easy_install import main
+        main(list(argv)+[download_setuptools(delay=0)])
+        sys.exit(0) # try to force an exit
+    else:
+        if argv:
+            from setuptools.command.easy_install import main
+            main(argv)
+        else:
+            print "Setuptools version",version,"or greater has been installed."
+            print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)'
+
+def update_md5(filenames):
+    """Update our built-in md5 registry"""
+
+    import re
+
+    for name in filenames:
+        base = os.path.basename(name)
+        f = open(name,'rb')
+        md5_data[base] = md5(f.read()).hexdigest()
+        f.close()
+
+    data = ["    %r: %r,\n" % it for it in md5_data.items()]
+    data.sort()
+    repl = "".join(data)
+
+    import inspect
+    srcfile = inspect.getsourcefile(sys.modules[__name__])
+    f = open(srcfile, 'rb'); src = f.read(); f.close()
+
+    match = re.search("\nmd5_data = {\n([^}]+)}", src)
+    if not match:
+        print >>sys.stderr, "Internal error!"
+        sys.exit(2)
+
+    src = src[:match.start(1)] + repl + src[match.end(1):]
+    f = open(srcfile,'w')
+    f.write(src)
+    f.close()
+
+
+if __name__=='__main__':
+    if len(sys.argv)>2 and sys.argv[1]=='--md5update':
+        update_md5(sys.argv[2:])
+    else:
+        main(sys.argv[1:])
+
+
+
+
+
+

lib/sqlalchemy/__init__.py

@@ -1,5 +1,5 @@
 # __init__.py
-# Copyright (C) 2005, 2006, 2007, 2008, 2009 Michael Bayer mike_mp@zzzcomputing.com
+# Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010 Michael Bayer mike_mp@zzzcomputing.com
 #
 # This module is part of SQLAlchemy and is released under
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
@@ -10,40 +10,6 @@ import sys
 import sqlalchemy.exc as exceptions
 sys.modules['sqlalchemy.exceptions'] = exceptions
 
-from sqlalchemy.types import (
-    BLOB,
-    BOOLEAN,
-    Binary,
-    Boolean,
-    CHAR,
-    CLOB,
-    DATE,
-    DATETIME,
-    DECIMAL,
-    Date,
-    DateTime,
-    FLOAT,
-    Float,
-    INT,
-    Integer,
-    Interval,
-    NCHAR,
-    NUMERIC,
-    Numeric,
-    PickleType,
-    SMALLINT,
-    SmallInteger,
-    String,
-    TEXT,
-    TIME,
-    TIMESTAMP,
-    Text,
-    Time,
-    Unicode,
-    UnicodeText,
-    VARCHAR,
-    )
-
 from sqlalchemy.sql import (
     alias,
     and_,
@@ -76,11 +42,52 @@ from sqlalchemy.sql import (
     select,
     subquery,
     text,
+    tuple_,
     union,
     union_all,
     update,
     )
 
+from sqlalchemy.types import (
+    BLOB,
+    BOOLEAN,
+    BigInteger,
+    Binary,
+    Boolean,
+    CHAR,
+    CLOB,
+    DATE,
+    DATETIME,
+    DECIMAL,
+    Date,
+    DateTime,
+    Enum,
+    FLOAT,
+    Float,
+    INT,
+    INTEGER,
+    Integer,
+    Interval,
+    LargeBinary,
+    NCHAR,
+    NVARCHAR,
+    NUMERIC,
+    Numeric,
+    PickleType,
+    SMALLINT,
+    SmallInteger,
+    String,
+    TEXT,
+    TIME,
+    TIMESTAMP,
+    Text,
+    Time,
+    Unicode,
+    UnicodeText,
+    VARCHAR,
+    )
+
+
 from sqlalchemy.schema import (
     CheckConstraint,
     Column,
@@ -107,6 +114,6 @@ from sqlalchemy.engine import create_engine, engine_from_config
 __all__ = sorted(name for name, obj in locals().items()
                  if not (name.startswith('_') or inspect.ismodule(obj)))
                  
-__version__ = '0.5.5'
+__version__ = '0.6beta1'
 
 del inspect, sys

lib/sqlalchemy/connectors/__init__.py

@@ -0,0 +1,6 @@
+
+
+class Connector(object):
+    pass
+    
+    

lib/sqlalchemy/connectors/mxodbc.py

@@ -0,0 +1,24 @@
+from sqlalchemy.connectors import Connector
+
+class MxODBCConnector(Connector):
+    driver='mxodbc'
+    supports_sane_rowcount = False
+    supports_sane_multi_rowcount = False
+    supports_unicode_statements = False
+    supports_unicode_binds = False
+
+    @classmethod
+    def import_dbapi(cls):
+        import mxODBC as module
+        return module
+
+    def create_connect_args(self, url):
+        '''Return a tuple of *args,**kwargs'''
+        # FIXME: handle mx.odbc.Windows proprietary args
+        opts = url.translate_connect_args(username='user')
+        opts.update(url.query)
+        argsDict = {}
+        argsDict['user'] = opts['user']
+        argsDict['password'] = opts['password']
+        connArgs = [[opts['dsn']], argsDict]
+        return connArgs