8000 Implement new ClauseElement role and coercion system · sqlalchemy/sqlalchemy@f07e050 · GitHub
[go: up one dir, main page]
Skip to content

Commit f07e050

Browse files
zzzeekzzzeek
committed
Implement new ClauseElement role and coercion system
A major refactoring of all the functions handle all detection of Core argument types as well as perform coercions into a new class hierarchy based on "roles", each of which identify a syntactical location within a SQL statement. In contrast to the ClauseElement hierarchy that identifies "what" each object is syntactically, the SQLRole hierarchy identifies the "where does it go" of each object syntactically. From this we define a consistent type checking and coercion system that establishes well defined behviors. This is a breakout of the patch that is reorganizing select() constructs to no longer be in the FromClause hierarchy. Also includes a rename of as_scalar() into scalar_subquery(); deprecates automatic coercion to scalar_subquery(). Partially-fixes: #4617 Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
1 parent 614dfb5 commit f07e050

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

85 files changed

+2216
-1179
lines changed

‎doc/build/changelog/unreleased_14/4617_coercion.rst

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
.. change::
2+
:tags: sql, change
3+
:tickets: 4617
4+
5+
The "clause coercion" system, which is SQLAlchemy Core's system of receiving
6+
arguments and resolving them into :class:`.ClauseElement` structures in order
7+
to build up SQL expression objects, has been rewritten from a series of
8+
ad-hoc functions to a fully consistent class-based system. This change
9+
is internal and should have no impact on end users other than more specific
10+
error messages when the wrong kind of argument is passed to an expression
11+
object, however the change is part of a larger set of changes involving
12+
the role and behavior of :func:`.select` objects.
13+

‎doc/build/changelog/unreleased_14/4617_scalar.rst

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
.. change::
2+
:tags: change, sql
3+
:tickets: 4617
4+
5+
The :meth:`.SelectBase.as_scalar` and :meth:`.Query.as_scalar` methods have
6+
been renamed to :meth:`.SelectBase.scalar_subquery` and :meth:`.Query.scalar_subquery`,
7+
respectively. The old names continue to exist within 1.4 series with a deprecation
8+
warning. In addition, the implicit coercion of :class:`.SelectBase`, :class:`.Alias`,
9+
and other SELECT oriented objects into scalar subqueries when evaluated in a column
10+
context is also deprecated, and emits a warning that the :meth:`.SelectBase.scalar_subquery`
11+
method should be called explicitly. This warning will in a later major release
12+
become an error, however the message will always be clear when :meth:`.SelectBase.scalar_subquery`
13+
needs to be invoked. The latter part of the change is for clarity and to reduce the
14+
implicit decisionmaking by the query coercion system.
15+
16+
This change is part of the larger change to convert :func:`.select` objects to no
17+
longer be directly part of the "from clause" class hierarchy, which also includes
18+
an overhaul of the clause coercion system.
19+

‎doc/build/core/tutorial.rst

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1590,24 +1590,24 @@ is often a :term:`correlated subquery`, which relies upon the enclosing
15901590
SELECT statement in order to acquire at least one of its FROM clauses.
15911591

15921592
The :func:`.select` construct can be modified to act as a
1593-
column expression by calling either the :meth:`~.SelectBase.as_scalar`
1593+
column expression by calling either the :meth:`~.SelectBase.scalar_subquery`
15941594
or :meth:`~.SelectBase.label` method:
15951595

15961596
.. sourcecode:: pycon+sql
15971597

1598-
>>> stmt = select([func.count(addresses.c.id)]).\
1598+
>>> subq = select([func.count(addresses.c.id)]).\
15991599
... where(users.c.id == addresses.c.user_id).\
1600-
... as_scalar()
1600+
... scalar_subquery()
16011601

16021602
The above construct is now a :class:`~.expression.ScalarSelect` object,
1603-
and is no longer part of the :class:`~.expression.FromClause` hierarchy;
1604-
it instead is within the :class:`~.expression.ColumnElement` family of
1605-
expression constructs. We can place this construct the same as any
1603+
which is an adapter around the original :class:`.~expression.Select`
1604+
object; it participates within the :class:`~.expression.ColumnElement`
1605+
family of expression constructs. We can place this construct the same as any
16061606
other column within another :func:`.select`:
16071607

16081608
.. sourcecode:: pycon+sql
16091609

1610-
>>> conn.execute(select([users.c.name, stmt])).fetchall()
1610+
>>> conn.execute(select([users.c.name, subq])).fetchall()
16111611
{opensql}SELECT users.name, (SELECT count(addresses.id) AS count_1
16121612
FROM addresses
16131613
WHERE users.id = addresses.user_id) AS anon_1
@@ -1620,10 +1620,10 @@ it using :meth:`.SelectBase.label` instead:
16201620

16211621
.. sourcecode:: pycon+sql
16221622

1623-
>>> stmt = select([func.count(addresses.c.id)]).\
1623+
>>> subq = select([func.count(addresses.c.id)]).\
16241624
... where(users.c.id == addresses.c.user_id).\
16251625
... label("address_count")
1626-
>>> conn.execute(select([users.c.name, stmt])).fetchall()
1626+
>>> conn.execute(select([users.c.name, subq])).fetchall()
16271627
{opensql}SELECT users.name, (SELECT count(addresses.id) AS count_1
16281628
FROM addresses
16291629
WHERE users.id = addresses.user_id) AS address_count
@@ -1633,7 +1633,7 @@ it using :meth:`.SelectBase.label` instead:
16331633

16341634
.. seealso::
16351635

1636-
:meth:`.Select.as_scalar`
1636+
:meth:`.Select.scalar_subquery`
16371637

16381638
:meth:`.Select.label`
16391639

@@ -1642,7 +1642,7 @@ it using :meth:`.SelectBase.label` instead:
16421642
Correlated Subqueries
16431643
---------------------
16441644

1645-
Notice in the examples on :ref:`scalar_selects`, the FROM clause of each embedded
1645+
In the examples on :ref:`scalar_selects`, the FROM clause of each embedded
16461646
select did not contain the ``users`` table in its FROM clause. This is because
16471647
SQLAlchemy automatically :term:`correlates` embedded FROM objects to that
16481648
of an enclosing query, if present, and if the inner SELECT statement would
@@ -1653,7 +1653,8 @@ still have at least one FROM clause of its own. For example:
16531653
>>> stmt = select([addresses.c.user_id]).\
16541654
... where(addresses.c.user_id == users.c.id).\
16551655
... where(addresses.c.email_address == 'jack@yahoo.com')
1656-
>>> enclosing_stmt = select([users.c.name]).where(users.c.id == stmt)
1656+
>>> enclosing_stmt = select([users.c.name]).\
1657+
... where(users.c.id == stmt.scalar_subquery())
16571658
>>> conn.execute(enclosing_stmt).fetchall()
16581659
{opensql}SELECT users.name
16591660
FROM users
@@ -1679,7 +1680,7 @@ may be correlated:
16791680
>>> enclosing_stmt = sele 10000 ct(
16801681
... [users.c.name, addresses.c.email_address]).\
16811682
... select_from(users.join(addresses)).\
1682-
... where(users.c.id == stmt)
1683+
... where(users.c.id == stmt.scalar_subquery())
16831684
>>> conn.execute(enclosing_stmt).fetchall()
16841685
{opensql}SELECT users.name, addresses.email_address
16851686
FROM users JOIN addresses ON users.id = addresses.user_id
@@ -1698,7 +1699,7 @@ as the argument:
16981699
... where(users.c.name == 'wendy').\
16991700
... correlate(None)
17001701
>>> enclosing_stmt = select([users.c.name]).\
1701-
... where(users.c.id == stmt)
1702+
... where(users.c.id == stmt.scalar_subquery())
17021703
>>> conn.execute(enclosing_stmt).fetchall()
17031704
{opensql}SELECT users.name
17041705
FROM users
@@ -1721,7 +1722,7 @@ by telling it to correlate all FROM clauses except for ``users``:
17211722
>>> enclosing_stmt = select(
17221723
... [users.c.name, addresses.c.email_address]).\
17231724
... select_from(users.join(addresses)).\
1724-
... where(users.c.id == stmt)
1725+
... where(users.c.id == stmt.scalar_subquery())
17251726
>>> conn.execute(enclosing_stmt).fetchall()
17261727
{opensql}SELECT users.name, addresses.email_address
17271728
FROM users JOIN addresses ON users.id = addresses.user_id

‎lib/sqlalchemy/dialects/mssql/base.py

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -672,6 +672,7 @@ class MyTable(Base):
672672
from ...engine import default
673673
from ...engine import reflection
674674
from ...sql import compiler
675+
from ...sql import elements
675676
from ...sql import expression
676677
from ...sql import quoted_name
677678
from ...sql import util as sql_util
@@ -1671,9 +1672,7 @@ def visit_column(self, column, add_to_result_map=None, **kw):
16711672
# translate for schema-qualified table aliases
16721673
t = self._schema_aliased_table(column.table)
16731674
if t is not None:
1674-
converted = expression._corresponding_column_or_error(
1675-
t, column
1676-
)
1675+
converted = elements._corresponding_column_or_error(t, column)
16771676
if add_to_result_map is not None:
16781677
add_to_result_map(
16791678
column.name,

‎lib/sqlalchemy/dialects/mysql/base.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -788,8 +788,10 @@ def process(element, compiler, **kw):
788788
from ... import util
789789
from ...engine import default
790790
from ...engine import reflection
791+
from ...sql import coercions
791792
from ...sql import compiler
792793
from ...sql import elements
794+
from ...sql import roles
793795
from ...types import BINARY
794796
from ...types import BLOB
795797
from ...types import BOOLEAN
@@ -1218,7 +1220,7 @@ def visit_json_path_getitem_op_binary(self, binary, operator, **kw):
12181220
def visit_on_duplicate_key_update(self, on_duplicate, **kw):
12191221
if on_duplicate._parameter_ordering:
12201222
parameter_ordering = [
1221-
elements._column_as_key(key)
1223+
coercions.expect(roles.DMLColumnRole, key)
12221224
for key in on_duplicate._parameter_ordering
12231225
]
12241226
ordered_keys = set(parameter_ordering)
@@ -1238,7 +1240,7 @@ def visit_on_duplicate_key_update(self, on_duplicate, **kw):
12381240
val = on_duplicate.update.get(column.key)
12391241
if val is None:
12401242
continue
1241-
elif elements._is_literal(val):
1243+
elif coercions._is_literal(val):
12421244
val = elements.BindParameter(None, val, type_=column.type)
12431245
value_text = self.process(val.self_group(), use_schema=False)
12441246
elif isinstance(val, elements.BindParameter) and val.type._isnull:

‎lib/sqlalchemy/dialects/postgresql/base.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -932,9 +932,11 @@ def bind_expression(self, bindvalue):
932932
from ... import util
933933
from ...engine import default
934934
from ...engine import reflection
935+
from ...sql import coercions
935936
from ...sql import compiler
936937
from ...sql import elements
937938
from ...sql import expression
939+
from ...sql import roles
938940
from ...sql import sqltypes
939941
from ...sql import util as sql_util
940942
from ...types import BIGINT
@@ -1774,7 +1776,7 @@ def visit_on_conflict_do_update(self, on_conflict, **kw):
17741776
col_key = c.key
17751777
if col_key in set_parameters:
17761778
value = set_parameters.pop(col_key)
1777-
if elements._is_literal(value):
1779+
if coercions._is_literal(value):
17781780
value = elements.BindParameter(None, value, type_=c.type)
17791781

17801782
else:
@@ -1806,7 +1808,8 @@ def visit_on_conflict_do_update(self, on_conflict, **kw):
18061808
else self.process(k, use_schema=False)
18071809
)
18081810
value_text = self.process(
1809-
elements._literal_as_binds(v), use_schema=False
1811+
coercions.expect(roles.ExpressionElementRole, v),
1812+
use_schema=False,
18101813
)
18111814
action_set_ops.append("%s = %s" % (key_text, value_text))
18121815

‎lib/sqlalchemy/dialects/postgresql/ext.py

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,12 @@
66
# the MIT License: http://www.opensource.org/licenses/mit-license.php
77

88
from .array import ARRAY
9+
from ... import util
10+
from ...sql import coercions
911
from ...sql import elements
1012
from ...sql import expression
1113
from ...sql import functions
14+
from ...sql import roles
1215
from ...sql.schema import ColumnCollectionConstraint
1316

1417

@@ -50,16 +53,18 @@ class aggregate_order_by(expression.ColumnElement):
5053
__visit_name__ = "aggregate_order_by"
5154

5255
def __init__(self, target, *order_by):
53-
self.target = elements._literal_as_binds(target)
56+
self.target = coercions.expect(roles.ExpressionElementRole, target)
5457

5558
_lob = len(order_by)
5659
if _lob == 0:
5760
raise TypeError("at least one ORDER BY element is required")
5861
elif _lob == 1:
59-
self.order_by = elements._literal_as_binds(order_by[0])
62+
self.order_by = coercions.expect(
63+
roles.ExpressionElementRole, order_by[0]
64+
)
6065
else:
6166
self.order_by = elements.ClauseList(
62-
*order_by, _literal_as_text=elements._literal_as_binds
67+
*order_by, _literal_as_text_role=roles.ExpressionElementRole
6368
)
6469

6570
def self_group(self, against=None):
@@ -166,7 +171,10 @@ def __init__(self, *elements, **kw):
166171
expressions, operators = zip(*elements)
167172

168173
for (expr, column, strname, add_element), operator in zip(
169-
self._extract_col_expression_collection(expressions), operators
174+
coercions.expect_col_expression_collection(
175+
roles.DDLConstraintColumnRole, expressions
176+
),
177+
operators,
170178
):
171179
if add_element is not None:
172180
columns.append(add_element)
@@ -177,8 +185,6 @@ def __init__(self, *elements, **kw):
177185
# backwards compat
178186
self.operators[name] = operator
179187

180-
expr = expression._literal_as_column(expr)
181-
182188
render_exprs.append((expr, name, operator))
183189

184190
self._render_exprs = render_exprs
@@ -193,9 +199,21 @@ def __init__(self, *elements, **kw):
193199
self.using = kw.get("using", "gist")
194200
where = kw.get("where")
195201
if where is not None:
196-
self.where = expression._literal_as_text(
197-
where, allow_coercion_to_text=True
202+
self.where = coercions.expect(roles.StatementOptionRole, where)
203+
204+
def _set_parent(self, table):
205+
super(ExcludeConstraint, self)._set_parent(table)
206+
207+
self._render_exprs = [
208+
(
209+
expr if isinstance(expr, elements.ClauseElement) else colexpr,
210+
name,
211+
operator,
212+
)
213+
for (expr, name, operator), colexpr in util.zip_longest(
214+
self._render_exprs, self.columns
198215
)
216+
]
199217

200218
def copy(self, **kw):
201219
elements = [(col, self.operators[col]) for col in self.columns.keys()]

‎lib/sqlalchemy/engine/result.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -626,7 +626,7 @@ def _key_fallback(self, key, raiseerr=True):
626626
if raiseerr:
627627
raise exc.NoSuchColumnError(
628628
"Could not locate column in row for column '%s'"
629-
% expression._string_or_unprintable(key)
629+
% util.string_or_unprintable(key)
630630
)
631631
else:
632632
return None

‎lib/sqlalchemy/ext/baked.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -269,7 +269,8 @@ def to_query(self, query_or_session):
269269
User.id == Address.user_id).correlate(Address)
270270
271271
main_bq = self.bakery(
272-
lambda s: s.query(Address.id, sub_bq.to_query(q).as_scalar())
272+
lambda s: s.query(
273+
Address.id, sub_bq.to_query(q).scalar_subquery())
273274
)
274275
275276
:param query_or_session: a :class:`.Query` object or a class

‎lib/sqlalchemy/ext/declarative/clsregistry.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,7 @@ def __getattr__(self, key):
217217
mp = class_mapper(self.cls, configure=False)
218218
if mp:
219219
if key not in mp.all_orm_descriptors:
220-
raise exc.InvalidRequestError(
220+
raise AttributeError(
221221
"Class %r does not have a mapped column named %r"
222222
% (self.cls, key)
223223
)

0 commit comments

Comments
 (0)
0