8000 Log the conflicts while applying changes in logical replication. · postgres/postgres@9758174 · GitHub
[go: up one dir, main page]
Skip to content

Commit 9758174

Browse files
author
Amit Kapila
committed
Log the conflicts while applying changes in logical replication.
This patch provides the additional logging information in the following conflict scenarios while applying changes: insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint. update_differ: Updating a row that was previously modified by another origin. update_exists: The updated row value violates a NOT DEFERRABLE unique constraint. update_missing: The tuple to be updated is missing. delete_differ: Deleting a row that was previously modified by another origin. delete_missing: The tuple to be deleted is missing. For insert_exists and update_exists conflicts, the log can include the origin and commit timestamp details of the conflicting key with track_commit_timestamp enabled. update_differ and delete_differ conflicts can only be detected when track_commit_timestamp is enabled on the subscriber. We do not offer additional logging for exclusion constraint violations because these constraints can specify rules that are more complex than simple equality checks. Resolving such conflicts won't be straightforward. This area can be further enhanced if required. Author: Hou Zhijie Reviewed-by: Shveta Malik, Amit Kapila, Nisha Moond, Hayato Kuroda, Dilip Kumar Discussion: https://postgr.es/m/OS0PR01MB5716352552DFADB8E9AD1D8994C92@OS0PR01MB5716.jpnprd01.prod.outlook.com
1 parent adf97c1 commit 9758174

File tree

18 files changed

+1033
-143
lines changed

18 files changed

+1033
-143
lines changed

‎doc/src/sgml/logical-replication.sgml

Lines changed: 98 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1579,8 +1579,91 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
15791579
node. If incoming data violates any constraints the replication will
15801580
stop. This is referred to as a <firstterm>conflict</firstterm>. When
15811581
replicating <command>UPDATE</command> or <command>DELETE</command>
1582-
operations, missing data will not produce a conflict and such operations
1583-
will simply be skipped.
1582+
operations, missing data is also considered as a
1583+
<firstterm>conflict</firstterm>, but does not result in an error and such
1584+
operations will simply be skipped.
1585+
</para>
1586+
1587+
<para>
1588+
Additional logging is triggered in the following <firstterm>conflict</firstterm>
1589+
cases:
1590+
<variablelist>
1591+
<varlistentry>
1592+
<term><literal>insert_exists</literal></term>
1593+
<listitem>
1594+
<para>
1595+
Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
1596+
unique constraint. Note that to log the origin and commit
1597+
timestamp details of the conflicting key,
1598+
<link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
1599+
should be enabled on the subscriber. In this case, an error will be
1600+
raised until the conflict is resolved manually.
1601+
</para>
1602+
</listitem>
1603+
</varlistentry>
1604+
<varlistentry>
1605+
<term><literal>update_differ</literal></term>
1606+
<listitem>
1607+
<para>
1608+
Updating a row that was previously modified by another origin.
1609+
Note that this conflict can only be detected when
1610+
<link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
1611+
is enabled on the subscriber. Currenly, the update is always applied
1612+
regardless of the origin of the local row.
1613+
</para>
1614+
</listitem>
1615+
</varlistentry>
1616+
<varlistentry>
1617+
<term><literal>update_exists</literal></term>
1618+
<listitem>
1619+
<para>
1620+
The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
1621+
unique constraint. Note that to log the origin and commit
1622+
timestamp details of the conflicting key,
1623+
<link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
1624+
should be enabled on the subscriber. In this case, an error will be
1625+
raised until the conflict is resolved manually. Note that when updating a
1626+
partitioned table, if the updated row value satisfies another partition
1627+
constraint resulting in the row being inserted into a new partition, the
1628+
<literal>insert_exists</literal> conflict may arise if the new row
1629+
violates a <literal>NOT DEFERRABLE</literal> unique constraint.
1630+
</para>
1631+
</listitem>
1632+
</varlistentry>
1633+
<varlistentry>
1634+
<term><literal>update_missing</literal></term>
1635+
<listitem>
1636+
<para>
1637+
The tuple to be updated was not found. The update will simply be
1638+
skipped in this scenario.
1639+
</para>
1640+
</listitem>
1641+
</varlistentry>
1642+
<varlistentry>
1643+
<term><literal>delete_differ</literal></term>
1644+
<listitem>
1645+
<para>
1646+
Deleting a row that was previously modified by another origin. Note that
1647+
this conflict can only be detected when
1648+
<link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
1649+
is enabled on the subscriber. Currenly, the delete is always applied
1650+
regardless of the origin of the local row.
1651+
</para>
1652+
</listitem>
1653+
</varlistentry>
1654+
<varlistentry>
1655+
<term><literal>delete_missing</literal></term>
1656+
<listitem>
1657+
<para>
1658+
The tuple to be deleted was not found. The delete will simply be
1659+
skipped in this scenario.
1660+
</para>
1661+
</listitem>
1662+
</varlistentry>
1663+
</variablelist>
1664+
Note that there are other conflict scenarios, such as exclusion constraint
1665+
violations. Currently, we do not provide additional details for them in the
1666+
log.
15841667
</para>
15851668

15861669
<para>
@@ -1597,7 +1680,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
15971680
</para>
15981681

15991682
<para>
1600-
A conflict will produce an error and will stop the replication; it must be
1683+
A conflict that produces an error will stop the replication; it must be
16011684
resolved manually by the user. Details about the conflict can be found in
16021685
the subscriber's server log.
16031686
</para>
@@ -1609,8 +1692,9 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
16091692
an error, the replication won't proceed, and the logical replication worker will
16101693
emit the following kind of message to the subscriber's server log:
16111694
<screen>
1612-
ERROR: duplicate key value violates unique constraint "test_pkey"
1613-
DETAIL: Key (c)=(1) already exists.
1695+
ERROR: conflict detected on relation "public.test": conflict=insert_exists
1696+
DETAIL: Key already exists in unique index "t_pkey", which was modified locally in transaction 740 at 2024-06-26 10:47:04.727375+08.
1697+
Key (c)=(1); existing local tuple (1, 'local'); remote tuple (1, 'remote').
16141698
CONTEXT: processing remote data for replication origin "pg_16395" during "INSERT" for replication target relation "public.test" in transaction 725 finished at 0/14C0378
16151699
</screen>
16161700
The LSN of the transaction that contains the change violating the constraint and
@@ -1636,6 +1720,15 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
16361720
Please note that skipping the whole transaction includes skipping changes that
16371721
might not violate any constraint. This can easily make the subscriber
16381722
inconsistent.
1723+
The additional details regarding conflicting rows, such as their origin and
1724+
commit timestamp can be seen in the <literal>DETAIL</literal> line of the
1725+
log. But note that this information is only available when
1726+
<link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
1727+
is enabled on the subscriber. Users can use this information to decide
1728+
whether to retain the local change or adopt the remote alteration. For
1729+
instance, the <literal>DETAIL</literal> line in the above log indicates that
1730+
the existing row was modified locally. Users can manually perform a
1731+
remote-change-win.
16391732
</para>
16401733

16411734
<para>

‎src/backend/access/index/genam.c

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,8 +154,9 @@ IndexScanEnd(IndexScanDesc scan)
154154
*
155155
* Construct a string describing the contents of an index entry, in the
156156
* form "(key_name, ...)=(key_value, ...)". This is currently used
157-
* for building unique-constraint and exclusion-constraint error messages,
158-
* so only key columns of the index are checked and printed.
157+
* for building unique-constraint, exclusion-constraint error messages, and
158+
* logical replication conflict error messages so only key columns of the index
159+
* are checked and printed.
159160
*
160161
* Note that if the user does not have permissions to view all of the
161162
* columns involved then a NULL is returned. Returning a partial key seems

‎src/backend/catalog/index.c

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2631,8 +2631,9 @@ CompareIndexInfo(const IndexInfo *info1, const IndexInfo *info2,
26312631
* Add extra state to IndexInfo record
26322632
*
26332633
* For unique indexes, we usually don't want to add info to the IndexInfo for
2634-
* checking uniqueness, since the B-Tree AM handles that directly. However,
2635-
* in the case of speculative insertion, additional support is required.
2634+
* checking uniqueness, since the B-Tree AM handles that directly. However, in
2635+
* the case of speculative insertion and conflict detection in logical
2636+
* replication, additional support is required.
26362637
*
26372638
* Do this processing here rather than in BuildIndexInfo() to not incur the
26382639
* overhead in the common non-speculative cases.

‎src/backend/executor/execIndexing.c

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
207207
ii = BuildIndexInfo(indexDesc);
208208

209209
/*
210-
* If the indexes are to be used for speculative insertion, add extra
211-
* information required by unique index entries.
210+
* If the indexes are to be used for speculative insertion or conflict
211+
* detection in logical replication, add extra information required by
212+
* unique index entries.
212213
*/
213214
if (speculative && ii->ii_Unique)
214215
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -519,14 +520,18 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
519520
*
520521
* Note that this doesn't lock the values in any way, so it's
521522
* possible that a conflicting tuple is inserted immediately
522-
* after this returns. But this can be used for a pre-check
523-
* before insertion.
523+
* after this returns. This can be used for either a pre-check
524+
* before insertion or a re-check after finding a conflict.
525+
*
526+
* 'tupleid' should be the TID of the tuple that has been recently
527+
* inserted (or can be invalid if we haven't inserted a new tuple yet).
528+
* This tuple will be excluded from conflict checking.
524529
* ----------------------------------------------------------------
525530
*/
526531
bool
527532
ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
528533
EState *estate, ItemPointer conflictTid,
529-
List *arbiterIndexes)
534+
ItemPointer tupleid, List *arbiterIndexes)
530535
{
531536
int i;
532537
int numIndices;
@@ -629,7 +634,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
629634

630635
satisfiesConstraint =
631636
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
632-
indexInfo, &invalidItemPtr,
637+
indexInfo, tupleid,
633638
values, isnull, estate, false,
634639
CEOUC_WAIT, true,
635640
conflictTid);

‎src/backend/executor/execMain.c

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -88,11 +88,6 @@ static bool ExecCheckPermissionsModified(Oid relOid, Oid userid,
8888
Bitmapset *modifiedCols,
8989
AclMode requiredPerms);
9090
static void ExecCheckXactReadOnly(PlannedStmt *plannedstmt);
91-
static char *ExecBuildSlotValueDescription(Oid reloid,
92-
TupleTableSlot *slot,
93-
TupleDesc tupdesc,
94-
Bitmapset *modifiedCols,
95-
int maxfieldlen);
9691
static void EvalPlanQualStart(EPQState *epqstate, Plan *planTree);
9792

9893
/* end of local decls */
@@ -2210,7 +2205,7 @@ ExecWithCheckOptions(WCOKind kind, ResultRelInfo *resultRelInfo,
22102205
* column involved, that subset will be returned with a key identifying which
22112206
* columns they are.
22122207
*/
2213-
static char *
2208+
char *
22142209
ExecBuildSlotValueDescription(Oid reloid,
22152210
TupleTableSlot *slot,
22162211
TupleDesc tupdesc,

0 commit comments

Comments
 (0)
0