8000 Add logical_decoding_work_mem to limit ReorderBuffer memory usage. · postgres/postgres@cec2edf · GitHub
[go: up one dir, main page]
Skip to content

Commit cec2edf

Browse files
author
Amit Kapila
committed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
Instead of deciding to serialize a transaction merely based on the number of changes in that xact (toplevel or subxact), this makes the decisions based on amount of memory consumed by the changes. The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this SET logical_decoding_work_mem = '128kB' to reduce the memory usage of walsenders or set the higher value to reduce disk writes. The minimum value is 64kB. When adding a change to a transaction, we account for the size in two places. Firstly, in the ReorderBuffer, which is then used to decide if we reached the total memory limit. And secondly in the transaction the change belongs to, so that we can pick the largest transaction to evict (and serialize to disk). We still use max_changes_in_memory when loading changes serialized to disk. The trouble is we can't use the memory limit directly as there might be multiple subxact serialized, we need to read all of them but we don't know how many are there (and which subxact to read first). We do not serialize the ReorderBufferTXN entries, so if there is a transaction with many subxacts, most memory may be in this type of objects. Those records are not included in the memory accounting. We also do not account for INTERNAL_TUPLECID changes, which are kept in a separate list and not evicted from memory. Transactions with many CTID changes may consume significant amounts of memory, but we can't really do much about that. The current eviction algorithm is very simple - the transaction is picked merely by size, while it might be useful to also consider age (LSN) of the changes for example. With the new Generational memory allocator, evicting the oldest changes would make it more likely the memory gets actually pfreed. The logical_decoding_work_mem can be set in postgresql.conf, in which case it serves as the default for all publishers on that instance. Author: Tomas Vondra, with changes by Dilip Kumar and Amit Kapila Reviewed-by: Dilip Kumar and Amit Kapila Tested-By: Vignesh C Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
1 parent 2110f71 commit cec2edf

File tree

6 files changed

+333
-12
lines changed

6 files changed

+333
-12
lines changed

‎contrib/test_decoding/logical.conf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
wal_level = logical
22
max_replication_slots = 4
3+
logical_decoding_work_mem = 64kB

‎doc/src/sgml/config.sgml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1732,6 +1732,27 @@ include_dir 'conf.d'
17321732
</listitem>
17331733
</varlistentry>
17341734

1735+
<varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
1736+
<term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
1737+
<indexterm>
1738+
<primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
1739+
</indexterm>
1740+
</term>
1741+
<listitem>
1742+
<para>
1743+
Specifies the maximum amount of memory to be used by logical decoding,
1744+
before some of the decoded changes are written to local disk. This
1745+
limits the amount of memory used by logical streaming replication
1746+
connections. It defaults to 64 megabytes (<literal>64MB</literal>).
1747+
Since each replication connection only uses a single buffer of this size,
1748+
and an installation normally doesn't have many such connections
1749+
concurrently (as limited by <varname>max_wal_senders</varname>), it's
1750+
safe to set this value significantly higher than <varname>work_mem</varname>,
1751+
reducing the amount of decoded changes written to disk.
1752+
</para>
1753+
</listitem>
1754+
</varlistentry>
1755+
17351756
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
17361757
<term><varname>max_stack_depth</varname> (<type>integer</type>)
17371758
<indexterm>

0 commit comments

Comments
 (0)
0