10000 Add support for LZ4 with compression of full-page writes in WAL · postgrespro/postgres@4035cd5 · GitHub
[go: up one dir, main page]
Skip to content

Commit 4035cd5

Browse files
michaelpqmichaelpq
committed
Add support for LZ4 with compression of full-page writes in WAL
The logic is implemented so as there can be a choice in the compression used when building a WAL record, and an extra per-record bit is used to track down if a block is compressed with PGLZ, LZ4 or nothing. wal_compression, the existing parameter, is changed to an enum with support for the following backward-compatible values: - "off", the default, to not use compression. - "pglz" or "on", to compress FPWs with PGLZ. - "lz4", the new mode, to compress FPWs with LZ4. Benchmarking has showed that LZ4 outclasses easily PGLZ. ZSTD would be also an interesting choice, but going just with LZ4 for now makes the patch minimalistic as toast compression is already able to use LZ4, so there is no need to worry about any build-related needs for this implementation. Author: Andrey Borodin, Justin Pryzby Reviewed-by: Dilip Kumar, Michael Paquier Discussion: https://postgr.es/m/3037310D-ECB7-4BF1-AF20-01C10BB33A33@yandex-team.ru
1 parent cc2c7d6 commit 4035cd5

File tree

13 files changed

+191
-52
lines changed
  • xlogrecord.h
  • tools/pgindent

    • 13 files changed

    +191
    -52
    lines changed

    ‎doc/src/sgml/config.sgml

    Lines changed: 9 additions & 5 deletions
    Original file line numberDiff line numberDiff line change
    @@ -3128,23 +3128,27 @@ include_dir 'conf.d'
    31283128
    </varlistentry>
    31293129

    31303130
    <varlistentry id="guc-wal-compression" xreflabel="wal_compression">
    3131-
    <term><varname>wal_compression</varname> (<type>boolean</type>)
    3131+
    <term><varname>wal_compression</varname> (<type>enum</type>)
    31323132
    <indexterm>
    31333133
    <primary><varname>wal_compression</varname> configuration parameter</primary>
    31343134
    </indexterm>
    31353135
    </term>
    31363136
    <listitem>
    31373137
    <para>
    3138-
    When this parameter is <literal>on</literal>, the <productname>PostgreSQL</productname>
    3138+
    This parameter enables compression of WAL using the specified
    3139+
    compression method.
    3140+
    When enabled, the <productname>PostgreSQL</productname>
    31393141
    server compresses full page images written to WAL when
    31403142
    <xref linkend="guc-full-page-writes"/> is on or during a base backup.
    31413143
    A compressed page image will be decompressed during WAL replay.
    3142-
    The default value is <literal>off</literal>.
    3143-
    Only superusers can change this setting.
    3144+
    The supported methods are <literal>pglz</literal> and
    3145+
    <literal>lz4</literal> (if <productname>PostgreSQL</productname> was
    3146+
    compiled with <option>--with-lz4</option>). The default value is
    3147+
    <literal>off</literal>. Only superusers can change this setting.
    31443148
    </para>
    31453149

    31463150
    <para>
    3147-
    Turning this parameter on can reduce the WAL volume without
    3151+
    Enabling compression can reduce the WAL volume without
    31483152
    increasing the risk of unrecoverable data corruption,
    31493153
    but at the cost of some extra CPU spent on the compression during
    31503154
    WAL logging and on the decompression during WAL replay.

    ‎doc/src/sgml/install-windows.sgml

    Lines changed: 1 addition & 1 deletion
    Original file line numberDiff line numberDiff line change
    @@ -299,7 +299,7 @@ $ENV{MSBFLAGS}="/m";
    299299
    <term><productname>LZ4</productname></term>
    300300
    <listitem><para>
    301301
    Required for supporting <productname>LZ4</productname> compression
    302-
    method for compressing the table data. Binaries and source can be
    302+
    method for compressing table or WAL data. Binaries and source can be
    303303
    downloaded from
    304304
    <ulink url="https://github.com/lz4/lz4/releases"></ulink>.
    305305
    </para></listitem>

    ‎doc/src/sgml/installation.sgml

    Lines changed: 3 additions & 2 deletions
    Original file line numberDiff line numberDiff line change
    @@ -270,7 +270,8 @@ su - postgres
    270270
    <para>
    271271
    You need <productname>LZ4</productname>, if you want to support
    272272
    compression of data with this method; see
    273-
    <xref linkend="guc-default-toast-compression"/>.
    273+
    <xref linkend="guc-default-toast-compression"/> and
    274+
    <xref linkend="guc-wal-compression"/>.
    274275
    </para>
    275276
    </listitem>
    276277

    @@ -980,7 +981,7 @@ build-postgresql:
    980981
    <para>
    981982
    Build with <productname>LZ4</productname> compression support.
    982983
    This allows the use of <productname>LZ4</productname> for
    983-
    compression of table data.
    984+
    compression of table and WAL data.
    984985
    </para>
    985986
    </listitem>
    986987
    </varlistentry>

    ‎doc/src/sgml/standalone-profile.xsl

    Lines changed: 4 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -52,6 +52,10 @@ variant without links and references to the main documentation.
    5252
    <xsl:text>the configuration parameter default_toast_compression</xsl:text>
    5353
    </xsl:template>
    5454

    55 F438 +
    <xsl:template match="xref[@linkend='guc-wal-compression']">
    56+
    <xsl:text>the configuration parameter wal_compression</xsl:text>
    57+
    </xsl:template>
    58+
    5559
    <xsl:template match="xref[@linkend='install-windows']">
    5660
    <xsl:text>the documentation</xsl:text>
    5761
    </xsl:template>

    ‎src/backend/access/transam/xlog.c

    Lines changed: 1 addition & 1 deletion
    Original file line numberDiff line numberDiff line change
    @@ -98,7 +98,7 @@ char *XLogArchiveCommand = NULL;
    9898
    bool EnableHotStandby = false;
    9999
    bool fullPageWrites = true;
    100100
    bool wal_log_hints = false;
    101-
    bool wal_compression = false;
    101+
    int wal_compression = WAL_COMPRESSION_NONE;
    102102
    char *wal_consistency_checking_string = NULL;
    103103
    bool *wal_consistency_checking = NULL;
    104104
    bool wal_init_zero = true;

    ‎src/backend/access/transam/xloginsert.c

    Lines changed: 64 additions & 9 deletions
    Original file line numberDiff line numberDiff line change
    @@ -33,8 +33,20 @@
    3333
    #include "storage/proc.h"
    3434
    #include "utils/memutils.h"
    3535

    36-
    /* Buffer size required to store a compressed version of backup block image */
    37-
    #define PGLZ_MAX_BLCKSZ PGLZ_MAX_OUTPUT(BLCKSZ)
    36+
    /*
    37+
    * Guess the maximum buffer size required to store a compressed version of
    38+
    * backup block image.
    39+
    */
    40+
    #ifdef USE_LZ4
    41+
    #include <lz4.h>
    42+
    #define LZ4_MAX_BLCKSZ LZ4_COMPRESSBOUND(BLCKSZ)
    43+
    #else
    44+
    #define LZ4_MAX_BLCKSZ 0
    45+
    #endif
    46+
    47+
    #define PGLZ_MAX_BLCKSZ PGLZ_MAX_OUTPUT(BLC 10000 KSZ)
    48+
    49+
    #define COMPRESS_BUFSIZE Max(PGLZ_MAX_BLCKSZ, LZ4_MAX_BLCKSZ)
    3850

    3951
    /*
    4052
    * For each block reference registered with XLogRegisterBuffer, we fill in
    @@ -58,7 +70,7 @@ typedef struct
    5870
    * backup block data in XLogRecordAssemble() */
    5971

    6072
    /* buffer to store a compressed version of backup block image */
    61-
    char compressed_page[PGLZ_MAX_BLCKSZ];
    73+
    char compressed_page[COMPRESS_BUFSIZE];
    6274
    } registered_buffer;
    6375

    6476
    static registered_buffer *registered_buffers;
    @@ -628,7 +640,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
    628640
    /*
    629641
    * Try to compress a block image if wal_compression is enabled
    630642
    */
    631-
    if (wal_compression)
    643+
    if (wal_compression != WAL_COMPRESSION_NONE)
    632644
    {
    633645
    is_compressed =
    634646
    XLogCompressBackupBlock(page, bimg.hole_offset,
    @@ -665,8 +677,29 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
    665677

    666678
    if (is_compressed)
    667679
    {
    680+
    /* The current compression is stored in the WAL record */
    668681
    bimg.length = compressed_len;
    669-
    bimg.bimg_info |= BKPIMAGE_IS_COMPRESSED;
    682+
    683+
    /* Set the compression method used for this block */
    684+
    switch ((WalCompression) wal_compression)
    685+
    {
    686+
    case WAL_COMPRESSION_PGLZ:
    687+
    bimg.bimg_info |= BKPIMAGE_COMPRESS_PGLZ;
    688+
    break;
    689+
    690+
    case WAL_COMPRESSION_LZ4:
    691+
    #ifdef USE_LZ4
    692+
    bimg.bimg_info |= BKPIMAGE_COMPRESS_LZ4;
    693+
    #else
    694+
    elog(ERROR, "LZ4 is not supported by this build");
    695+
    #endif
    696+
    break;
    697+
    698+
    case WAL_COMPRESSION_NONE:
    699+
    Assert(false); /* cannot happen */
    700+
    break;
    701+
    /* no default case, so that compiler will warn */
    702+
    }
    670703

    671704
    rdt_datas_last->data = regbuf->compressed_page;
    672705
    rdt_datas_last->len = compressed_len;
    @@ -853,12 +886,34 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
    853886
    else
    854887
    source = page;
    855888

    889+
    switch ((WalCompression) wal_compression)
    890+
    {
    891+
    case WAL_COMPRESSION_PGLZ:
    892+
    len = pglz_compress(source, orig_len, dest, PGLZ_strategy_default);
    893+
    break;
    894+
    895+
    case WAL_COMPRESSION_LZ4:
    896+
    #ifdef USE_LZ4
    897+
    len = LZ4_compress_default(source, dest, orig_len,
    898+
    COMPRESS_BUFSIZE);
    899+
    if (len <= 0)
    900+
    len = -1; /* failure */
    901+
    #else
    902+
    elog(ERROR, "LZ4 is not supported by this build");
    903+
    #endif
    904+
    break;
    905+
    906+
    case WAL_COMPRESSION_NONE:
    907+
    Assert(false); /* cannot happen */
    908+
    break;
    909+
    /* no default case, so that compiler will warn */
    910+
    }
    911+
    856912
    /*
    857-
    * We recheck the actual size even if pglz_compress() reports success and
    858-
    * see if the number of bytes saved by compression is larger than the
    859-
    * length of extra data needed for the compressed version of block image.
    913+
    * We recheck the actual size even if compression reports success and see
    914+
    * if the number of bytes saved by compression is larger than the length
    915+
    * of extra data needed for the compressed version of block image.
    860916
    */
    861-
    len = pglz_compress(source, orig_len, dest, PGLZ_strategy_default);
    862917
    if (len >= 0 &&
    863918
    < F438 span class=pl-s1>len + extra_bytes < orig_len)
    864919
    {

    ‎src/backend/access/transam/xlogreader.c

    Lines changed: 46 additions & 12 deletions
    Original file line numberDiff line numberDiff line change
    @@ -18,6 +18,9 @@
    1818
    #include "postgres.h"
    1919

    2020
    #include <unistd.h>
    21+
    #ifdef USE_LZ4
    22+
    #include <lz4.h>
    23+
    #endif
    2124

    2225
    #include "access/transam.h"
    2326
    #include "access/xlog_internal.h"
    @@ -1290,7 +1293,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
    12901293

    12911294
    blk->apply_image = ((blk->bimg_info & BKPIMAGE_APPLY) != 0);
    12921295

    1293-
    if (blk->bimg_info & BKPIMAGE_IS_COMPRESSED)
    1296+
    if (BKPIMAGE_COMPRESSED(blk->bimg_info))
    12941297
    {
    12951298
    if (blk->bimg_info & BKPIMAGE_HAS_HOLE)
    12961299
    COPY_HEADER_FIELD(&blk->hole_length, sizeof(uint16));
    @@ -1335,29 +1338,28 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
    13351338
    }
    13361339

    13371340
    /*
    1338-
    * cross-check that bimg_len < BLCKSZ if the IS_COMPRESSED
    1339-
    * flag is set.
    1341+
    * Cross-check that bimg_len < BLCKSZ if it is compressed.
    13401342
    */
    1341-
    if ((blk->bimg_info & BKPIMAGE_IS_COMPRESSED) &&
    1343+
    if (BKPIMAGE_COMPRESSED(blk->bimg_info) &&
    13421344
    blk->bimg_len == BLCKSZ)
    13431345
    {
    13441346
    report_invalid_record(state,
    1345-
    "BKPIMAGE_IS_COMPRESSED set, but block image length %u at %X/%X",
    1347+
    "BKPIMAGE_COMPRESSED set, but block image length %u at %X/%X",
    13461348
    (unsigned int) blk->bimg_len,
    13471349
    LSN_FORMAT_ARGS(state->ReadRecPtr));
    13481350
    goto err;
    13491351
    }
    13501352

    13511353
    /*
    1352-
    * cross-check that bimg_len = BLCKSZ if neither HAS_HOLE nor
    1353-
    * IS_COMPRESSED flag is set.
    1354+
    * cross-check that bimg_len = BLCKSZ if neither HAS_HOLE is
    1355+
    * set nor COMPRESSED().
    13541356
    */
    13551357
    if (!(blk->bimg_info & BKPIMAGE_HAS_HOLE) &&
    1356-
    !(blk->bimg_info & BKPIMAGE_IS_COMPRESSED) &&
    1358+
    !BKPIMAGE_COMPRESSED(blk->bimg_info) &&
    13571359
    blk->bimg_len != BLCKSZ)
    13581360
    {
    13591361
    report_invalid_record(state,
    1360-
    "neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_IS_COMPRESSED set, but block image length is %u at %X/%X",
    1362+
    "neither BKPIMAGE_HAS_HOLE nor BKPIMAGE_COMPRESSED set, but block image length is %u at %X/%X",
    13611363
    (unsigned int) blk->data_len,
    13621364
    LSN_FORMAT_ARGS(state->ReadRecPtr));
    13631365
    goto err;
    @@ -1555,17 +1557,49 @@ RestoreBlockImage(XLogReaderState *record, uint8 block_id, char *page)
    15551557
    bkpb = &record->blocks[block_id];
    15561558
    ptr = bkpb->bkp_image;
    15571559

    1558-
    if (bkpb->bimg_info & BKPIMAGE_IS_COMPRESSED)
    1560+
    if (BKPIMAGE_COMPRESSED(bkpb->bimg_info))
    15591561
    {
    15601562
    /* If a backup block image is compressed, decompress it */
    1561-
    if (pglz_decompress(ptr, bkpb->bimg_len, tmp.data,
    1562-
    BLCKSZ - bkpb->hole_length, true) < 0)
    1563+
    bool decomp_success = true;
    1564+
    1565+
    if ((bkpb->bimg_info & BKPIMAGE_COMPRESS_PGLZ) != 0)
    1566+
    {
    1567+
    if (pglz_decompress(ptr, bkpb->bimg_len, tmp.data,
    1568+
    BLCKSZ - bkpb->hole_length, true) < 0)
    1569+
    decomp_success = false;
    1570+
    }
    1571+
    else if ((bkpb->bimg_info & BKPIMAGE_COMPRESS_LZ4) != 0)
    1572+
    {
    1573+
    #ifdef USE_LZ4
    1574+
    if (LZ4_decompress_safe(ptr, tmp.data,
    1575+
    bkpb->bimg_len, BLCKSZ - bkpb->hole_length) <= 0)
    1576+
    decomp_success = false;
    1577+
    #else
    1578+
    report_invalid_record(record, "image at %X/%X compressed with %s not supported by build, block %d",
    1579+
    (uint32) (record->ReadRecPtr >> 32),
    1580+
    (uint32) record->ReadRecPtr,
    1581+
    "LZ4",
    1582+
    block_id);
    1583+
    return false;
    1584+
    #endif
    1585+
    }
    1586+
    else
    1587+
    {
    1588+
    report_invalid_record(record, "image at %X/%X compressed with unknown method, block %d",
    1589+
    (uint32) (record->ReadRecPtr >> 32),
    1590+
    (uint32) record->ReadRecPtr,
    1591+
    block_id);
    1592+
    return false;
    1593+
    }
    1594+
    1595+
    if (!decomp_success)
    15631596
    {
    15641597
    report_invalid_record(record, "invalid compressed image at %X/%X, block %d",
    15651598
    LSN_FORMAT_ARGS(record->ReadRecPtr),
    15661599
    block_id);
    15671600
    return false;
    15681601
    }
    1602+
    15691603
    ptr = tmp.data;
    15701604
    }
    15711605

    ‎src/backend/utils/misc/guc.c

    Lines changed: 26 additions & 10 deletions
    Original file line numberDiff line numberDiff line change
    @@ -540,6 +540,22 @@ static struct config_enum_entry default_toast_compression_options[] = {
    540540
    {NULL, 0, false}
    541541
    };
    542542

    543+
    static const struct config_enum_entry wal_compression_options[] = {
    544+
    {"pglz", WAL_COMPRESSION_PGLZ, false},
    545+
    #ifdef USE_LZ4
    546+
    {"lz4", WAL_COMPRESSION_LZ4, false},
    547+
    #endif
    548+
    {"on", WAL_COMPRESSION_PGLZ, false},
    549+
    {"off", WAL_COMPRESSION_NONE, false},
    550+
    {"true", WAL_COMPRESSION_PGLZ, true},
    551+
    {"false", WAL_COMPRESSION_NONE, true},
    552+
    {"yes", WAL_COMPRESSION_PGLZ, true},
    553+
    {"no", WAL_COMPRESSION_NONE, true},
    554+
    {"1", WAL_COMPRESSION_PGLZ, true},
    555+
    {"0", WAL_COMPRESSION_NONE, true},
    556+
    {NULL, 0, false}
    557+
    };
    558+
    543559
    /*
    544560
    * Options for enum values stored in other modules
    545561
    */
    @@ -1304,16 +1320,6 @@ static struct config_bool ConfigureNamesBool[] =
    13041320
    NULL, NULL, NULL
    13051321
    },
    13061322

    1307-
    {
    1308-
    {"wal_compression", PGC_SUSET, WAL_SETTINGS,
    1309-
    gettext_noop("Compresses full-page writes written in WAL file."),
    1310-
    NULL
    1311-
    },
    1312-
    &wal_compression,
    1313-
    false,
    1314-
    NULL, NULL, NULL
    1315-
    },
    1316-
    13171323
    {
    13181324
    {"wal_init_zero", PGC_SUSET, WAL_SETTINGS,
    13191325
    gettext_noop("Writes zeroes to new WAL files before first use."),
    @@ -4816,6 +4822,16 @@ static struct config_enum ConfigureNamesEnum[] =
    48164822
    NULL, NULL, NULL
    48174823
    },
    48184824

    4825+
    {
    4826+
    {"wal_compression", PGC_SUSET, WAL_SETTINGS,
    4827+
    gettext_noop("Compresses full-page writes written in WAL file with specified method."),
    4828+
    NULL
    4829+
    },
    4830+
    &wal_compression,
    4831+
    WAL_COMPRESSION_NONE, wal_compression_options,
    4832+
    NULL, NULL, NULL
    4833+
    },
    4834+
    48194835
    {
    48204836
    {"wal_level", PGC_POSTMASTER, WAL_SETTINGS,
    48214837
    gettext_noop("Sets the level of information written to the WAL."),

    ‎src/backend/utils/misc/postgresql.conf.sample

    Lines changed: 2 additions & 1 deletion
    Original file line numberDiff line numberDiff line change
    @@ -218,7 +218,8 @@
    218218
    #full_page_writes = on # recover from partial page writes
    219219
    #wal_log_hints = off # also do full page writes of non-critical updates
    220220
    # (change requires restart)
    221-
    #wal_compression = off # enable compression of full-page writes
    221+
    #wal_compression = off # enables compression of full-page writes;
    222+
    # off, pglz, lz4, or on
    222223
    #wal_init_zero = on # zero-fill new WAL files
    223224
    #wal_recycle = on # recycle WAL files
    224225
    #wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers

    0 commit comments

    Comments
     (0)
    0