From 4b71ace64db884ac3c2c94cecdf86446a1d3131c Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Thu, 8 May 2025 17:06:25 +0300 Subject: [PATCH 01/10] updated rc2 To temp hide indexing until release --- contrib/pg_tde/documentation/docs/release-notes/rc2.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index 3de65ba1311fa..9f0096aa0995a 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -1,3 +1,5 @@ + + # pg_tde Release Candidate ({{date.RC2}}) `pg_tde` extension brings in [Transparent Data Encryption (TDE)](../index/index.md) to PostgreSQL and enables you to keep sensitive data safe and secure. From e0a767d1b84cfffbcbe08f27f4ec44b671b6faf1 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 12 May 2025 09:51:07 +0300 Subject: [PATCH 02/10] Update mkdocs.yml --- contrib/pg_tde/documentation/mkdocs.yml | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/contrib/pg_tde/documentation/mkdocs.yml b/contrib/pg_tde/documentation/mkdocs.yml index 9a3027a979880..f0a31fda328e8 100644 --- a/contrib/pg_tde/documentation/mkdocs.yml +++ b/contrib/pg_tde/documentation/mkdocs.yml @@ -116,16 +116,16 @@ plugins: - macros: include_yaml: - 'variables.yml' # Use in markdown as '{{ VAR }}' - #- with-pdf: # https://github.com/orzih/mkdocs-with-pdf - # output_path: '_pdf/PerconaTDE.pdf' - # cover_title: 'Percona Transparent Data Encryption' - # cover_subtitle: Release Candidate (2025-03-27) - # author: 'Percona Technical Documentation Team' - # cover_logo: docs/_images/Percona_Logo_Color.png - # debug_html: false -# two_columns_level: 3 -# custom_template_path: _resource/templates -# enabled_if_env: ENABLE_PDF_EXPORT + - with-pdf: # https://github.com/orzih/mkdocs-with-pdf + output_path: '_pdf/PerconaTDE.pdf' + cover_title: 'Percona Transparent Data Encryption' + cover_subtitle: Release Candidate (2025-03-27) + author: 'Percona Technical Documentation Team' + cover_logo: docs/_images/Percona_Logo_Color.png + debug_html: false + two_columns_level: 3 + custom_template_path: _resource/templates + enabled_if_env: ENABLE_PDF_EXPORT extra: version: @@ -193,6 +193,7 @@ nav: - faq.md - "Release Notes": - "pg_tde release notes": release-notes/release-notes.md + - release-notes/rc2.md - release-notes/rc.md - release-notes/beta2.md - release-notes/beta.md From 31fd33c26758feb705213d47dcbdc352ff8b3993 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 12 May 2025 17:06:03 +0300 Subject: [PATCH 03/10] updated with RC2 added RC2 information and added 1 small fix --- .../docs/index/tde-limitations.md | 2 +- .../documentation/docs/release-notes/rc2.md | 36 +++++++++---------- .../docs/release-notes/release-notes.md | 1 + contrib/pg_tde/documentation/variables.yml | 6 ++-- 4 files changed, 23 insertions(+), 22 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/index/tde-limitations.md b/contrib/pg_tde/documentation/docs/index/tde-limitations.md index 20b760a431026..f1c07db6f0c99 100644 --- a/contrib/pg_tde/documentation/docs/index/tde-limitations.md +++ b/contrib/pg_tde/documentation/docs/index/tde-limitations.md @@ -4,6 +4,6 @@ * System tables are currently not encrypted. This means that statistics data and database metadata are currently not encrypted. * `pg_rewind` doesn't work with encrypted WAL for now. We plan to fix it in future releases. -* `pb_tde` Release candidate is incompatible with `pg_tde`Beta2 due to significant changes in code. There is no direct upgrade flow from one version to another. You must [uninstall](../how-to/uninstall.md) `pg_tde` Beta2 first and then [install](../install.md) and configure the new Release Candidate version. +* `pg_tde` Release candidate is incompatible with `pg_tde`Beta2 due to significant changes in code. There is no direct upgrade flow from one version to another. You must [uninstall](../how-to/uninstall.md) `pg_tde` Beta2 first and then [install](../install.md) and configure the new Release Candidate version. [Versions and supported PostgreSQL deployments](supported-versions.md){.md-button} diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index 9f0096aa0995a..2d2e7821b96b2 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -1,6 +1,4 @@ - - -# pg_tde Release Candidate ({{date.RC2}}) +# pg_tde Release Candidate 2 ({{date.RC2}}) `pg_tde` extension brings in [Transparent Data Encryption (TDE)](../index/index.md) to PostgreSQL and enables you to keep sensitive data safe and secure. @@ -14,21 +12,23 @@ This release provides the following features and improvements: The approach to WAL encryption has been redesigned. Now, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on. The information about what is encrypted is stored in the internal key metadata. This change improves WAL encryption flow with native replication and increases performance for large scale databases. -* **Default encryption key for single-tenancy**. + It now supports the Vault keyring for secure key storage. + +* **Automatic WAL key rotation**. - The new functionality allows you to set a default principal key for the entire database cluster. This key is used to encrypt all databases and tables that do not have a custom principal key set. This feature simplifies encryption configuration and management in single-tenant environments where each user has their own database instance. + A new automatic WAL key rotation occurs on server start. This ensures each server instance uses a fresh internal key for WAL encryption, improving cryptographic hygiene and reducing the risk of key reuse. -* **Ability to change key provider configuration** +* **Fixed external tablespace data loss with encrypted partitions** - You no longer need to configure a new key provider and set a new principal key if the provider's configuration changed. Now can change the key provider configuration both for the current database and the entire PostgreSQL cluster using [functions](../functions.md#key-provider-management). This enhancement lifts existing limitations and is a native and common way to operate in PostgreSQL. + An issue was fixed where data could be lost when the encrypted partitioned tables were moved to external tablespaces. -* **Key management permissions** +* **New visibility and verification functions for default principal keys** - The new functions allow you to manage permissions for global and database key management separately. This feature provides more granular control over key management operations and allows you to delegate key management tasks to different roles. + Added additional functions to help you verify and inspect the state of default principal keys more easily. -* **Additional information about principal keys and providers** +* **Restricted key provider configuration to superusers** - The new functions allow you to display additional information about principal keys and providers. This feature helps you to understand the current key configuration and troubleshoot issues related to key management. + The database owners can no longer configure key providers directly. Instead, they must refer to the superuser who manages the provider setup. This security improvement clearly separates the responsibilities between users and administrators. * **`tde_heap_basic` access method deprecation** @@ -36,7 +36,7 @@ This release provides the following features and improvements: ## Upgrade considerations -`pg_tde` Release Candidate is not backward compatible with `pg_tde` Beta2 due to significant changes in code. This means you cannot directly upgrade from one version to another. You must [uninstall](../how-to/uninstall.md) `pg_tde` Beta2 first and then [install](../install.md) and configure the new Release Candidate version. +`pg_tde` Release Candidate 2 is not backward compatible with `pg_tde` Beta2 due to significant changes in code. This means you cannot directly upgrade from one version to another. You must [uninstall](../how-to/uninstall.md) `pg_tde` Beta2 first and then [install](../install.md) and configure the new Release Candidate version. ## Known issues @@ -55,17 +55,17 @@ This release provides the following features and improvements: ### New Features -* [PG-1234](https://perconadev.atlassian.net/browse/PG-1234) - Added functions for separate global and database key management permissions. +[PG-830]() – Added support for full WAL encryption using Vault keyring, significantly improving data security in transit. -* [PG-1255](https://perconadev.atlassian.net/browse/PG-1255) - Added functionality to delete key providers. +[PG-1460] – Enabled automatic rotation of WAL internal keys on server startup for improved cryptographic hygiene. -* [PG-1256](https://perconadev.atlassian.net/browse/PG-1256) - Added single-tenant support via the default principal key functionality. +[PG-1455] – Introduced random base numbers to encryption IVs to enhance uniqueness and security. -* [PG-1258](https://perconadev.atlassian.net/browse/PG-1258) - Added functions to display additional information about principal keys / providers. +[PG-1506] – Added a new parameter for passing client certificates while registering a KMIP provider. -* [PG-1294](https://perconadev.atlassian.net/browse/PG-1294) - Redesigned WAL encryption. +[PG-1458] – Added missing information and verification functions for default principal keys. -* [PG-1303](https://perconadev.atlassian.net/browse/PG-1303) - Deprecated tde_heap_basic access method. +[PG-1304] – Removed the deprecated tde_heap_basic access method in preparation for GA. ## Improvements diff --git a/contrib/pg_tde/documentation/docs/release-notes/release-notes.md b/contrib/pg_tde/documentation/docs/release-notes/release-notes.md index e383f05c3aab9..47e4bcc9ec751 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/release-notes.md +++ b/contrib/pg_tde/documentation/docs/release-notes/release-notes.md @@ -2,6 +2,7 @@ `pg_tde` extension brings in [Transparent Data Encryption (TDE)](../index/index.md) to PostgreSQL and enables you to keep sensitive data safe and secure. +* [pg_tde Release Candidate 2 {{date.RC}}](rc2.md) * [pg_tde Release Candidate {{date.RC}}](rc.md) * [pg_tde Beta2 (2024-12-16)](beta2.md) * [pg_tde Beta (2024-06-30)](beta.md) diff --git a/contrib/pg_tde/documentation/variables.yml b/contrib/pg_tde/documentation/variables.yml index a7e36ebfc24a0..a9944a2204fc3 100644 --- a/contrib/pg_tde/documentation/variables.yml +++ b/contrib/pg_tde/documentation/variables.yml @@ -1,9 +1,9 @@ #Variables used throughout the docs -release: 'RC' -pgversion17: '17.4' +release: 'RC2' +pgversion17: '17.5.1' tdebranch: TDE_REL_17_STABLE date: - RC2: + RC2: '2025-05-14' RC: '2025-03-27' From 822548579bac20b4b41e4e03ca274fda827cd9c5 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 12 May 2025 17:17:08 +0300 Subject: [PATCH 04/10] Update rc2.md --- contrib/pg_tde/documentation/docs/release-notes/rc2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index 2d2e7821b96b2..f0a8257f7eeef 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -28,7 +28,7 @@ This release provides the following features and improvements: * **Restricted key provider configuration to superusers** - The database owners can no longer configure key providers directly. Instead, they must refer to the superuser who manages the provider setup. This security improvement clearly separates the responsibilities between users and administrators. + The database owners can no longer configure key providers directly. Instead, they must refer to the superuser who manages the provider setup. This security improvement clearly separates the responsibilities between users and administrators * **`tde_heap_basic` access method deprecation** From ddba3d41102a472cfe22051df559850cb69460db Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 12 May 2025 18:50:09 +0300 Subject: [PATCH 05/10] updated RC2 With latest notes based on RC2 jira RN. --- .../documentation/docs/release-notes/rc2.md | 148 ++++++++++-------- 1 file changed, 79 insertions(+), 69 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index f0a8257f7eeef..eb653c0700e1a 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -8,16 +8,24 @@ This release provides the following features and improvements: -* **Improved performance with redesigned WAL encryption**. +* **Restricted key provider configuration to superusers** + + The database owners can no longer configure key providers directly. Instead, they must refer to the superuser who manages the provider setup. This security improvement clearly separates the responsibilities between users and administrators. - The approach to WAL encryption has been redesigned. Now, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on. The information about what is encrypted is stored in the internal key metadata. This change improves WAL encryption flow with native replication and increases performance for large scale databases. +* **WAL encryption supports Vault**. - It now supports the Vault keyring for secure key storage. + `pg_tde` now supports using the Vault keyring for secure storage and management of WAL encryption keys. * **Automatic WAL key rotation**. A new automatic WAL key rotation occurs on server start. This ensures each server instance uses a fresh internal key for WAL encryption, improving cryptographic hygiene and reducing the risk of key reuse. +* **Proper removal of relation-level encryption keys on table drop** + + Previously, encrypted relation keys persisted even after dropping the associated tables, potentially leaving orphaned entries in the map file. This is now corrected, when an encrypted table is dropped, its corresponding key is also removed from the key map. + + You can verify the cleanup by running `hexdump -C` on the map file and confirming the related entries are zeroed out. + * **Fixed external tablespace data loss with encrypted partitions** An issue was fixed where data could be lost when the encrypted partitioned tables were moved to external tablespaces. @@ -26,13 +34,11 @@ This release provides the following features and improvements: Added additional functions to help you verify and inspect the state of default principal keys more easily. -* **Restricted key provider configuration to superusers** - - The database owners can no longer configure key providers directly. Instead, they must refer to the superuser who manages the provider setup. This security improvement clearly separates the responsibilities between users and administrators +* **Fixed SQL failures caused by inconsistent key provider switching** -* **`tde_heap_basic` access method deprecation** - - The `tde_heap_basic` access method has limitations in encryption capabilities and affects performance. Also, it poses a potential security risk when used in production environments due to indexes remaining unencrypted. Considering all the above, we decided to deprecate this access method and remove it in future releases. Use the `tde_heap` access method instead that is available with Percona Server for PostgreSQL 17 - a drop-in replacement for PostgreSQL Community. + An issue was resolved where SQL queries could fail after switching key providers while the server was running. + This occurred because principal keys became inaccessible when spread across multiple keyring backends, triggering the single-provider-at-a-time design constraint. + `pg_tde` now enforces consistency during provider changes to prevent a corrupted key state and query errors. ## Upgrade considerations @@ -49,73 +55,77 @@ This release provides the following features and improvements: Adjust the limits with caution since it affects other processes running in your system. -* You can now delete global key providers even when their associated principal key is still in use. This known issue will be fixed in the next release. For now, avoid deleting global key providers. - ## Changelog ### New Features -[PG-830]() – Added support for full WAL encryption using Vault keyring, significantly improving data security in transit. - -[PG-1460] – Enabled automatic rotation of WAL internal keys on server startup for improved cryptographic hygiene. - -[PG-1455] – Introduced random base numbers to encryption IVs to enhance uniqueness and security. - -[PG-1506] – Added a new parameter for passing client certificates while registering a KMIP provider. - -[PG-1458] – Added missing information and verification functions for default principal keys. - -[PG-1304] – Removed the deprecated tde_heap_basic access method in preparation for GA. +* [PG-813](https://perconadev.atlassian.net/browse/PG-813) – Enabled support for logical replication in `pg_tde`. +* [PG-817](https://perconadev.atlassian.net/browse/PG-817) – Added fuzz testing to `pstress` to strengthen validation and resilience. +* [PG-824](https://perconadev.atlassian.net/browse/PG-824) – Ensured fsync is called on `pg_tde.map`, `pg_tde.dat`, and FS key provider files. +* [PG-830](https://perconadev.atlassian.net/browse/PG-830) – Implemented full WAL encryption using Vault keyring. +* [PG-831](https://perconadev.atlassian.net/browse/PG-831) – Tested WAL recovery and both streaming and logical replication compatibility. +* [PG-855](https://perconadev.atlassian.net/browse/PG-855) – Added a contributor guide to help new developers engage with pg_tde. +* [PG-938](https://perconadev.atlassian.net/browse/PG-938) – Evaluated use of `pg_basebackup` for automated backup validation with pg_tde. +* [PG-962](https://perconadev.atlassian.net/browse/PG-962) – Automated test cases to validate data integrity after PostgreSQL restart. +* [PG-1001](https://perconadev.atlassian.net/browse/PG-1001) – Verified encryption behavior of temporary tables. +* [PG-1099](https://perconadev.atlassian.net/browse/PG-1099) – Developed automation for bare-metal performance benchmarking. +* [PG-1232](https://perconadev.atlassian.net/browse/PG-1232) – Documented the architectural design behind pg_tde. +* [PG-1289](https://perconadev.atlassian.net/browse/PG-1289) – Added test cases for verifying compatibility with different PostgreSQL versions. +* [PG-1295](https://perconadev.atlassian.net/browse/PG-1295) – Introduced TAP test cases to validate WAL encryption across access methods. +* [PG-1444](https://perconadev.atlassian.net/browse/PG-1444) – Implemented support for removing relation-level encryption keys when dropping tables. +* [PG-1455](https://perconadev.atlassian.net/browse/PG-1455) – Introduced random base numbers in encryption IVs for enhanced security. +* [PG-1458](https://perconadev.atlassian.net/browse/PG-1458) – Added visibility and verification functions for default principal keys. +* [PG-1460](https://perconadev.atlassian.net/browse/PG-1460) – Enabled automatic rotation of WAL internal keys on server start. +* [PG-1461](https://perconadev.atlassian.net/browse/PG-1461) – Implemented random IV initialization for WAL keys. +* [PG-1506](https://perconadev.atlassian.net/browse/PG-1506) – Added parameter support for client certificates in KMIP provider configuration. ## Improvements -* [PG-858](https://perconadev.atlassian.net/browse/PG-858) - Refactored internal/principal key LWLocks to make local databases inherit a global key provider. - -* [PG-1243](https://perconadev.atlassian.net/browse/PG-1243) - Investigated performance issues at a specific threshold and large databases and updated documentation about handling hint bits. - -* [PG-1310](https://perconadev.atlassian.net/browse/PG-1310) - Added access method enforcement via the GUC variable. - -* [PG-1361](https://perconadev.atlassian.net/browse/PG-1361) - Fixed pg_tde relocatability. - -* [PG-1380](https://perconadev.atlassian.net/browse/PG-1380) - Added support for `pg_tde_is_encrypted()` function on indexes and sequences. +* [PG-826](https://perconadev.atlassian.net/browse/PG-826) – Documented how to encrypt and decrypt existing tables using pg_tde. +* [PG-827](https://perconadev.atlassian.net/browse/PG-827) – Fixed CI pipeline tests on the smgr branch. +* [PG-834](https://perconadev.atlassian.net/browse/PG-834) – Resolved issues with `CREATE ... USING pg_tde` on the smgr branch. +* [PG-838](https://perconadev.atlassian.net/browse/PG-838) – Added documentation for setting up streaming replication with pg_tde. +* [PG-1294](https://perconadev.atlassian.net/browse/PG-1294) – Improved WAL encryption design and performance. +* [PG-1304](https://perconadev.atlassian.net/browse/PG-1304) – Removed deprecated `tde_heap_basic` access method before GA. +* [PG-1307](https://perconadev.atlassian.net/browse/PG-1307) – Assessed overall development effort required for WAL encryption. +* [PG-1392](https://perconadev.atlassian.net/browse/PG-1392) – Investigated encryption coverage gaps in freespace and visibility maps. +* [PG-1419](https://perconadev.atlassian.net/browse/PG-1419) – Changed provider validation to occur during provider addition instead of key setup. +* [PG-1427](https://perconadev.atlassian.net/browse/PG-1427) – Tested and fixed KMIP implementation for Thales support. +* [PG-1437](https://perconadev.atlassian.net/browse/PG-1437) – Cleaned up and resolved TODO items in the codebase. +* [PG-1440](https://perconadev.atlassian.net/browse/PG-1440) – Restricted key provider configuration to superusers for improved security. +* [PG-1441](https://perconadev.atlassian.net/browse/PG-1441) – Prevented replication of table-level encryption keys. +* [PG-1446](https://perconadev.atlassian.net/browse/PG-1446) – Improved locking mechanisms in pg_tde. +* [PG-1447](https://perconadev.atlassian.net/browse/PG-1447) – Verified compatibility of encryption with template databases. +* [PG-1457](https://perconadev.atlassian.net/browse/PG-1457) – Renamed key management functions for clarity and consistency. +* [PG-1467](https://perconadev.atlassian.net/browse/PG-1467) – Added Clang-based CI integration on GitHub. +* [PG-1507](https://perconadev.atlassian.net/browse/PG-1507) – Handled ALTER TYPE operations in the TDE event trigger. +* [PG-1508](https://perconadev.atlassian.net/browse/PG-1508) – Fixed encryption state inconsistencies when altering inherited tables. +* [PG-1550](https://perconadev.atlassian.net/browse/PG-1550) – Restricted database owners from creating key providers to improve security. +* [PG-1586](https://perconadev.atlassian.net/browse/PG-1586) – Verified and fixed KMIP compatibility with Fortanix HSM. ### Bugs Fixed -* [PG-821](https://perconadev.atlassian.net/browse/PG-821) - Fixed the issue with `pg_basebackup` failing when configuring replication. - -* [PG-847](https://perconadev.atlassian.net/browse/PG-847) - Fixed the issue with `pg_basebackup` and `pg_checksum` throwing an error on files created by `pg_tde` when the checksum is enabled on the database cluster. - -* [PG-1004](https://perconadev.atlassian.net/browse/PG-1004) - Fixed the issue with `pg_checksums` utility failing during checksum verification on `pg_tde` tables. Now `pg_checksum` skips encrypted relations by looking if the relation has a custom storage manager (SMGR) key. - -* [PG-1373](https://perconadev.atlassian.net/browse/PG-1373) - Fixed the issue with potential unterminated strings by using the `memcpy()` or `strlcpy()` instead of the `strncpy()` function. - -* [PG-1378](https://perconadev.atlassian.net/browse/PG-1378) - Fixed the issue with toast tables created by the `ALTER TABLE` command not being encrypted. - -* [PG-1379](https://perconadev.atlassian.net/browse/PG-1379) - Fixed sequence and alter table handling in the event trigger. - -* [PG-1222](https://perconadev.atlassian.net/browse/PG-1222) - Fixed the bug with confused relations with the same `RelFileNumber` in different databases. - -* [PG-1400](https://perconadev.atlassian.net/browse/PG-1400) - Corrected the pg_tde_change_key_provider naming in help. - -* [PG-1401](https://perconadev.atlassian.net/browse/PG-1401) - Fixed the issue with inheriting an encryption status during the ALTER TABLE SET access method command execution by basing a new encryption status only on the new encryption setting. - -* [PG-1414](https://perconadev.atlassian.net/browse/PG-1414) - Fixed the error message wording when configuring WAL encryption by referencing to a correct function. - -* [PG-1450](https://perconadev.atlassian.net/browse/PG-1450) - Fixed the `pg_tde_delete_key_provider()` function behavior when called multiple times by ignoring already deleted records. - -* [PG-1451](https://perconadev.atlassian.net/browse/PG-1451) -Fixed the issue with the repeating error message about inability to retrieve a principal key even when a user creates non-encrypted tables by checking the current transaction ID in both the event trigger start function and during a file creation. If the transaction changed during the setup of the current event trigger data, the event trigger is reset. - -* [PG-1473](https://perconadev.atlassian.net/browse/PG-1473) - Allowed only users with key viewer privileges to execute `pg_tde_verify_principal_key()` and `pg_tde_verify_global_principal_key()` functions. - -* [PG-1474](https://perconadev.atlassian.net/browse/PG-1474) - Fixed the issue with the principal key reference corruption when reassigning it to a key provider with the same name by setting the key name in vault/kmip getters. - -* [PG-1476](https://perconadev.atlassian.net/browse/PG-1476) - Fixed the issue with the server failing to start when WAL encryption is enabled by creating a new principal key for WAL in case only one default key exists in the database. - -* [PG-1479](https://perconadev.atlassian.net/browse/PG-1479), [PG-1480](https://perconadev.atlassian.net/browse/PG-1480) - Fixed the issue with the lost access to data after the global key provider change and the server restart by fixing the incorrect parameter order in default key rotation. - -* [PG-1489](https://perconadev.atlassian.net/browse/PG-1489) - Fixed the issue with replicating the keys and key provider configuration by creating the `pg_tde` directory on the replica server. -/browse/PG-1476) - Fixed the issue with the server failing to start when WAL encryption is enabled by creating a new principal key for WAL in case only one default key exists in the database. - -* [PG-1479](https://perconadev.atlassian.net/browse/PG-1479), [PG-1480](https://perconadev.atlassian.net/browse/PG-1480) - Fixed the issue with the lost access to data after the global key provider change and the server restart by fixing the incorrect parameter order in default key rotation. - -* [PG-1489](https://perconadev.atlassian.net/browse/PG-1489) - Fixed the issue with replicating the keys and key provider configuration by creating the `pg_tde` directory on the replica server. +* [PG-1397](https://perconadev.atlassian.net/browse/PG-1397) – Fixed segmentation fault during replication with WAL encryption enabled. +* [PG-1413](https://perconadev.atlassian.net/browse/PG-1413) – Resolved invalid WAL magic number errors after toggling encryption. +* [PG-1416](https://perconadev.atlassian.net/browse/PG-1416) – Fixed SQL query failures caused by inconsistent key provider switching. +* [PG-1468](https://perconadev.atlassian.net/browse/PG-1468) – Fixed WAL read failures on replicas after key rotation. +* [PG-1491](https://perconadev.atlassian.net/browse/PG-1491) – Corrected `pg_tde_is_encrypted()` behavior for partitioned tables. +* [PG-1493](https://perconadev.atlassian.net/browse/PG-1493) – Fixed data loss when encrypted partitioned tables were moved to external tablespaces. +* [PG-1503](https://perconadev.atlassian.net/browse/PG-1503) – Blocked deletion of global key providers still associated with principal keys. +* [PG-1504](https://perconadev.atlassian.net/browse/PG-1504) – Ensured correct encryption inheritance in partitioned `tde_heap` tables. +* [PG-1510](https://perconadev.atlassian.net/browse/PG-1510) – Used different keys and IVs for PostgreSQL forks to prevent conflicts. +* [PG-1530](https://perconadev.atlassian.net/browse/PG-1530) – Fixed inability to read WAL after toggling WAL encryption. +* [PG-1532](https://perconadev.atlassian.net/browse/PG-1532) – Resolved errors rewriting owned sequences when pg_tde isn't in the default schema. +* [PG-1535](https://perconadev.atlassian.net/browse/PG-1535) – Prevented server crash on calling `pg_tde_principal_key_info()`. +* [PG-1537](https://perconadev.atlassian.net/browse/PG-1537) – Fixed crash on NULL input in user-facing functions. +* [PG-1539](https://perconadev.atlassian.net/browse/PG-1539) – Handled principal key header verification errors gracefully. +* [PG-1540](https://perconadev.atlassian.net/browse/PG-1540) – Ensured sequences are assigned correct encryption status. +* [PG-1541](https://perconadev.atlassian.net/browse/PG-1541) – Resolved WAL decryption failure after key rotation. +* [PG-1543](https://perconadev.atlassian.net/browse/PG-1543) – Fixed validation error when multiple server keys exist. +* [PG-1545](https://perconadev.atlassian.net/browse/PG-1545) – Resolved error from `pg_tde_grant_grant_management_to_role()` execution. +* [PG-1546](https://perconadev.atlassian.net/browse/PG-1546) – Fixed incorrect behavior in role grant function. +* [PG-1551](https://perconadev.atlassian.net/browse/PG-1551) – Improved handling of short reads and errors in WAL storage code. +* [PG-1571](https://perconadev.atlassian.net/browse/PG-1571) – Fixed WAL decryption failure due to corrupted or mismatched principal keys. +* [PG-1573](https://perconadev.atlassian.net/browse/PG-1573) – Prevented crash during WAL replay when lock was not held. +* [PG-1574](https://perconadev.atlassian.net/browse/PG-1574) – Ensured encrypted WAL is readable by streaming replica. +* [PG-1576](https://perconadev.atlassian.net/browse/PG-1576) – Resolved crash from malformed JSON in user-facing functions. From d1105af1368f882f64e2d13fd426b449504843b1 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Tue, 13 May 2025 09:46:46 +0300 Subject: [PATCH 06/10] updated Automatic WAL key rotation Improved wording as new internal key is generated after server start --- contrib/pg_tde/documentation/docs/release-notes/rc2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index eb653c0700e1a..a0bfb5689d70e 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -16,9 +16,9 @@ This release provides the following features and improvements: `pg_tde` now supports using the Vault keyring for secure storage and management of WAL encryption keys. -* **Automatic WAL key rotation**. +* **Automatic WAL internal key generation at server startup**. - A new automatic WAL key rotation occurs on server start. This ensures each server instance uses a fresh internal key for WAL encryption, improving cryptographic hygiene and reducing the risk of key reuse. + On each server start, a new internal key is generated for encrypting subsequent WAL records (assuming WAL encryption is enabled). The existing WAL records and their keys remain unchanged, this ensures continuity and secure key management without affecting historical data. * **Proper removal of relation-level encryption keys on table drop** From 3d22c01d71b55395387d5603079da1f1508e6057 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Tue, 13 May 2025 09:50:01 +0300 Subject: [PATCH 07/10] Updated **Additional information about principal keys and providers** Removed last sentence that was for internal use. --- contrib/pg_tde/documentation/docs/release-notes/rc2.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index a0bfb5689d70e..2d3159053466b 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -24,8 +24,6 @@ This release provides the following features and improvements: Previously, encrypted relation keys persisted even after dropping the associated tables, potentially leaving orphaned entries in the map file. This is now corrected, when an encrypted table is dropped, its corresponding key is also removed from the key map. - You can verify the cleanup by running `hexdump -C` on the map file and confirming the related entries are zeroed out. - * **Fixed external tablespace data loss with encrypted partitions** An issue was fixed where data could be lost when the encrypted partitioned tables were moved to external tablespaces. From c8f1fbe80e0d605319ec5760f5f802d6f13653e7 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 19 May 2025 11:02:23 +0300 Subject: [PATCH 08/10] Update rc2.md --- contrib/pg_tde/documentation/docs/release-notes/rc2.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/release-notes/rc2.md b/contrib/pg_tde/documentation/docs/release-notes/rc2.md index 2d3159053466b..d1f736a30278a 100644 --- a/contrib/pg_tde/documentation/docs/release-notes/rc2.md +++ b/contrib/pg_tde/documentation/docs/release-notes/rc2.md @@ -2,7 +2,7 @@ `pg_tde` extension brings in [Transparent Data Encryption (TDE)](../index/index.md) to PostgreSQL and enables you to keep sensitive data safe and secure. -[Get started](../install.md){.md-button} +[Get Started](../install.md){.md-button} ## Release Highlights @@ -46,12 +46,12 @@ This release provides the following features and improvements: * The default `mlock` limit on Rocky Linux 8 for ARM64-based architectures equals the memory page size and is 64 Kb. This results in the child process with `pg_tde` failing to allocate another memory page because the max memory limit is reached by the parent process. - To prevent this, you can change the `mlock` limit to be at least twice bigger than the memory page size: +To prevent this, you can change the `mlock` limit to be at least twice bigger than the memory page size: - * temporarily for the current session using the `ulimit -l ` command. - * set a new hard limit in the `/etc/security/limits.conf` file. To do so, you require the superuser privileges. +* temporarily for the current session using the `ulimit -l ` command. +* set a new hard limit in the `/etc/security/limits.conf` file. To do so, you require the superuser privileges. - Adjust the limits with caution since it affects other processes running in your system. +Adjust the limits with caution since it affects other processes running in your system. ## Changelog From 7939387e91033f96c63d9fa358ec364eab284647 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 19 May 2025 11:05:12 +0300 Subject: [PATCH 09/10] Update variables.yml --- contrib/pg_tde/documentation/variables.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/pg_tde/documentation/variables.yml b/contrib/pg_tde/documentation/variables.yml index a9944a2204fc3..365412e50b1a3 100644 --- a/contrib/pg_tde/documentation/variables.yml +++ b/contrib/pg_tde/documentation/variables.yml @@ -5,5 +5,5 @@ pgversion17: '17.5.1' tdebranch: TDE_REL_17_STABLE date: - RC2: '2025-05-14' + RC2: '2025-05-21' RC: '2025-03-27' From 9f4005df0cd59b7cc776469f2a73baabc94a1752 Mon Sep 17 00:00:00 2001 From: Dragos Andriciuc Date: Mon, 19 May 2025 14:54:00 +0300 Subject: [PATCH 10/10] Index and functions updates Updated based on Mohit's feedback --- .../documentation/docs/architecture/index.md | 38 +++++++------------ .../pg_tde/documentation/docs/functions.md | 7 +++- 2 files changed, 20 insertions(+), 25 deletions(-) diff --git a/contrib/pg_tde/documentation/docs/architecture/index.md b/contrib/pg_tde/documentation/docs/architecture/index.md index d5ece3f2a8d17..6cf02d416e668 100644 --- a/contrib/pg_tde/documentation/docs/architecture/index.md +++ b/contrib/pg_tde/documentation/docs/architecture/index.md @@ -21,8 +21,6 @@ Let's break down what it means. * Sequences * Temporary tables * Write Ahead Log (WAL) -* System tables (not yet implemented) -* Temporary files (not yet implemented) **Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL. @@ -273,7 +271,8 @@ These functions return a list of provider names, type and configuration. For keys and providers administration, it provides two pair of functions: ```sql -pg_tde_(grant/revoke)_database_key_management_to_role +pg_tde_GRANT_database_key_management_TO_role +pg_tde_REVOKE_database_key_management_FROM_role ``` ### Creating and rotating keys @@ -297,20 +296,13 @@ With this feature, it is possible for the entire database server to easily use t A default key can be managed with the following functions: ```sql -pg_tde_set_default_key('key-name', 'provider-name', ensure_new_key) -pg_tde_drop_default_key() -- not yet implemented +pg_tde_set_default_key_using_global_key_provider('key-name', 'provider-name', 'true/false') ``` `DROP` is only possible if there's no table currently using the default principal key. Changing the default principal key will rotate the encryption of internal keys for all databases using the current default principal key. -### Removing key (not yet implemented) - -`pg_tde_drop_key` removes the principal key for the current database. If the current database has any encrypted tables, and there isn't a default principal key configured, it reports an error instead. If there are encrypted tables, but there's also a global default principal key, internal keys will be encrypted with the default key. - -It isn't possible to remove the WAL (server) principal key. - ### Current key details `pg_tde_key_info()` returns the name of the current principal key, and the provider it uses. @@ -321,23 +313,20 @@ It isn't possible to remove the WAL (server) principal key. `pg_tde_verify_key()` checks that the key provider is accessible, that the current principal key can be downloaded from it, and that it is the same as the current key stored in memory - if any of these fail, it reports an appropriate error. -### Listing all active keys (not yet implemented) - -SUPERusers are able to use the following function: - -`pg_tde_list_active_keys()` - -Which reports all the actively used keys by all databases on the current server. Similarly to `pg_tde_key_info()`, it only shows names and associated providers, it doesn't reveal any sensitive information about the providers. - ### Key permissions -Users with management permissions to a specific database `(pg_tde_(grant/revoke)_(global/databse)_key_management_to_role)` can change the keys for the database, and use the current key functions. This includes creating keys using global providers, if `pg_tde.inherit_global_providers` is enabled. +Users with management permissions to a specific database `(pg_tde_(grant/revoke)_(global/databse)_key_management_(to/from)_role)` can change the keys for the database, and use the current key functions. This includes creating keys using global providers, if `pg_tde.inherit_global_providers` is enabled. Also the `pg_tde_(grant/revoke)_database_key_management_to_role` function deals with only the specific permission for the above function: it allows a user to change the key for the database, but not to modify the provider configuration. ### Creating encrypted tables -To create an encrypted table or modify an existing table to be encrypted, simply use `USING tde_heap` in the `CREATE` statement. +To create an encrypted table or modify an existing table to be encrypted, use the following commands: + +```sql +CREATE TABLE t1(a INT) USING tde_heap; +ALTER TABLE t1 SET ACCESS METHOD tde_heap; +``` ### Changing the `pg_tde.inherit_global_keys` setting @@ -349,12 +338,13 @@ In this case existing references to global providers, or the global default prin ### Simple "one principal key" encryption -1. Installing the extension: `shared_preload_libraries` + `pg_tde.wal_encrypt` +1. Passing the option from the postgres config file the extension: `shared_preload_libraries=‘pg_tde’` 2. `CREATE EXTENSION pg_tde;` in `template1` 3. Adding a global key provider 4. Adding a default principal key using the same global provider -5. Changing the WAL encryption to use the default principal key -6. Optionally: setting the `default_table_access_method` to `tde_heap` so that tables are encrypted by default +5. Enable WAL encryption to use the default principal key using `ALTER SYSTEM SET pg_tde.wal_encrypt=‘ON’` +6. Restart the server +7. Optionally: setting the `default_table_access_method` to `tde_heap` so that tables are encrypted by default Database users don't need permissions to any of the encryption functions: encryption is managed by the admins, normal users only have to create tables with encryption, which requires no specific permissions. diff --git a/contrib/pg_tde/documentation/docs/functions.md b/contrib/pg_tde/documentation/docs/functions.md index f7bc4647b8990..0f5a69a67787a 100644 --- a/contrib/pg_tde/documentation/docs/functions.md +++ b/contrib/pg_tde/documentation/docs/functions.md @@ -4,7 +4,12 @@ The `pg_tde` extension provides functions for managing different aspects of its ## Permission management -By default, `pg_tde` is restrictive. It doesn't allow any operations until permissions are granted to the user. Only superusers can create or modify to key providers or modify objects in the global scope. Functions for viewing keys and for setting the principal key in a database local key provider can on the other hand be run by the database owner and be delegated to normal users using the `GRANT EXECUTE` and `REVOKE EXECUTE` commands. +By default, `pg_tde` is locked down. No one is allowed to do any operations until you grant them permissions. Only superusers may add or alter global key providers. + +However, database owners can run the “view keys” and “set principal key” functions on their own databases. You can delegate these rights to other roles with the following commands: + +* `GRANT EXECUTE` +* `REVOKE EXECUTE` The following functions are also provided for easier management of functionality groups: