8000 Tde documentation updates for ga by Andriciuc · Pull Request #251 · percona/postgres · GitHub
[go: up one dir, main page]

Skip to content

Tde documentation updates for ga #251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
May 8, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
fdd464b
Added and reorg files
Andriciuc Apr 17, 2025
f3d8379
Split Home and created TDE Overview
Andriciuc Apr 17, 2025
e2d4be2
Further refined and split into topics
Andriciuc Apr 22, 2025
f5db064
Renamed files and folders for better SEO
Andriciuc Apr 23, 2025
4869f64
Merge remote-tracking branch 'origin/pg_tde_docs_updates_for_GA' into…
Andriciuc Apr 23, 2025
ed361e7
updated with Anders branch
Andriciuc Apr 24, 2025
b5c9e89
Small updates to yml and apt
Andriciuc Apr 28, 2025
d5a0629
updated index folder to reflect chapter name, overview
Andriciuc Apr 28, 2025
ef82b1f
updated overview doc to index again
Andriciuc Apr 28, 2025
cbd4017
updated links to reflect proper structure
Andriciuc Apr 28, 2025
778851b
Setup.md and structure changes
Andriciuc Apr 28, 2025
1e3c160
A handful of fixes
Andriciuc May 1, 2025
043b44e
removed PG Community mentions
Andriciuc May 1, 2025
0876d20
updated cli and kmip server code
Andriciuc May 5, 2025
106856d
small fixes
Andriciuc May 5, 2025
6dc7d79
Update functions.md
Andriciuc May 5, 2025
9b8d6c2
updated toc
Andriciuc May 5, 2025
9302f0f
small fixes to buttons added
Andriciuc May 5, 2025
7d4b514
smaller updates
Andriciuc May 6, 2025
6f1c3b8
smaller updates
Andriciuc May 6, 2025
12230fe
Update requirements.txt
Andriciuc May 6, 2025
31c0975
fixes part 1/2 from feedback review
Andriciuc May 6, 2025
dce23bd
updated features.md
Andriciuc May 7, 2025
1e61a45
removed compliance/regulations
Andriciuc May 7, 2025
f636e82
fix for PG-1587
Andriciuc May 7, 2025
45b470e
fixes for nastia 1/2 and for jz 2/2
Andriciuc May 7, 2025
104710e
all fixes accounted for
Andriciuc May 7, 2025
a39e568
Resolve merge conflicts
Andriciuc May 7, 2025
31bea87
fixes to titles
Andriciuc May 8, 2025
be6e6ef
fixed notes in features
Andriciuc May 8, 2025
7b76758
updated function parameters
Andriciuc May 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions contrib/pg_tde/documentation/docs/features.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# Features

We provide `pg_tde` in two versions for both PostgreSQL Community and [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/). The difference between the versions is in the set of included features which in its turn depends on the Storage Manager API. While PostgreSQL Community uses the default Storage Manager API, Percona Server for PostgreSQL extends the Storage Manager API enabling to integrate custom storage managers.
`pg_tde` is available in two variants (or "flavors"):

The following table provides features available for each version:
* PostgreSQL Community
* [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/)
The key difference between these variants lies in the set of supported features, which depends on the underlying Storage Manager API. The PostgreSQL Community variant uses the default Storage Manager API, while the Percona Server for PostgreSQL provides an extended Storage Manager API that allows integration with custom storage managers.

The following table presents the features available for each variant:

| Percona Server for PostgreSQL version | PostgreSQL Community version (deprecated) |
|-------------------------------|----------------------|
Expand All @@ -14,7 +18,4 @@ The following table provides features available for each version:
| Key management via: <br> - HashiCorp Vault; <br> - KMIP server; <br> - Local keyfile | Key management via: <br> - HashiCorp Vault; <br> - Local keyfile |
| Logical replication of encrypted tables | |




[Get started](install.md){.md-button}
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,11 @@ Check that the table is not encrypted:
SELECT pg_tde_is_encrypted('mytable');
```

The output returns `f` meaning that the table is no longer encrypted.


The output returns `f` meaning that the table is no longer encrypted.

## Method 2. Create a new not encrypted table on the base of the encrypted one

Alternatively, you can create a new not encrypted table with the same structure and data as the initial table. For example, the original encrypted table is `EncryptedCustomers`. Use the following command to create a new table `Customers`:
Alternatively, you can create a new not encrypted table with the same structure and data as the initial table. For example, the original encrypted table is `EncryptedCustomers`. Use the following command to create a new table `Customers`:

```
CREATE TABLE Customers AS
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ Here's how to do it:
DROP EXTENSION pg_tde;
```

This command will fail if there are still encrypted tables in the database.
This command will fail if there are still encrypted tables in the database.

In this case, you must drop the dependent objects manually. Alternatively, you can run the `DROP EXTENSION ... CASCADE` command to drop all dependent objects automatically.
In this case, you must drop the dependent objects manually. Alternatively, you can run the `DROP EXTENSION ... CASCADE` command to drop all dependent objects automatically.

Note that the `DROP EXTENSION` command does not delete the `pg_tde` data files related to the database.

Expand All @@ -22,20 +22,20 @@ Here's how to do it:

4. Modify the `shared_preload_libraries` and remove the 'pg_tde' from it. Use the `ALTER SYSTEM` command for this purpose, or edit the configuration file.

!!! warning
!!! warning

Once `pg_tde` is removed from the `shared_preload_libraries`, reading any leftover encrypted files will fail. Removing the extension from the `shared_preload_libraries` is also possible if the extension is still installed in some databases.

Make sure to do this only if the server has no encrypted files in its data directory.

5. Start or restart the `postgresql` cluster to apply the changes.

* On Debian and Ubuntu:
* On Debian and Ubuntu:

```sh
sudo systemctl restart postgresql
```

* On RHEL and derivatives

```sh
Expand Down
48 changes: 5 additions & 43 deletions contrib/pg_tde/documentation/docs/index.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,13 @@
# `pg_tde` documentation

`pg_tde` is the open source PostgreSQL extension that provides Transparent Data Encryption (TDE) to protect data at rest. This ensures that the data stored on disk is encrypted, and no one can read it without the proper encryption keys, even if they gain access to the physical storage media.
`pg_tde` is the open source, community driven and futureproof PostgreSQL extension that provides Transparent Data Encryption (TDE) to protect data at rest. `pg_tde` ensures that the data stored on disk is encrypted, and that no one can read it without the proper encryption keys, even if they gain access to the physical storage media.

Learn more [what Transparent Data Encryption is](tde.md#how-does-it-work) and [why you need it](tde.md#why-do-you-need-tde).
Learn more about [what Transparent Data Encryption is](tde.md#how-does-it-work) and [why do you need it](tde.md#why-do-you-need-tde).

!!! important

This is the {{release}} version of the extension and it is not meant for production use yet. We encourage you to use it in testing environments and [provide your feedback](https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82).

[Get started](install.md){.md-button}
[What's new in pg_tde {{release}}](release-notes/release-notes.md){.md-button}

## What's encrypted:

* User data in tables, including TOAST tables, that are created using the extension. Metadata of those tables is not encrypted.
* Temporary tables created during the database operation for data tables created using the extension
* Write-Ahead Log (WAL) data for the entire database cluster. This includes WAL data in encrypted and non-encrypted tables
* Indexes on encrypted tables
* Logical replication on encrypted tables

[Check the full feature list](features.md){.md-button}

## Known limitations

* Keys in the local keyfile are stored unencrypted. For better security we recommend using the Key management storage.
* System tables are currently not encrypted. This means that statistics data and database metadata are currently not encrypted.

* `pg_rewind` doesn't work with encrypted WAL for now. We plan to fix it in future releases.
* `pb_tde` Release candidate is incompatible with `pg_tde`Beta2 due to significant changes in code. There is no direct upgrade flow from one version to another. You must [uninstall](uninstall.md) `pg_tde` Beta2 first and then [install](install.md) and configure the new Release Candidate version.



## Versions and supported PostgreSQL deployments

The `pg_tde` extension comes in two distinct versions with specific access methods to encrypt the data. These versions are database-specific and differ in terms of what they encrypt and with what access method. Each version is characterized by the database it supports, the access method it provides, and the scope of encryption it offers.

* **Version for Percona Server for PostgreSQL**

This `pg_tde` version is based on and supported for [Percona Server for PostgreSQL 17.x :octicons-link-external-16:](https://docs.percona.com/postgresql/17/postgresql-server.html) - an open source binary drop-in replacement for PostgreSQL Community. It provides the `tde_heap` access method and offers [full encryption capabilities](features.md).

* **Community version** (deprecated)

This version is available with PostgreSQL Community 16 and 17, and Percona Distribution for PostgreSQL 16. It provides the `tde_heap_basic` access method, offering limited encryption features. The limitations are in encrypting WAL data only for tables created using the extension and no support of index encryption nor logical replication.

### Which version to choose?

Enjoy full encryption with the Percona Server for PostgreSQL version and the `tde_heap` access method. The Community version is deprecated and is planned to be removed in future releases.

Still not sure? [Contact our experts](https://www.percona.com/about/contact) to find the best solution for you.

[TDE Overview](tde_encrypts.md){.md-button}
[Get started](install F438 .md){.md-button}
[What's new in pg_tde {{release}}](release-notes/release-notes.md){.md-button}
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# What is Transparent Data Encryption (TDE)

Transparent Data Encryption is a technology to protect data at rest. The encryption process happens transparently in the background, without affecting database operations. Data is automatically encrypted as it's written to the disk and decrypted as it's read, all in real-time. Users and applications interact with the data as usual without noticing any difference.

## How does it work?
# How does TDE work?

To encrypt the data, two types of keys are used:

Expand All @@ -23,21 +19,3 @@ When a user creates an encrypted table using `pg_tde`, a new random key is gener
The internal key itself is encrypted using the principal key. The principal key is stored externally in the key management store.

Similarly when the user queries the encrypted table, the principal key is retrieved from the key store to decrypt the internal key. Then the same unique internal key for that table is used to decrypt the data, and unencrypted data gets returned to the user. So, effectively, every TDE table has a unique key, and each table key is encrypted using the principal key.

## Why do you need TDE?

Using TDE has the following benefits:

* For organizations:

- Ensure data safety when it is stored on disk and in backups
- Comply with security and legal standards like HIPAA, PCI DSS, SOC 2, ISO 27001

* For DBAs:

- Granular encryption of specific tables and reducing the performance overhead that encryption brings
- Additional layer of security to existing security measures like storage-level encryption, data encryption in transit using TLS, access control and more.

!!! admonition "See also"

Percona Blog: [Transparent Data Encryption (TDE)](https://www.percona.com/blog/transparent-data-encryption-tde/)
18 changes: 18 additions & 0 deletions contrib/pg_tde/documentation/docs/index/how-tde-helps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# How does TDE help?

Using TDE provides the following benefits:

* For organizations:

- Ensure data safety when it is stored on disk and in backups
- Comply with security and legal standards like HIPAA, PCI DSS, SOC 2, ISO 27001

* For DBAs:

- Granular encryption of specific tables and reducing the performance overhead that encryption brings
- Additional layer of security to existing security measures like storage-level encryption, data encryption in transit using TLS, access control and more.

!!! admonition "See also"

Percona Blog: [Transparent Data Encryption (TDE)](https://www.percona.com/blog/transparent-data-encryption-tde/)

17 changes: 17 additions & 0 deletions contrib/pg_tde/documentation/docs/index/supported-versions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Versions and supported PostgreSQL deployments

The `pg_tde` extension comes in two distinct versions with specific access methods to encrypt the data. These versions are database-specific and differ in terms of what they encrypt and with what access method. Each version is characterized by the database it supports, the access method it provides, and the scope of encryption it offers.

* **Version for Percona Server for PostgreSQL**

This `pg_tde` version is based on and supported for [Percona Server for PostgreSQL 17.x :octicons-link-external-16:](https://docs.percona.com/postgresql/17/postgresql-server.html) - an open source binary drop-in replacement for PostgreSQL Community. It provides the `tde_heap` access method and offers [full encryption capabilities](features.md).

* **Community version** (deprecated)

This version is available with PostgreSQL Community 16 and 17, and Percona Distribution for PostgreSQL 16. It provides the `tde_heap_basic` access method, offering limited encryption features. The limitations are in encrypting WAL data only for tables created using the extension and no support of index encryption nor logical replication.

## Which version to choose?

Enjoy full encryption with the Percona Server for PostgreSQL version and the `tde_heap` access method. The Community version is deprecated and is planned to be removed in future releases.

Still not sure? [Contact our experts](https://www.percona.com/about/contact) to find the best solution for you.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Table access method
# How TDE uses table access methods

A table access method is the way how PostgreSQL stores the data in a table. The default table access method is `heap`. PostgreSQL organizes data in a heap structure, meaning there is no particular order to the rows in the table. Each row is stored independently and identified by its unique row identifier (TID).

Expand Down Expand Up @@ -30,28 +30,27 @@ CREATE TABLE table_name (

### How `tde_heap` works

The `tde_heap` access method works on top of the default `heap` access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension.
The `tde_heap` access method works on top of the default `heap` access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension.

Every data modification operation is first sent to the Buffer Manager, which updates the buffer cache. Then, it is passed to the storage manager, which then writes it to disk. When a table requires encryption, the data is sent to the TDE storage manager, where it is encrypted before written to disk.
Every data modification operation is first sent to the Buffer Manager, which updates the buffer cache. Then, it is passed to the storage manager, which then writes it to disk. When a table requires encryption, the data is sent to the TDE storage manager, where it is encrypted before written to disk.

Similarly, when a client queries the database, the PostgreSQL core sends the request to the Buffer Manager which checks if the requested data is already in the buffer cache. If it’s not there, the Buffer Manager requests the data from the storage manager. The TDE storage manager reads the encrypted data from disk, decrypts it and loads it to the buffer cache. The Buffer Manager sends the requested data to the PostgreSQL core and then to the client.
Similarly, when a client queries the database, the PostgreSQL core sends the request to the Buffer Manager which checks if the requested data is already in the buffer cache. If it’s not there, the Buffer Manager requests the data from the storage manager. The TDE storage manager reads the encrypted data from disk, decrypts it and loads it to the buffer cache. The Buffer Manager sends the requested data to the PostgreSQL core and then to the client.


Thus, the encryption is done at the storage manager level.
Thus, the encryption is done at the storage manager level.

## Changing the default table access method

You can change the default table access method so that every table in the entire database cluster is created using the custom access method. For example, you can enable data encryption by default by defining the `tde_heap` as the default table access method.
You can change the default table access method so that every table in the entire database cluster is created using the custom access method. For example, you can enable data encryption by default by defining the `tde_heap` as the default table access method.

However, consider the following before making this change:

* This is a global setting and applies across the entire database cluster and not just a single database.
We recommend setting it with caution because all tables and materialized views created without an explicit access method in their `CREATE` statement will default to the specified table access method.
* This is a global setting and applies across the entire database cluster and not just a single database.
We recommend setting it with caution because all tables and materialized views created without an explicit access method in their `CREATE` statement will default to the specified table access method.
* You must create the `pg_tde` extension and configure the key provider for all databases before you modify the configuration. Otherwise PostgreSQL won't find the specified access method and will throw an error.

Here's how you can set the new default table access method:
Here is how you can set the new default table access method:

1. Add the access method to the `default_table_access_method` parameter.
1. Add the access method to the `default_table_access_method` parameter:

=== "via the SQL statement"

Expand Down Expand Up @@ -88,4 +87,3 @@ Here's how you can set the new default table access method:
```sql
SELECT pg_reload_conf();
```

11 changes: 11 additions & 0 deletions contrib/pg_tde/documentation/docs/index/tde-encrypts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# What does TDE encrypt?

`pg_tde` encrypts the following:

* User data in tables, including TOAST tables, that are created using the extension. The metadata of those tables is not encrypted.
* Temporary tables created during the database operation for data tables created using the extension
* Write-Ahead Log (WAL) data for the entire database cluster. This includes WAL data in encrypted and non-encrypted tables
* Indexes on encrypted tables
* Logical replication on encrypted tables

[Check the full feature list](features.md){.md-button}
7 changes: 7 additions & 0 deletions contrib/pg_tde/documentation/docs/index/tde-limitations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Known limitations

* Keys in the local keyfile are stored unencrypted. For better security we recommend using the Key management storage.
* System tables are currently not encrypted. This means that statistics data and database metadata are currently not encrypted.

* `pg_rewind` doesn't work with encrypted WAL for now. We plan to fix it in future releases.
* `pb_tde` Release candidate is incompatible with `pg_tde`Beta2 due to significant changes in code. There is no direct upgrade flow from one version to another. You must [uninstall](how-to/uninstall.md) `pg_tde` Beta2 first and then [install](install.md) and configure the new Release Candidate version.
3 changes: 3 additions & 0 deletions contrib/pg_tde/documentation/docs/index/what-is-tde.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# What is Transparent Data Encryption (TDE)?

Transparent Data Encryption is a technology to protect data at rest. The encryption process happens transparently in the background, without affecting database operations. Data is automatically encrypted as it's written to the disk and decrypted as it's read, all in real-time. Users and applications interact with the data as usual without noticing any difference.
1 change: 0 additions & 1 deletion contrib/pg_tde/documentation/docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ Install `pg_tde` using one of the available installation methods:

[Install from tarballs :material-arrow-right:](https://docs.percona.com/postgresql/17/tarball.html){.md-button}


## Next steps

[Setup :material-arrow-right:](setup.md){.md-button}
4 changes: 4 additions & 0 deletions contrib/pg_tde/documentation/docs/replication.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Replication

<!-- To add information on Replication affecting pg_tde for customers and how it interacts with it
Also, don't forget to add it to TOC>
Loading
0