8000 Add function to get memory context stats for processes · postgres/postgres@042a662 · GitHub
[go: up one dir, main page]
Skip to content

Commit 042a662

Browse files
danielgustafssondanielgustafsson
committed
Add function to get memory context stats for processes
This adds a function for retrieving memory context statistics and information from backends as well as auxiliary processes. The intended usecase is cluster debugging when under memory pressure or unanticipated memory usage characteristics. When calling the function it sends a signal to the specified process to submit statistics regarding its memory contexts into dynamic shared memory. Each memory context is returned in detail, followed by a cumulative total in case the number of contexts exceed the max allocated amount of shared memory. Each process is limited to use at most 1Mb memory for this. A summary can also be explicitly requested by the user, this will return the TopMemoryContext and a cumulative total of all lower contexts. In order to not block on busy processes the caller specifies the number of seconds during which to retry before timing out. In the case where no statistics are published within the set timeout, the last known statistics are returned, or NULL if no previously published statistics exist. This allows dash- board type queries to continually publish even if the target process is temporarily congested. Context records contain a timestamp to indicate when they were submitted. Author: Rahila Syed <rahilasyed90@gmail.com> Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Tomas Vondra <tomas@vondra.me> Reviewed-by: Atsushi Torikoshi <torikoshia@oss.nttdata.com> Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com> Reviewed-by: Alexander Korotkov <aekorotkov@gmail.com> Discussion: https://postgr.es/m/CAH2L28v8mc9HDt8QoSJ8TRmKau_8FM_HKS41NeO9-6ZAkuZKXw@mail.gmail.com
1 parent 15f0cb2 commit 042a662

File tree

26 files changed

+1385
-45
lines changed

26 files changed

+1385
-45
lines changed

‎doc/src/sgml/func.sgml

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28663,6 +28663,144 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
2866328663
</para></entry>
2866428664
</row>
2866528665

28666+
<row>
28667+
<entry role="func_table_entry"><para role="func_signature">
28668+
<indexterm>
28669+
<primary>pg_get_process_memory_contexts</primary>
28670+
</indexterm>
28671+
<function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
28672+
<returnvalue>setof record</returnvalue>
28673+
( <parameter>name</parameter> <type>text</type>,
28674+
<parameter>ident</parameter> <type>text</type>,
28675+
<parameter>type</parameter> <type>text</type>,
28676+
<parameter>path</parameter> <type>integer[]</type>,
28677+
<parameter>level</parameter> <type>integer</type>,
28678+
<parameter>total_bytes</parameter> <type>bigint</type>,
28679+
<parameter>total_nblocks</parameter> <type>bigint</type>,
28680+
<parameter>free_bytes</parameter> <type>bigint</type>,
28681+
<parameter>free_chunks</parameter> <type>bigint</type>,
28682+
<parameter>used_bytes</parameter> <type>bigint</type>,
28683+
<parameter>num_agg_contexts</parameter> <type>integer</type>,
28684+
<parameter>stats_timestamp</parameter> <type>timestamptz</type> )
28685+
</para>
28686+
<para>
28687+
This function handles requests to display the memory contexts of a
28688+
<productname>PostgreSQL</productname> process with the specified
28689+
process ID. The function can be used to send requests to backends as
28690+
well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
28691+
</para>
28692+
<para>
28693+
The returned record contains extended statistics per each memory
28694+
context:
28695+
<itemizedlist spacing="compact">
28696+
<listitem>
28697+
<para>
28698+
<parameter>name</parameter> - The name of the memory context.
28699+
</para>
28700+
</listitem>
28701+
<listitem>
28702+
<para>
28703+
<parameter>ident</parameter> - Memory context ID (if any).
28704+
</para>
28705+
</listitem>
28706+
<listitem>
28707+
& F438 lt;para>
28708+
<parameter>type</parameter> - The type of memory context, possible
28709+
values are: AllocSet, Generation, Slab and Bump.
28710+
</para>
28711+
</listitem>
28712+
<listitem>
28713+
<para>
28714+
<parameter>path</parameter> - Memory contexts are organized in a
28715+
tree model with TopMemoryContext as the root, and all other memory
28716+
contexts as nodes in the tree. The <parameter>path</parameter>
28717+
displays the path from the root to the current memory context. The
28718+
path is limited to 100 children per node, which each node limited
28719+
to a max depth of 100, to preserve memory during reporting. The
28720+
printed path will also be limited to 100 nodes counting from the
28721+
TopMemoryContext.
28722+
</para>
28723+
</listitem>
28724+
<listitem>
28725+
<para>
28726+
<parameter>level</parameter> - The level in the tree of the current
28727+
memory context.
28728+
</para>
28729+
</listitem>
28730+
<listitem>
28731+
<para>
28732+
<parameter>total_bytes</parameter> - The total number of bytes
28733+
allocated to this memory context.
28734+
</para>
28735+
</listitem>
28736+
<listitem>
28737+
<para>
28738+
<parameter>total_nblocks</parameter> - The total number of blocks
28739+
used for the allocated memory.
28740+
</para>
28741+
</listitem>
28742+
<listitem>
28743+
<para>
28744+
<parameter>free_bytes</parameter> - The amount of free memory in
28745+
this memory context.
28746+
</para>
28747+
</listitem>
28748+
<listitem>
28749+
<para>
28750+
<parameter>free_chunks</parameter> - The number of chunks that
28751+
<parameter>free_bytes</parameter> corresponds to.
28752+
</para>
28753+
</listitem>
28754+
<listitem>
28755+
<para>
28756+
<parameter>used_bytes</parameter> - The total number of bytes
28757+
currently occupied.
28758+
</para>
28759+
</listitem>
28760+
<listitem>
28761+
<para>
28762+
<parameter>num_agg_contexts</parameter> - The number of memory
28763+
contexts aggregated in the displayed statistics.
28764+
</para>
28765+
</listitem>
28766+
<listitem>
28767+
<para>
28768+
<parameter>stats_timestamp</parameter> - When the statistics were
28769+
extracted from the process.
28770+
</para>
28771+
</listitem>
28772+
</itemizedlist>
28773+
</para>
28774+
<para>
28775+
When <parameter>summary</parameter> is <literal>true</literal>, statistics
28776+
for memory contexts at levels 1 and 2 are displayed, with level 1
28777+
representing the root node (i.e., <literal>TopMemoryContext</literal>).
28778+
Statistics for contexts on level 2 and below are aggregates of all
28779+
child contexts' statistics, where <literal>num_agg_contexts</literal>
28780+
indicate the number aggregated child contexts. When
28781+
<parameter>summary</parameter> is <literal>false</literal>,
28782+
<literal>the num_agg_contexts</literal> value is <literal>1</literal>,
28783+
indicating that individual statistics are being displayed. The levels
28784+
are limited to the first 100 contexts.
28785+
</para>
28786+
<para>
28787+
Busy processes can delay reporting memory context statistics,
28788+
<parameter>timeout</parameter> specifies the number of seconds
28789+
to wait for updated statistics. <parameter>timeout</parameter> can be
28790+
specified in fractions of a second.
28791+
</para>
28792+
<para>
28793+
After receiving memory context statistics from the target process, it
28794+
returns the results as one row per context. If all the contexts don't
28795+
fit within the pre-determined size limit, the remaining context
28796+
statistics are aggregated and a cumulative total is displayed. The
28797+
<literal>num_agg_contexts</literal> column indicates the number of
28798+
contexts aggregated in the displayed statistics. When
28799+
<literal>num_agg_contexts</literal> is <literal>1</literal> is means
28800+
that the context statistics are displayed separately.
28801+
</para></entry>
28802+
</row>
28803+
2866628804
<row>
2866728805
<entry role="func_table_entry"><para role="func_signature">
2866828806
<indexterm>
@@ -28802,6 +28940,40 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
2880228940
because it may generate a large number of log messages.
2880328941
</para>
2880428942

28943+
<para>
28944+
<function>pg_get_process_memory_contexts</function> can be used to request
28945+
memory contexts statistics of any <productname>PostgreSQL</productname>
28946+
process. For example:
28947+
<programlisting>
28948+
postgres=# SELECT * FROM pg_get_process_memory_contexts(
28949+
(SELECT pid FROM pg_stat_activity
28950+
WHERE backend_type = 'checkpointer'),
28951+
false, 0.5) LIMIT 1;
28952+
-[ RECORD 1 ]----+------------------------------
28953+
name | TopMemoryContext
28954+
ident |
28955+
type | AllocSet
28956+
path | {1}
28957+
level | 1
28958+
total_bytes | 90304
28959+
total_nblocks | 3
28960+
free_bytes | 2880
28961+
free_chunks | 1
28962+
used_bytes | 87424
28963+
num_agg_contexts | 1
28964+
stats_timestamp | 2025-03-24 13:55:47.796698+01
28965+
</programlisting>
28966+
<note>
28967+
<para>
28968+
While <function>pg_get_process_memory_contexts</function> can be used to
28969+
query memory contexts of the local backend,
28970+
<structname>pg_backend_memory_contexts</structname>
28971+
(see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
28972+
will be less resource intensive when only the local backend is of interest.
28973+
</para>
28974+
</note>
28975+
</para>
28976+
2880528977
</sect2>
2880628978

2880728979
<sect2 id="functions-admin-backup">

‎src/backend/catalog/system_views.sql

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -674,6 +674,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
674674
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
675675
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
676676

677+
REVOKE EXECUTE ON FUNCTION
678+
pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
679+
GRANT EXECUTE ON FUNCTION
680+
pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
681+
677682
-- Statistics views
678683

679684
CREATE VIEW pg_stat_all_tables AS

‎src/backend/postmaster/autovacuum.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -781,6 +781,10 @@ ProcessAutoVacLauncherInterrupts(void)
781781
if (LogMemoryContextPending)
782782
ProcessLogMemoryContextInterrupt();
783783

784+
/* Publish memory contexts of this process */
785+
if (PublishMemoryContextPending)
786+
ProcessGetMemoryContextInterrupt();
787+
784788
/* Process sinval catchup interrupts that happened while sleeping */
785789
ProcessCatchupInterrupt();
786790
}

‎src/backend/postmaster/checkpointer.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -663,6 +663,10 @@ ProcessCheckpointerInterrupts(void)
663663
/* Perform logging of memory contexts of this process */
664664
if (LogMemoryContextPending)
665665
ProcessLogMemoryContextInterrupt();
666+
667+
/* Publish memory contexts of this process */
668+
if (PublishMemoryContextPending)
669+
ProcessGetMemoryContextInterrupt();
666670
}
667671

668672
/*

‎src/backend/postmaster/interrupt.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
4848
/* Perform logging of memory contexts of this process */
4949
if (LogMemoryContextPending)
5050
ProcessLogMemoryContextInterrupt();
51+
52+
/* Publish memory contexts of this process */
53+
if (PublishMemoryContextPending)
54+
ProcessGetMemoryContextInterrupt();
5155
}
5256

5357
/*

‎src/backend/postmaster/pgarch.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
867867
if (LogMemoryContextPending)
868868
ProcessLogMemoryContextInterrupt();
869869

870+
/* Publish memory contexts of this process */
871+
if (PublishMemoryContextPending)
872+
ProcessGetMemoryContextInterrupt();
873+
870874
if (ConfigReloadPending)
871875
{
872876
char *archiveLib = pstrdup(XLogArchiveLibrary);

‎src/backend/postmaster/startup.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
192192
/* Perform logging of memory contexts of this process */
193193
if (LogMemoryContextPending)
194194
ProcessLogMemoryContextInterrupt();
195+
196+
/* Publish memory contexts of this process */
197+
if (PublishMemoryContextPending)
198+
ProcessGetMemoryContextInterrupt();
195199
}
196200

197201

‎src/backend/postmaster/walsummarizer.c

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
879879
/* Perform logging of memory contexts of this process */
880880
if (LogMemoryContextPending)
881881
ProcessLogMemoryContextInterrupt();
882+
883+
/* Publish memory contexts of this process */
884+
if (PublishMemoryContextPending)
885+
ProcessGetMemoryContextInterrupt();
882886
}
883887

884888
/*

‎src/backend/storage/ipc/ipci.c

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@
5151
#include "storage/sinvaladt.h"
5252
#include "utils/guc.h"
5353
#include "utils/injection_point.h"
54+
#include "utils/memutils.h"
5455

5556
/* GUCs */
5657
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
150151
size = add_size(size, InjectionPointShmemSize());
151152
size = add_size(size, SlotSyncShmemSize());
152153
size = add_size(size, AioShmemSize());
154+
size = add_size(size, MemoryContextReportingShmemSize());
153155

154156
/* include additional requested shmem from preload libraries */
155157
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
343345
WaitEventCustomShmemInit();
344346
InjectionPointShmemInit();
345347
AioShmemInit();
348+
MemoryContextReportingShmemInit();
346349
}
347350

348351
/*

‎src/backend/storage/ipc/procsignal.c

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -690,6 +690,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
690690
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
691691
HandleLogMemoryContextInterrupt();
692692

693+
if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
694+
HandleGetMemoryContextInterrupt();
695+
693696
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
694697
HandleParallelApplyMessageInterrupt();
695698

‎src/backend/storage/lmgr/lwlock.c

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,8 @@ static const char *const BuiltinTrancheNames[] = {
178178
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
179179
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
180180
[LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
181+
[LWTRANCHE_MEMORY_CONTEXT_REPORTING_STATE] = "MemoryContextReportingState",
182+
[LWTRANCHE_MEMORY_CONTEXT_REPORTING_PROC] = "MemoryContextReportingPerProcess",
181183
};
182184

183185
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==

‎src/backend/storage/lmgr/proc.c

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
#include "storage/procsignal.h"
5151
#include "storage/spin.h"
5252
#include "storage/standby.h"
53+
#include "utils/memutils.h"
5354
#include "utils/timeout.h"
5455
#include "utils/timestamp.h"
5556

‎src/backend/tcop/postgres.c

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3535,6 +3535,9 @@ ProcessInterrupts(void)
35353535
if (LogMemoryContextPending)
35363536
ProcessLogMemoryContextInterrupt();
35373537

3538+
if (PublishMemoryContextPending)
3539+
ProcessGetMemoryContextInterrupt();
3540+
35383541
if (ParallelApplyMessagePending)
35393542
ProcessParallelApplyMessages();
35403543
}

‎src/backend/utils/activity/wait_event_names.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
161161
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
162162
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
163163
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
164+
MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
164165

165166
ABI_compatibility:
166167

0 commit comments

Comments
 (0)
0