You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{"payload":{"commit":{"oid":"522baf14847a7e4cc97c49c7b1c28d21bc33921f","url":"/postgres/postgres/commit/522baf14847a7e4cc97c49c7b1c28d21bc33921f","authoredDate":"2019-09-04T13:21:11.000+09:00","committedDate":"2019-09-04T13:21:11.000+09:00","shortMessage":null,"shortMessageMarkdown":"\u003cdiv\u003eDelay fsyncs of pg_basebackup until the end of backup\u003c/div\u003e","shortMessageMarkdownLink":null,"bodyMessageHtml":"Since the addition of fsync requests in bc34223 to make base backup data\nconsistent on disk once pg_basebackup finishes, each tablespace tar file\nis individually flushed once completed, with an additional flush of the\nparent directory when the base backup finishes. While holding a\nconnection to the server, a fsync request taking a long time may cause a\nfailure of the base backup, which is annoying for any integration. A\nrecent example of breakage can involve tcp_user_timeout, but\nwal_sender_timeout can cause similar problems.\n\nWhile reviewing the code, there was a second issue causing too many\nfsync requests to be done for the same WAL data. As recursive fsyncs\nare done at the end of the backup for both the plain and tar formats\nfrom the base target directory where everything is written, it is fine\nto disable fsyncs when fetching or streaming WAL.\n\nReported-by: Ryohei Takahashi\nAuthor: Michael Paquier\nReviewed-by: Ryohei Takahashi\nDiscussion: \u003ca href=\"https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com\" rel=\"nofollow\"\u003ehttps://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com\u003c/a\u003e\nBackpatch-through: 10","authors":[{"login":"michaelpq","displayName":"Michael Paquier","avatarUrl":"https://avatars.githubusercontent.com/u/985332?v=4","path":"/michaelpq","isGitHub":false}],"committerAttribution":false,"committer":{"login":"michaelpq","displayName":"Michael Paquier","avatarUrl":"https://avatars.githubusercontent.com/u/985332?v=4","path":"/michaelpq","isGitHub":false},"parents":["25dcc9d35dfeb027047ebaea9b27cda1eaa9b393"],"globalRelayId":"MDY6Q29tbWl0OTI3NDQyOjUyMmJhZjE0ODQ3YTdlNGNjOTdjNDljN2IxYzI4ZDIxYmMzMzkyMWY=","sha1":"25dcc9d35dfeb027047ebaea9b27cda1eaa9b393","sha2":"522baf14847a7e4cc97c49c7b1c28d21bc33921f"},"currentUser":null,"repo":{"id":927442,"defaultBranch":"master","name":"postgres","ownerLogin":"postgres","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2010-09-21T11:35:45.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/177543?v=4","public":true,"private":false,"isOrgOwned":true},"diffEntryData":[{"diffLines":[{"stylingDirective":null,"type":"HUNK","blobLineNumber":485,"text":"@@ -486,15 +486,18 @@ LogStreamerMain(logstreamer_param *param)","html":"@@ -486,15 +486,18 @@ LogStreamerMain(logstreamer_param *param)","displayNoNewLineWarning":false,"position":0,"left":485,"right":485},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":486,"text":" #endif","html":" \u003cspan class=pl-k\u003e#endif\u003c/span\u003e","displayNoNewLineWarning":false,"position":1,"left":486,"right":486},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":487,"text":" \tstream.standby_message_timeout = standby_message_timeout;","html":" \t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003estandby_message_timeout\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-s1\u003estandby_message_timeout\u003c/span\u003e;","displayNoNewLineWarning":false,"position":2,"left":487,"right":487},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":488,"text":" \tstream.synchronous = false;","html":" \t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003esynchronous\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e false;","displayNoNewLineWarning":false,"position":3,"left":488,"right":488},{"stylingDirective":null,"type":"DELETION","blobLineNumber":489,"text":"-\tstream.do_sync = do_sync;","html":"-\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003edo_sync\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-s1\u003edo_sync\u003c/span\u003e;","displayNoNewLineWarning":false,"position":4,"left":489,"right":488},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":489,"text":"+\t/* fsync happens at the end of pg_basebackup for all data */","html":"+\t\u003cspan class=pl-c\u003e/* fsync happens at the end of pg_basebackup for all data */\u003c/span\u003e","displayNoNewLineWarning":false,"position":5,"left":489,"right":489},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":490,"text":"+\tstream.do_sync = false;","html":"+\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003edo_sync\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e false;","displayNoNewLineWarning":false,"position":6,"left":489,"right":490},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":491,"text":" \tstream.mark_done = true;","html":" \t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003emark_done\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e true;","displayNoNewLineWarning":false,"position":7,"left":490,"right":491},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":492,"text":" \tstream.partial_suffix = NULL;","html":" \t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003epartial_suffix\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-c1\u003eNULL\u003c/span\u003e;","displayNoNewLineWarning":false,"position":8,"left":491,"right":492},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":493,"text":" \tstream.replication_slot = replication_slot;","html":" \t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003ereplication_slot\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-s1\u003ereplication_slot\u003c/span\u003e;","displayNoNewLineWarning":false,"position":9,"left":492,"right":493},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":494,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":10,"left":493,"right":494},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":495,"text":" \tif (format == 'p')","html":" \t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-s1\u003eformat\u003c/span\u003e \u003cspan class=pl-c1\u003e==\u003c/span\u003e \u003cspan class=pl-c1\u003e\u0026#39;p\u0026#39;\u003c/span\u003e)","displayNoNewLineWarning":false,"position":11,"left":494,"right":495},{"stylingDirective":null,"type":"DELETION","blobLineNumber":495,"text":"-\t\tstream.walmethod = CreateWalDirectoryMethod(param-\u003exlog, 0, do_sync);","html":"-\t\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003ewalmethod\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-en\u003eCreateWalDirectoryMethod\u003c/span\u003e(\u003cspan class=pl-s1\u003eparam\u003c/span\u003e\u003cspan class=pl-c1\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=pl-c1\u003exlog\u003c/span\u003e, \u003cspan class=pl-c1\u003e0\u003c/span\u003e, \u003cspan class=pl-s1\u003edo_sync\u003c/span\u003e);","displayNoNewLineWarning":false,"position":12,"left":495,"right":495},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":496,"text":"+\t\tstream.walmethod = CreateWalDirectoryMethod(param-\u003exlog, 0,","html":"+\t\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003ewalmethod\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-en\u003eCreateWalDirectoryMethod\u003c/span\u003e(\u003cspan class=pl-s1\u003eparam\u003c/span\u003e\u003cspan class=pl-c1\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=pl-c1\u003exlog\u003c/span\u003e, \u003cspan class=pl-c1\u003e0\u003c/span\u003e,","displayNoNewLineWarning":false,"position":13,"left":495,"right":496},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":497,"text":"+\t\t\t\t\t\t\t\t\t\t\t\t\tstream.do_sync);","html":"+\t\t\t\t\t\t\t\t\t\t\t\t\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003edo_sync\u003c/span\u003e);","displayNoNewLineWarning":false,"position":14,"left":495,"right":497},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":498,"text":" \telse","html":" \t\u003cspan class=pl-k\u003eelse\u003c/span\u003e","displayNoNewLineWarning":false,"position":15,"left":496,"right":498},{"stylingDirective":null,"type":"DELETION","blobLineNumber":497,"text":"-\t\tstream.walmethod = CreateWalTarMethod(param-\u003exlog, compresslevel, do_sync);","html":"-\t\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003ewalmethod\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-en\u003eCreateWalTarMethod\u003c/span\u003e(\u003cspan class=pl-s1\u003eparam\u003c/span\u003e\u003cspan class=pl-c1\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=pl-c1\u003exlog\u003c/span\u003e, \u003cspan class=pl-s1\u003ecompresslevel\u003c/span\u003e, \u003cspan class=pl-s1\u003edo_sync\u003c/span\u003e);","displayNoNewLineWarning":false,"position":16,"left":497,"right":498},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":499,"text":"+\t\tstream.walmethod = CreateWalTarMethod(param-\u003exlog, compresslevel,","html":"+\t\t\u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003ewalmethod\u003c/span\u003e \u003cspan class=pl-c1\u003e=\u003c/span\u003e \u003cspan class=pl-en\u003eCreateWalTarMethod\u003c/span\u003e(\u003cspan class=pl-s1\u003eparam\u003c/span\u003e\u003cspan class=pl-c1\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=pl-c1\u003exlog\u003c/span\u003e, \u003cspan class=pl-s1\u003ecompresslevel\u003c/span\u003e,","displayNoNewLineWarning":false,"position":17,"left":497,"right":499},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":500,"text":"+\t\t\t\t\t\t\t\t\t\t\t stream.do_sync);","html":"+\t\t\t\t\t\t\t\t\t\t\t \u003cspan class=pl-s1\u003estream\u003c/span\u003e.\u003cspan class=pl-c1\u003edo_sync\u003c/span\u003e);","displayNoNewLineWarning":false,"position":18,"left":497,"right":500},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":501,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":19,"left":498,"right":501},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":502,"text":" \tif (!ReceiveXlogStream(param-\u003ebgconn, \u0026stream))","html":" \t\u003cspan class=pl-k\u003eif\u003c/span\u003e (!\u003cspan class=pl-en\u003eReceiveXlogStream\u003c/span\u003e(\u003cspan class=pl-s1\u003eparam\u003c/span\u003e\u003cspan class=pl-c1\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=pl-c1\u003ebgconn\u003c/span\u003e, \u003cspan class=pl-c1\u003e\u0026amp;\u003c/span\u003e\u003cspan class=pl-s1\u003estream\u003c/span\u003e))","displayNoNewLineWarning":false,"position":20,"left":499,"right":502},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":503,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":21,"left":500,"right":503},{"stylingDirective":null,"type":"HUNK","blobLineNumber":1348,"text":"@@ -1346,9 +1349,10 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)","html":"@@ -1346,9 +1349,10 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)","displayNoNewLineWarning":false,"position":22,"left":1345,"right":1348},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1349,"text":" \tif (copybuf != NULL)","html":" \t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-s1\u003ecopybuf\u003c/span\u003e \u003cspan class=pl-c1\u003e!=\u003c/span\u003e \u003cspan class=pl-c1\u003eNULL\u003c/span\u003e)","displayNoNewLineWarning":false,"position":23,"left":1346,"right":1349},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1350,"text":" \t\tPQfreemem(copybuf);","html":" \t\t\u003cspan class=pl-en\u003ePQfreemem\u003c/span\u003e(\u003cspan class=pl-s1\u003ecopybuf\u003c/span\u003e);","displayNoNewLineWarning":false,"position":24,"left":1347,"right":1350},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1351,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":25,"left":1348,"right":1351},{"stylingDirective":null,"type":"DELETION","blobLineNumber":1349,"text":"-\t/* sync the resulting tar file, errors are not considered fatal */","html":"-\t\u003cspan class=pl-c\u003e/* sync the resulting tar file, errors are not considered fatal */\u003c/span\u003e","displayNoNewLineWarning":false,"position":26,"left":1349,"right":1351},{"stylingDirective":null,"type":"DELETION","blobLineNumber":1350,"text":"-\tif (do_sync \u0026\u0026 strcmp(basedir, \"-\") != 0)","html":"-\t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-s1\u003edo_sync\u003c/span\u003e \u003cspan class=pl-c1\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=pl-en\u003estrcmp\u003c/span\u003e(\u003cspan class=pl-s1\u003ebasedir\u003c/span\u003e, \u003cspan class=pl-s\u003e\u0026quot;-\u0026quot;\u003c/span\u003e) \u003cspan class=pl-c1\u003e!=\u003c/span\u003e \u003cspan class=pl-c1\u003e0\u003c/span\u003e)","displayNoNewLineWarning":false,"position":27,"left":1350,"right":1351},{"stylingDirective":null,"type":"DELETION","blobLineNumber":1351,"text":"-\t\t(void) fsync_fname(filename, false);","html":"-\t\t(\u003cspan class=pl-smi\u003evoid\u003c/span\u003e) \u003cspan class=pl-en\u003efsync_fname\u003c/span\u003e(\u003cspan class=pl-s1\u003efilename\u003c/span\u003e, false);","displayNoNewLineWarning":false,"position":28,"left":1351,"right":1351},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":1352,"text":"+\t/*","html":"+\t\u003cspan class=pl-c\u003e/*\u003c/span\u003e","displayNoNewLineWarning":false,"position":29,"left":1351,"right":1352},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":1353,"text":"+\t * Do not sync the resulting tar file yet, all files are synced once at","html":"+\u003cspan class=pl-c\u003e\t * Do not sync the resulting tar file yet, all files are synced once at\u003c/span\u003e","displayNoNewLineWarning":false,"position":30,"left":1351,"right":1353},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":1354,"text":"+\t * the end.","html":"+\u003cspan class=pl-c\u003e\t * the end.\u003c/span\u003e","displayNoNewLineWarning":false,"position":31,"left":1351,"right":1354},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":1355,"text":"+\t */","html":"+\u003cspan class=pl-c\u003e\t */\u003c/span\u003e","displayNoNewLineWarning":false,"position":32,"left":1351,"right":1355},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1356,"text":" }","html":" }","displayNoNewLineWarning":false,"position":33,"left":1352,"right":1356},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1357,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":34,"left":1353,"right":1357},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":1358,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":35,"left":1354,"right":1358},{"stylingDirective":null,"type":"HUNK","blobLineNumber":2141,"text":"@@ -2138,9 +2142,9 @@ BaseBackup(void)","html":"@@ -2138,9 +2142,9 @@ BaseBackup(void)","displayNoNewLineWarning":false,"position":36,"left":2137,"right":2141},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2142,"text":" ","html":"\u003cbr\u003e","displayNoNewLineWarning":false,"position":37,"left":2138,"right":2142},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2143,"text":" \t/*","html":" \t\u003cspan class=pl-c\u003e/*\u003c/span\u003e","displayNoNewLineWarning":false,"position":38,"left":2139,"right":2143},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2144,"text":" \t * Make data persistent on disk once backup is completed. For tar format","html":" \u003cspan class=pl-c\u003e\t * Make data persistent on disk once backup is completed. For tar format\u003c/span\u003e","displayNoNewLineWarning":false,"position":39,"left":2140,"right":2144},{"stylingDirective":null,"type":"DELETION","blobLineNumber":2141,"text":"-\t * once syncing the parent directory is fine, each tar file created per","html":"-\u003cspan class=\"pl-c\"\u003e\t * \u003cspan class=\"x x-first x-last\"\u003eonce syncing \u003c/span\u003ethe parent directory \u003cspan class=\"x x-first x-last\"\u003eis fine, \u003c/span\u003eeach tar file \u003cspan class=\"x x-first x-last\"\u003ecreated per\u003c/span\u003e\u003c/span\u003e","displayNoNewLineWarning":false,"position":40,"left":2141,"right":2144},{"stylingDirective":null,"type":"DELETION","blobLineNumber":2142,"text":"-\t * tablespace has been already synced. In plain format, all the data of","html":"-\u003cspan class=\"pl-c\"\u003e\t * \u003cspan class=\"x x-first x-last\"\u003etablespace has been already synced. \u003c/span\u003eIn plain format, all the data of\u003c/span\u003e","displayNoNewLineWarning":false,"position":41,"left":2142,"right":2144},{"stylingDirective":null,"type":"DELETION","blobLineNumber":2143,"text":"-\t * the base directory is synced, taking into account all the tablespaces.","html":"-\u003cspan class=\"pl-c\"\u003e\t * \u003cspan class=\"x x-first x-last\"\u003ethe \u003c/span\u003ebase directory is synced, taking into account all the tablespaces.\u003c/span\u003e","displayNoNewLineWarning":false,"position":42,"left":2143,"right":2144},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":2145,"text":"+\t * sync the parent directory and all its contents as each tar file was not","html":"+\u003cspan class=\"pl-c\"\u003e\t * \u003cspan class=\"x x-first x-last\"\u003esync \u003c/span\u003ethe parent directory \u003cspan class=\"x x-first x-last\"\u003eand all its contents as \u003c/span\u003eeach tar file \u003cspan class=\"x x-first x-last\"\u003ewas not\u003c/span\u003e\u003c/span\u003e","displayNoNewLineWarning":false,"position":43,"left":2143,"right":2145},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":2146,"text":"+\t * synced after being completed. In plain format, all the data of the","html":"+\u003cspan class=\"pl-c\"\u003e\t * \u003cspan class=\"x x-first x-last\"\u003esynced after being completed. \u003c/span\u003eIn plain format, all the data of\u003cspan class=\"x x-first x-last\"\u003e the\u003c/span\u003e\u003c/span\u003e","displayNoNewLineWarning":false,"position":44,"left":2143,"right":2146},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":2147,"text":"+\t * base directory is synced, taking into account all the tablespaces.","html":"+\u003cspan class=\"pl-c\"\u003e\t * base directory is synced, taking into account all the tablespaces.\u003c/span\u003e","displayNoNewLineWarning":false,"position":45,"left":2143,"right":2147},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2148,"text":" \t * Errors are not considered fatal.","html":" \u003cspan class=pl-c\u003e\t * Errors are not considered fatal.\u003c/span\u003e","displayNoNewLineWarning":false,"position":46,"left":2144,"right":2148},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2149,"text":" \t */","html":" \u003cspan class=pl-c\u003e\t */\u003c/span\u003e","displayNoNewLineWarning":false,"position":47,"left":2145,"right":2149},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2150,"text":" \tif (do_sync)","html":" \t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-s1\u003edo_sync\u003c/span\u003e)","displayNoNewLineWarning":false,"position":48,"left":2146,"right":2150},{"stylingDirective":null,"type":"HUNK","blobLineNumber":2153,"text":"@@ -2150,7 +2154,7 @@ BaseBackup(void)","html":"@@ -2150,7 +2154,7 @@ BaseBackup(void)","displayNoNewLineWarning":false,"position":49,"left":2149,"right":2153},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2154,"text":" \t\tif (format == 't')","html":" \t\t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-s1\u003eformat\u003c/span\u003e \u003cspan class=pl-c1\u003e==\u003c/span\u003e \u003cspan class=pl-c1\u003e\u0026#39;t\u0026#39;\u003c/span\u003e)","displayNoNewLineWarning":false,"position":50,"left":2150,"right":2154},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2155,"text":" \t\t{","html":" \t\t{","displayNoNewLineWarning":false,"position":51,"left":2151,"right":2155},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2156,"text":" \t\t\tif (strcmp(basedir, \"-\") != 0)","html":" \t\t\t\u003cspan class=pl-k\u003eif\u003c/span\u003e (\u003cspan class=pl-en\u003estrcmp\u003c/span\u003e(\u003cspan class=pl-s1\u003ebasedir\u003c/span\u003e, \u003cspan class=pl-s\u003e\u0026quot;-\u0026quot;\u003c/span\u003e) \u003cspan class=pl-c1\u003e!=\u003c/span\u003e \u003cspan class=pl-c1\u003e0\u003c/span\u003e)","displayNoNewLineWarning":false,"position":52,"left":2152,"right":2156},{"stylingDirective":null,"type":"DELETION","blobLineNumber":2153,"text":"-\t\t\t\t(void) fsync_fname(basedir, true);","html":"-\t\t\t\t(\u003cspan class=\"pl-smi\"\u003evoid\u003c/span\u003e) \u003cspan class=\"pl-en x x-first x-last\"\u003efsync_fname\u003c/span\u003e(\u003cspan class=\"pl-s1\"\u003ebasedir\u003c/span\u003e\u003cspan class=\"x x-first x-last\"\u003e, true\u003c/span\u003e);","displayNoNewLineWarning":false,"position":53,"left":2153,"right":2156},{"stylingDirective":null,"type":"ADDITION","blobLineNumber":2157,"text":"+\t\t\t\t(void) fsync_dir_recurse(basedir);","html":"+\t\t\t\t(\u003cspan class=\"pl-smi\"\u003evoid\u003c/span\u003e) \u003cspan class=\"pl-en x x-first x-last\"\u003efsync_dir_recurse\u003c/span\u003e(\u003cspan class=\"pl-s1\"\u003ebasedir\u003c/span\u003e);","displayNoNewLineWarning":false,"position":54,"left":2153,"right":2157},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2158,"text":" \t\t}","html":" \t\t}","displayNoNewLineWarning":false,"position":55,"left":2154,"right":2158},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2159,"text":" \t\telse","html":" \t\t\u003cspan class=pl-k\u003eelse\u003c/span\u003e","displayNoNewLineWarning":false,"position":56,"left":2155,"right":2159},{"stylingDirective":null,"type":"CONTEXT","blobLineNumber":2160,"text":" \t\t{","html":" \t\t{","displayNoNewLineWarning":false,"position":57,"left":2156,"right":2160}],"diffNumber":0,"diffSize":"0 Bytes","isBinary":false,"isTooBig":false,"collapsed":false,"isSubmodule":false,"lineCount":2555,"linesChanged":24,"newTreeEntry":{"lineCount":2555,"path":"src/bin/pg_basebackup/pg_basebackup.c","mode":100644,"isGenerated":false},"oldTreeEntry":{"lineCount":0,"path":"src/bin/pg_basebackup/pg_basebackup.c","mode":100644},"linesAdded":14,"linesDeleted":10,"path":"src/bin/pg_basebackup/pg_basebackup.c","pathDigest":"ed4d82fd0d77f33a1f9489cade8a2a8168589d6e299043252d0d372373447e29","status":"MODIFIED","truncatedReason":null,"oldOid":"25dcc9d35dfeb027047ebaea9b27cda1eaa9b393","newOid":"522baf14847a7e4cc97c49c7b1c28d21bc33921f","copilotChatReference":null,"deletedSha":"25dcc9d35dfeb027047ebaea9b27cda1eaa9b393","canToggleRichDiff":false,"defaultToRichDiff":false,"proseDifffHtml":null,"renderInfo":null,"dependencyDiffPath":null,"submodule":null}],"splitViewPreference":"unified","ignoreWhitespace":false,"repoOwnerGlobalRelayId":"MDEyOk9yZ2FuaXphdGlvbjE3NzU0Mw==","commentsPreference":"visible","diffLineSpacingPreference":"relaxed","useMonospaceFont":false,"pasteUrlLinkAsPlainText":false,"userNotices":[],"path":"/postgres/postgres/commit/522baf14847a7e4cc97c49c7b1c28d21bc33921f","fileTreeExpanded":true,"headerInfo":{"additions":14,"deletions":10,"filesChanged":1,"filesChangedString":"1"},"moreDiffsToLoad":false,"asyncDiffLoadInfo":{"startIndex":1,"truncated":false,"byteCount":2024,"lineShownCount":58},"commentInfo":{"canComment":false,"locked":false,"canLock":false,"repoArchived":false},"csrf_tokens":{"/users/diffview?diff=split":{"post":"Ocw8D1WS3CSKKNjRvYT8ZdiTrHAsWa2hBbSiXMy--dBN2me-U2o0Jo9FjLKf9UDKqp9IwR5kUpctPBuDqKlM9g"},"/users/diffview?diff=unified":{"post":"73HGQkSSHKkCzqewZvZh_WNBdHpaCsO5eA53KOrd6LCbZ53zQmr0qwej89NEh91SEU2Qy2g3PI9Qhs73jspdlg"},"/notifications/thread":{"post":"44k65CucXsfMXT6P1CkfH6_ZJx1mXl7Fks1zC6ryXhxIlUiWEVN_CHl2DFtmIqkd4F-LBQUVepglalkODqGx1w"}}},"title":"Delay fsyncs of pg_basebackup until the end of backup · postgres/postgres@522baf1","appPayload":{"helpUrl":"https://docs.github.com","findInDiffWorkerPath":"/assets-cdn/worker/find-in-diff-worker-2bfe39677d14.js","enabled_features":{"diff_ux_refresh_beta":false,"diff_inline_comments":true,"diff_ux_refresh_ssr_five":false,"diff_ux_refresh_ssr_ten":false,"react_diff_line_type_character_correction":true}}}
Delay fsyncs of pg_basebackup until the end of backup
Since the addition of fsync requests in bc34223 to make base backup data
consistent on disk once pg_basebackup finishes, each tablespace tar file
is individually flushed once completed, with an additional flush of the
parent directory when the base backup finishes. While holding a
connection to the server, a fsync request taking a long time may cause a
failure of the base backup, which is annoying for any integration. A
recent example of breakage can involve tcp_user_timeout, but
wal_sender_timeout can cause similar problems.
While reviewing the code, there was a second issue causing too many
fsync requests to be done for the same WAL data. As recursive fsyncs
are done at the end of the backup for both the plain and tar formats
from the base target directory where everything is written, it is fine
to disable fsyncs when fetching or streaming WAL.
Reported-by: Ryohei Takahashi
Author: Michael Paquier
Reviewed-by: Ryohei Takahashi
Discussion: https://postgr.es/m/OSBPR01MB4550DAE2F8C9502894A45AAB82BE0@OSBPR01MB4550.jpnprd01.prod.outlook.com
Backpatch-through: 10
0 commit comments