Database migration temporary API sometimes uses a US-ASCII Content-Type with non-US-ASCII data #10499
Description
What happened?
Syncthing version 1.x uses LevelDB. Syncthing version 2.x uses SQLite. When you upgrade from Syncthing version 1.x to Syncthing version 2.x, Syncthing does a database migration. During this database migration, Syncthing’s regular GUI/REST API is not available. Instead, a temporary API is made available in its place.
The temporary API seems to always use text/plain as its Content-Type. Section 4.1.2 of RFC 2046 says:
A critical parameter that may be specified in the Content-Type field
for "text/plain" data is the character set. This is specified with a
"charset" parameter, as in:
Content-type: text/plain; charset=iso-8859-1
Unlike some other parameter values, the values of the charset
parameter are NOT case sensitive. The default character set, which
must be assumed in the absence of a charset parameter, is US-ASCII.
Additionally, section 4 of RFC 6657 says:
The default "charset" parameter value for "text/plain" is unchanged from [RFC2046] and remains as "US-ASCII".
Unfortunately, Syncthing’s temporary API sometimes produces HTTP responses that use that default US-ASCII charset value but contain non-US-ASCII data.
Steps to reproduce
These steps to reproduce will automatically create a virtual machine and compile Syncthing version 1.x and version 2.x inside that virtual machine. When compiling Syncthing, it uses -tags 🧌 in order to make sure that Syncthing logs a non-US-ASCII character at startup. After that, it starts and stops Syncthing version 1.x, and then starts Syncthing version 2.x in order to trigger a database migration. During the database migration, it uses curl --verbose in order to capture a Syncthing temporary API HTTP response that uses text/plain as its Content-Type but contains non-US-ASCII data.
These steps to reproduce patch Syncthing version 2.x in order add a delay to the migration process that ensures that the database migration takes long enough for us to use the temporary API. I did it that way because I wasn’t sure how else to make sure that the database migration would take long enough.
-
Make sure that you have the Nix package manager installed by running this command:
nix-shell --version
If that command finishes successfully, then you have the Nix package manager installed. If that command gives you an error, then you need to install the Nix package manager.
-
Create a file named
reproduce-syncthing-database-migration-temp-api-content-type-bug.nixthat contains the following Nix expression:# This file is dedicated to the public domain using 🅭🄍1.0: # <https://creativecommons.org/publicdomain/zero/1.0>. let nixpkgsCommit = "c6f52ebd45e5925c188d1a20119978aa4ffd5ef6"; nixpkgsDirectory = builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/${nixpkgsCommit}.tar.gz"; sha256 = "0gwxhs3j1nglyymbaqyqg8miz0rk84n4ijag5s4bx6yfb6vrd4lv"; }; pkgs = import nixpkgsDirectory { }; bashTestCode = pkgs.writeShellApplication { name = "bash-test-code"; runtimeInputs = [ pkgs.gcc pkgs.go pkgs.parallel ]; text = '' set -o errexit -o nounset -o pipefail function wait_for_internet_access { while ! curl --silent --show-error --head example.com > /dev/null do echo No Internet access at the moment. Waiting for Internet access… >&2 sleep 15 done } function attempt_to_stop_syncthing { echo Attempting to stop Syncthing… curl \ --silent \ --show-error \ --retry 60 \ --retry-delay 1 \ --retry-all-errors \ --header "X-API-Key: abc123" \ --data "" \ http://localhost:8384/rest/system/shutdown > /dev/null } export -f attempt_to_stop_syncthing function attempt_to_use_temporary_api_then_stop_syncthing { curl \ --silent \ --show-error \ --retry 60 \ --retry-delay 1 \ --retry-all-errors \ --verbose \ http://localhost:8384/rest/noauth/health &> 'curl temporary API usage log.txt' # Wait for the Syncthing to finish migrating. local content_type while true do content_type="$(curl \ --silent \ --show-error \ --retry 60 \ --retry-delay 1 \ --retry-all-errors \ --output /dev/null \ --write-out '%header{Content-Type}' \ http://localhost:8384/rest/noauth/health )" if printf %s "$content_type" | grep --ignore-case --perl-regexp '^text/plain($|([ \t]*;.*))' then echo Waiting for Syncthing to finish migrating… sleep 10 elif printf %s "$content_type" | grep --ignore-case --perl-regexp '^application/json($|([ \t]*;.*))' then echo Syncthing has finished migrating. break else printf 'ERROR: Syncthing responded with an unexpected Content-Type: %s\n' "$content_type" # This is the EX_PROTOCOL exit status from <man:sysexits.h(3head)>. exit 76 fi done attempt_to_stop_syncthing } export -f attempt_to_use_temporary_api_then_stop_syncthing cp --recursive /etc/syncthingVersion1SourceCode/ /etc/syncthingVersion2SourceCode/ . # In order to trigger this issue, the Syncthing database migration # process must last for long enough for us to access the temporary # GUI/API. This patch ensures that the database migration process will # take long enough. # # I created this patch because I couldn’t come up with a better way to # make sure that the migration process is slow enough. patch --strip=1 --directory=syncthingVersion2SourceCode < /etc/intentionallySlowMigrationPatch cd syncthingVersion1SourceCode wait_for_internet_access go run build.go -tags 🧌 cd ../syncthingVersion2SourceCode go run build.go -tags 🧌 cd .. # At this point, we no longer need Internet access, so we turn Internet # access off. nmcli networking off parallel --jobs 200% --ungroup --halt now,fail=1 ::: \ 'syncthingVersion1SourceCode/bin/syncthing --gui-apikey=abc123' \ attempt_to_stop_syncthing parallel --jobs 200% --ungroup --halt now,fail=1 ::: \ 'syncthingVersion2SourceCode/bin/syncthing --gui-apikey=abc123' \ attempt_to_use_temporary_api_then_stop_syncthing ''; }; nixosModule = { pkgs, ... }: { environment = { systemPackages = [ bashTestCode ]; etc = { syncthingVersion1SourceCode.source = pkgs.fetchFromGitHub { owner = "syncthing"; repo = "syncthing"; tag = "v1.30.0"; hash = "sha256-GKyzJ2kzs2h/tfb3StSleGBofiKk6FwVcSkCjsJRvRY="; }; syncthingVersion2SourceCode.source = pkgs.fetchFromGitHub { owner = "syncthing"; repo = "syncthing"; rev = "801ef0e22d354b23180508b4bde7f2537418569c"; hash = "sha256-7B6Ocq0xK/J5/8LnzuTTJr0j6cf/U0H19OtB9uFvE48="; }; intentionallySlowMigrationPatch.source = pkgs.fetchpatch { url = "https://codeberg.org/JasonYundt/syncthing/compare/intentionally-slow-migration-rev-0^...intentionally-slow-migration-rev-0.patch"; hash = "sha256-Zc2S+lr3E+WKHcsBx8xN6zmhSpI308QslUKZC/e/glk="; }; }; }; networking.networkmanager.enable = true; # Without this next part, the test virtual machine would run out of disk # space. virtualisation.diskSize = 4 * 1024; }; nixosIntegrationTest = pkgs.testers.runNixOSTest { name = "reproduce-syncthing-database-migration-temp-api-content-type-bug"; nodes.main = nixosModule; testScript = '' machine.succeed("bash-test-code") print( "Contents of “curl temporary API usage log.txt”:", machine.succeed("cat 'curl temporary API usage log.txt'"), sep="\n" ) ''; }; in # Normally, NixOS integration tests aren’t able to access the Internet. # Building nixosIntegrationTest.driver instead of just nixosIntegrationTest # allows us to create a NixOS integration test that is able to access the # Internet. nixosIntegrationTest.driver
-
Change directory into the directory that contains
reproduce-syncthing-database-migration-temp-api-content-type-bug.nixby running this command:cd <path to directory that contains reproduce-syncthing-database-migration-temp-api-content-type-bug.nix>
-
Run the reproducer by running this command:
nix-shell --packages 'import ./reproduce-syncthing-database-migration-temp-api-content-type-bug.nix' --run nixos-test-driver
Expected results
I would expect that the temporary API would use text/plain; charset=utf-8 as its Content-Type just like the regular API’s /rest/system/log.txt endpoint does. After all, both the temporary API and the regular API’s /rest/system/log.txt endpoint return a chunk of text that contains Syncthing log messages.
Syncthing version
Upgrading from v1.30.0 to 801ef0e with a patch to intentionally slow down database migrations
Platform & operating system
Host: NixOS 25.05 x86_64. VM: NixOS 25.11 X86_64.
Browser version
No response
Relevant log output
I tried pasting the log here, but when I tried to post this issue, GitHub gave
me this error: “Field can not be longer than 65536 characters”.
As a workaround, I uploaded the log here:
https://gist.github.com/Jayman2000/91054582fd2520fd8e20f9ac189743a4.