From 3ac55ee10a73eea5ba45066f861392d618792eeb Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Wed, 22 Nov 2023 10:43:01 +0100 Subject: [PATCH 01/54] Update CHANGELOG.md --- CHANGELOG.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2489a4c..059feae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,54 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.15.3 (2023-11-22) + +### New features + +- A new compilation flag (`--fail-on-ambiguous-relation-name`) has been added to specify + that compilation should fail if the compiler generates an ambiguous relation name. +- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the + linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of + database completeness) for C++ projects where this part of database creation consumes + too much memory. This option is available in the commands `database create`, + `database finalize`, `database import`, `dataset import`, `test extract`, and + `test run`. +- The CodeQL language server now provides basic support for Rename, and you can + now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The + current Rename support is less a refactoring tool and more a labor-saving + device. You may have to perform some manual edits after using Rename, but it + should still be faster and less work than renaming a symbol manually. +- `codeql database analyze` now defaults to include markdown query help for all custom + queries with help files available. To change the default behaviour you can pass the + new flag `--sarif-include-query-help`, which provides the options `always` (which + includes query help for all queries), `custom_queries_only` (the default) and `never` + (which does not include query help for any query). The existing flag + `--sarif-add-query-help` has been deprecated and will be removed in a future release. + +### Improvements + +- The Find References feature in the CodeQL language server now supports all + CodeQL identifiers and offers improved performance compared to CodeQL CLI + 2.14 releases. +- The compiler generates shorter human-readable DIL and RA relation names. Due + to use of an extended character set, full VS Code support for short relation + names requires VS Code extension 1.9.4 or newer. +- `codeql database create` and `codeql database finalize` now log more diagnostic + information during database finalization, including the size of each relation, their + total size, and the rate at which they were written to disk. + +### Bugs fixed + +- Fixed an internal error in the compiler when arguments to the `codePointCount` string + primitive were not bound. +- Fixed a bug where `codeql database finalize` would fail if a + database under construction was moved between machines between + `codeql database init` and `codeql database finalize`. This should + now work, as long as both commands are run by the same _release_ of + the CodeQL CLI and the extractors used are the ones bundled with the + CLI. +- Fixed a bug where `codeql database run-queries` would fail in some + circumstances when the database path included an `@`. ## Release 2.15.2 (2023-11-13) @@ -65,7 +113,7 @@ ### Deprecations -- The accepted values of the `codeql database cleanup --mode=` command line option have been renamed to bring them in line with what they are called in the VSCode extension and the query server: +- The accepted values of the `--mode` option for `codeql database cleanup` have been renamed to bring them in line with what they are called in the VSCode extension and the query server: - `--mode=brutal` is now `--mode=clear`. - `--mode=normal` is now `--mode=trim`. - `--mode=light` is now `--mode=fit`. @@ -74,6 +122,9 @@ ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. +- The syntax of DIL now more closely resembles the QL source code that it is + compiled from. In particular, conjunctions and disjunctions now use the + familiar `and` and `or` keywords, and clauses are enclosed in curly braces. ### Bugs fixed From 21314e357e71c63c41f1a5e753bd6d95717fdbeb Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 11 Dec 2023 08:42:46 -0800 Subject: [PATCH 02/54] Revise changelog for 2.15.3 --- CHANGELOG.md | 49 ++++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 059feae..d727e50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,34 +20,31 @@ ### New features -- A new compilation flag (`--fail-on-ambiguous-relation-name`) has been added to specify - that compilation should fail if the compiler generates an ambiguous relation name. -- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the - linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of - database completeness) for C++ projects where this part of database creation consumes - too much memory. This option is available in the commands `database create`, - `database finalize`, `database import`, `dataset import`, `test extract`, and - `test run`. -- The CodeQL language server now provides basic support for Rename, and you can - now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The - current Rename support is less a refactoring tool and more a labor-saving - device. You may have to perform some manual edits after using Rename, but it - should still be faster and less work than renaming a symbol manually. - `codeql database analyze` now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag `--sarif-include-query-help`, which provides the options `always` (which includes query help for all queries), `custom_queries_only` (the default) and `never` (which does not include query help for any query). The existing flag `--sarif-add-query-help` has been deprecated and will be removed in a future release. +- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the + linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of + database completeness) for C++ projects where this part of database creation consumes + too much memory. This option is available in the commands `database create`, + `database finalize`, `database import`, `dataset import`, `test extract`, and + `test run`. +- The CodeQL language server now provides basic support for Rename, and you can now use + the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename + support is less a refactoring tool and more a labor-saving device. You may have to + perform some manual edits after using Rename, but it should still be faster and less + work than renaming a symbol manually. ### Improvements -- The Find References feature in the CodeQL language server now supports all - CodeQL identifiers and offers improved performance compared to CodeQL CLI - 2.14 releases. -- The compiler generates shorter human-readable DIL and RA relation names. Due - to use of an extended character set, full VS Code support for short relation - names requires VS Code extension 1.9.4 or newer. +- The Find References feature in the CodeQL language server now supports all CodeQL + identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. +- The compiler generates shorter human-readable DIL and RA relation names. Due to use + of an extended character set, full VS Code support for short relation names requires + VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. @@ -56,14 +53,12 @@ - Fixed an internal error in the compiler when arguments to the `codePointCount` string primitive were not bound. -- Fixed a bug where `codeql database finalize` would fail if a - database under construction was moved between machines between - `codeql database init` and `codeql database finalize`. This should - now work, as long as both commands are run by the same _release_ of - the CodeQL CLI and the extractors used are the ones bundled with the - CLI. -- Fixed a bug where `codeql database run-queries` would fail in some - circumstances when the database path included an `@`. +- Fixed a bug where `codeql database finalize` would fail if a database under construction + was moved between machines between `codeql database init` and `codeql database finalize`. + This should now work, as long as both commands are run by the same _release_ of the + CodeQL CLI and the extractors used are the ones bundled with the CLI. +- Fixed a bug where `codeql database run-queries` would fail in some circumstances when + the database path included an `@`. ## Release 2.15.2 (2023-11-13) From 51e566086be89a90231de896d9ed5d3b2f78566c Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 11 Dec 2023 08:43:38 -0800 Subject: [PATCH 03/54] Update changelog for 2.15.4 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d727e50..9e9af2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,14 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.15.4 (2023-12-11) + +### Improvements + +- Parallelism in the evaluator has been improved, resulting in faster analysis when + running with many threads, particularly for large databases. + ## Release 2.15.3 (2023-11-22) ### New features From e67d1c08b78afc431211f5907e83d366691d6337 Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Wed, 20 Dec 2023 08:57:23 -0800 Subject: [PATCH 04/54] Update changelog for 2.15.5 --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e9af2a..7c1e451 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,29 @@ you know what to do). --> +## Release 2.15.5 (2023-12-20) + +### New features + +- A new extractor option has been added to the JavaScript/TypeScript extractor. + Set the environment variable `CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES` + to `true` to skip the extraction of types in TypeScript files. + Use this to speed up extraction if your codebase has a high volume of + TypeScript type information that causes a noticeable bottleneck for + TypeScript extraction. The majority of analysis results should be preserved + even when no types are extracted. + +### Bugs fixed + +- Fixed an issue where CodeQL would sometimes incorrectly report that no files + were scanned when running on Windows. + This affected the human-readable summary produced by `codeql database analyze` + and `codeql database interpret-results`, but did not impact the file coverage + information produced in the SARIF output and displayed on the tool status page. +- When analyzing Swift codebases, CodeQL build tracing will now ignore the + `codesign` tool. This prevents errors in build commands or workflows on macOS + that include both CodeQL and code signing. + ## Release 2.15.4 (2023-12-11) ### Improvements From 1d24865e275defb303f35f4be496e780fbf48a5c Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Fri, 5 Jan 2024 14:23:59 +0000 Subject: [PATCH 05/54] Add change note re Java 21 support in 2.15.4 --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c1e451..2ed17bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,11 @@ ## Release 2.15.4 (2023-12-11) +### New features + +- Java 21 is now fully supported, including support for new language features such as + pattern switches and record patterns. + ### Improvements - Parallelism in the evaluator has been improved, resulting in faster analysis when From fb86fb9d41b02ee0c7de1dda971ffbc1a0a3e442 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 16 Jan 2024 13:00:26 +0000 Subject: [PATCH 06/54] Update CHANGELOG.md for 2.16.0 --- CHANGELOG.md | 51 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ed17bc..1290688 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,52 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.16.0 (2024-01-16) + +### New Features + +- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. + +### Improvements + +- The `Experimental` flag has been removed from all packaging and related commands. +- The RA pretty-printer omits names of internal RA nodes and pretty-prints + binary unions with nested internal unions as n-ary unions. VS Code extension + v1.11.0 or newer is required to compute join order badness metrics in VS Code + for the new RA format. + + +### Potentially breaking changes + +- The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting `CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1`. +- The `--ram` option to `codeql database run-queries` and other + commands that execute queries is now interpreted more strictly. + Previously it was mostly a rough hint for how much memory to use, + and the actual memory footprint of the CodeQL process could be + hundreds of megabytes higher. From this release, CodeQL tries harder + to keep its _total_ memory consumption during evaluation below the + given limit. + + The new behavior yields more predictable memory use, but since it + works by allocating less RAM, it can lead to more use of _disk_ + storage for intermediate results compared to earlier releases with + the same `--ram` value, and consequently a slight performance + loss. In rare cases, for large databases, analysis may fail with a + Java `OutOfMemoryError`. + + The cure for this is to increase `--ram` to be closer to the amount + of memory actually available for CodeQL. As a rule of thumb, it will + usually be possible to increase the value of `--ram` by 700 MB or + more, without actually using more resources than release 2.15.x + would with the old setting. An exact amount cannot stated, however, + since the actual memory footprint in earlier releases depended on + factors such as the size of the databases that were not fully taken + into account. + + If you use the CodeQL Action, you do not need to do anything unless + you have manually overridden the Action's RAM setting. The Action + will automatically select a `--ram` setting that matches the version + of the CLI it uses. ## Release 2.15.5 (2023-12-20) @@ -33,7 +79,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the @@ -44,8 +90,7 @@ ### New features -- Java 21 is now fully supported, including support for new language features such as - pattern switches and record patterns. +- Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns. ### Improvements From 03ea17bd85e4e9f8d0a0ee16be392a1a055344a7 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Thu, 25 Jan 2024 13:57:50 +0000 Subject: [PATCH 07/54] Update changelog for 2.16.1 --- CHANGELOG.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1290688..4531ba6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,22 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.16.1 (2024-01-25) + +### Improvements + +- When executing the `codeql database init` command, the CodeQL runner + executable path is now stored in the `CODEQL_RUNNER` environment variable. + Users of indirect tracing on MacOS with System Integrity Protection enabled + who previously had trouble with indirect tracing should prefix their build + command with this path. For example, `$CODEQL_RUNNER build.sh`. + +### QL language improvements + +- Name clashes between weak aliases (i.e. aliases that are not final aliases of + non-final entities) of the same target no longer cause ambiguity errors. + ## Release 2.16.0 (2024-01-16) ### New Features From 0a621d68fb21b5a9bdfc90b4313da47df74b6b00 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Mon, 12 Feb 2024 13:17:10 -0500 Subject: [PATCH 08/54] Update chanelog for 2.16.2 --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4531ba6..7eac2fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.16.2 (2024-02-12) + +- There are no user-facing changes in this release. + ## Release 2.16.1 (2024-01-25) ### Improvements From f6b18e6381860d453f6d3bff8de4f99eeb0a218b Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 22 Feb 2024 11:39:00 +0000 Subject: [PATCH 09/54] Update CHANGELOG.md for release 2.16.3 --- CHANGELOG.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7eac2fb..6042687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,32 @@ you know what to do). --> +## Release 2.16.3 (2024-02-22) + +### Security patches + +- Fixes CVE-2024-25129, a limited data exfiltration vulnerability that + could be triggered by untrusted databases or QL packs. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gf8p-v3g3-3wph) + for more information. + +### New Features + +- A new extractor option has been added to the Python extractor. + Set the new extractor option `python_executable_name` or the environment variable + `CODEQL_EXTRACTOR_PYTHON_OPTION_PYTHON_EXECUTABLE_NAME` to one of `py`, `python` or `python3` + to override the default Python executable search and selection behavior of the Python extractor. + For example, on Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. Setting this extractor option or environment variable allows + overriding this behavior to look for a different name like `python` or `python3`. + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + +### Bugs fixed + +- Fixed a bug where CodeQL may produce an invalid database when it exhausts + all available ID numbers. Now it detects the condition and reports an + error instead. + ## Release 2.16.2 (2024-02-12) - There are no user-facing changes in this release. @@ -99,7 +125,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From d6584c92c977a8553f4368fb38479a69335159b3 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 22 Feb 2024 11:43:27 +0000 Subject: [PATCH 10/54] Replace nbsp with regular space --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6042687..7e3c5f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -125,7 +125,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 6ffe4091e50dc73fa58bdcbc1837de13131702e8 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 22 Feb 2024 14:20:21 +0100 Subject: [PATCH 11/54] edit 2.16.3 changelog to explain how to set the extractor option --- CHANGELOG.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e3c5f7..490ae55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,12 +29,17 @@ ### New Features - A new extractor option has been added to the Python extractor. - Set the new extractor option `python_executable_name` or the environment variable - `CODEQL_EXTRACTOR_PYTHON_OPTION_PYTHON_EXECUTABLE_NAME` to one of `py`, `python` or `python3` - to override the default Python executable search and selection behavior of the Python extractor. - For example, on Windows machines, the Python extractor will expect to find `py.exe` on the - system `PATH` by default. Setting this extractor option or environment variable allows - overriding this behavior to look for a different name like `python` or `python3`. + Pass one of `--extractor-option python_executable_name=py` + or `--extractor-option python_executable_name=python` + or `--extractor-option python_executable_name=python3` + to `codeql database create` (or `codeql database trace-command` or, + for indirect tracing, `codeql database init`) to override the default + Python executable search and selection behavior of the Python + extractor. For example, on Windows machines, the Python extractor + will expect to find `py.exe` on the system `PATH` by default. + Setting this extractor option or environment variable allows + overriding this behavior to look for a different name. + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -125,7 +130,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 8b68ee28b278049d2233fe756cb8f934830f0add Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Thu, 29 Feb 2024 13:54:49 +0000 Subject: [PATCH 12/54] update CLI changelog to match unified changelog --- CHANGELOG.md | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 490ae55..aa39eed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,19 +28,14 @@ ### New Features -- A new extractor option has been added to the Python extractor. - Pass one of `--extractor-option python_executable_name=py` - or `--extractor-option python_executable_name=python` - or `--extractor-option python_executable_name=python3` - to `codeql database create` (or `codeql database trace-command` or, - for indirect tracing, `codeql database init`) to override the default - Python executable search and selection behavior of the Python - extractor. For example, on Windows machines, the Python extractor - will expect to find `py.exe` on the system `PATH` by default. - Setting this extractor option or environment variable allows - overriding this behavior to look for a different name. - - More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). +- A new extractor option has been added to the Python extractor: `python_executable_name`. + You can use this option to override the default process the extractor uses to find and select a Python executable. + Pass one of `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. + If the Python executable has a different name, you can set the new extractor option to override this value and look for `python.exe` or `python3.exe`. + + For more information about using the extractor option with the CodeQL CLI, see [Extractor options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed From 68c904ca588f6cc12dd774bbafa25497af36dd8b Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Fri, 1 Mar 2024 11:39:10 +0000 Subject: [PATCH 13/54] word wrapped at 80 chars --- CHANGELOG.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa39eed..3d13adb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,14 +28,22 @@ ### New Features -- A new extractor option has been added to the Python extractor: `python_executable_name`. - You can use this option to override the default process the extractor uses to find and select a Python executable. - Pass one of `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for example: `codeql database create`. - - On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. - If the Python executable has a different name, you can set the new extractor option to override this value and look for `python.exe` or `python3.exe`. - - For more information about using the extractor option with the CodeQL CLI, see [Extractor options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). +- A new extractor option has been added to the Python extractor: + `python_executable_name`. You can use this option to override the default + process the extractor uses to find and select a Python executable. Pass one of + `--extractor-option python_executable_name=py` or `--extractor-option + python_executable_name=python` or `--extractor-option + python_executable_name=python3` to commands that run the extractor, for + example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. If the Python executable has a different name, you + can set the new extractor option to override this value and look for + `python.exe` or `python3.exe`. + + For more information about using the extractor option with the CodeQL CLI, see + [Extractor + options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed From 29581c71881c943e103e1590ad04f94cb131bfdc Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Mon, 11 Mar 2024 10:41:31 -0700 Subject: [PATCH 14/54] Update CHANGELOG.md for release 2.16.4 --- CHANGELOG.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d13adb..13070ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,39 @@ you know what to do). --> +## Release 2.16.4 (2024-03-11) + +### Potentially breaking changes + +- A number of internal command line options (`--builtin_functions_file`, `--clang_builtin_functions`, + `--disable-objc-default-synthesize-properties`, `--list_builtin_functions`, `--memory-limit-bytes`, + `--mimic_config`, and `--objc`) has been removed from the C/C++ extractor. It has never been + possible to pass these options through the CLI itself, but some customers with advanced setups may + have been passing them through internal undocumented interfaces. All of the removed options were + already no-ops, and will now generate errors. + + The `--verbosity` command line option has also been removed. The option was an alias for + `--codeql-verbosity`, which should be used instead. + +### Improvements + +- The frontend of the C/C++ extractor has been updated, improving the + extractor's reliability and increasing its ability to extract source code. + +### Bugs fixed + +- When parsing user-authored YAML files such as `codeql-pack.yml`, + `qlpack.yml`, `codeql-workspace.yml`, and any YAML file defining a data + extension, unquoted string values starting with a `*` character are now + correctly interpreted as YAML aliases. Previously, they were interpreted + as strings, but with the first character skipped. + + If you see a parse error similar to `while scanning an alias... unexpected` + `character found *(42)`,it likely means that you need to add quotes around + the indicated string value. The most common cause is unquoted glob patterns + that start with `*`, such as `include: **/*.yml`, which will need to be + quoted as `include: "**/*.yml"`. + ## Release 2.16.3 (2024-02-22) ### Security patches From 77e4d7fdcae8722662c94951f81886c4612e0491 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 21 Mar 2024 13:15:30 +0100 Subject: [PATCH 15/54] Update CHANGELOG.md for release 2.16.5 --- CHANGELOG.md | 54 ++++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13070ae..41532b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.16.5 (2024-03-21) + +### New features + +- Beta support has been added for analyzing Java codebases without needing a working build. To enable + this, pass the `--build-mode none` option to `codeql database create`. + ## Release 2.16.4 (2024-03-11) ### Potentially breaking changes @@ -61,22 +68,19 @@ ### New Features -- A new extractor option has been added to the Python extractor: - `python_executable_name`. You can use this option to override the default - process the extractor uses to find and select a Python executable. Pass one of - `--extractor-option python_executable_name=py` or `--extractor-option - python_executable_name=python` or `--extractor-option - python_executable_name=python3` to commands that run the extractor, for - example: `codeql database create`. - - On Windows machines, the Python extractor will expect to find `py.exe` on the - system `PATH` by default. If the Python executable has a different name, you - can set the new extractor option to override this value and look for - `python.exe` or `python3.exe`. +- A new extractor option has been added to the Python extractor. + Pass one of `--extractor-option python_executable_name=py` + or `--extractor-option python_executable_name=python` + or `--extractor-option python_executable_name=python3` + to `codeql database create` (or `codeql database trace-command` or, + for indirect tracing, `codeql database init`) to override the default + Python executable search and selection behavior of the Python + extractor. For example, on Windows machines, the Python extractor + will expect to find `py.exe` on the system `PATH` by default. + Setting this extractor option or environment variable allows + overriding this behavior to look for a different name. - For more information about using the extractor option with the CodeQL CLI, see - [Extractor - options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -107,7 +111,7 @@ ### New Features -- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. +- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. ### Improvements @@ -115,7 +119,7 @@ - The RA pretty-printer omits names of internal RA nodes and pretty-prints binary unions with nested internal unions as n-ary unions. VS Code extension v1.11.0 or newer is required to compute join order badness metrics in VS Code - for the new RA format. + for the new RA format. ### Potentially breaking changes @@ -212,7 +216,7 @@ identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. - The compiler generates shorter human-readable DIL and RA relation names. Due to use of an extended character set, full VS Code support for short relation names requires - VS Code extension 1.9.4 or newer. + VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. @@ -242,7 +246,7 @@ - `codeql database analyze` and `codeql database interpret-results` can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the `GITHUB_SERVER_URL` and `GITHUB_REPOSITORY` environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server - version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. + version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. - CodeQL now supports distinguishing file coverage information between related languages C and C++, Java and Kotlin, and JavaScript and TypeScript. By default, file coverage information for each @@ -591,7 +595,7 @@ member predicates that had stronger binding sets than their root definitions. - Fixed a bug where a query could not be run from VS Code - when there were packs nested within sibling directories + when there were packs nested within sibling directories of the query. ## Release 2.13.2 @@ -617,7 +621,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6. @@ -693,7 +697,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. @@ -701,7 +705,7 @@ This release was skipped. causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. - Customers who wish to use CodeQL 2.12.6 or 2.13.0 can + Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing `--no-sarif-include-diagnostics` to any invocations of `codeql database analyze` or `codeql database interpret-results`. @@ -867,8 +871,8 @@ This release was skipped. `codeql database create` now accounts for [`paths` and `paths-ignore` configuration](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#specifying-directories-to-scan). - In the VS Code extension, recursive calls will be marked with inlay - hints. These can be disabled with the global inlay hints setting - (`editor.inlayHints.enabled`). If you just want to disable them for + hints. These can be disabled with the global inlay hints setting + (`editor.inlayHints.enabled`). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id is `ql`). See [Language Specific Editor Settings](https://code.visualstudio.com/docs/getstarted/settings#_language-specific-editor-settings) in the VS Code documentation for more information. From d9c8ca50e6f1186210bd2764f1f2b11e20e7f79d Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 26 Mar 2024 12:19:01 +0000 Subject: [PATCH 16/54] Update changelog for 2.16.6 --- CHANGELOG.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 41532b7..ff062a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.16.6 (2024-03-26) + +### Bugs fixed + +- Fixes a bug where extractor logs would be output at a lower than expected + verbosity level when using the `codeql database create` command. + ## Release 2.16.5 (2024-03-21) ### New features @@ -170,7 +177,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 7434aae9f5e2a2de1212e532fb6e993332476e8b Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 4 Apr 2024 08:26:56 -0700 Subject: [PATCH 17/54] Update CHANGELOG.md for release 2.17.0 --- CHANGELOG.md | 57 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ff062a5..d8a6f86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,34 @@ you know what to do). --> +## Release 2.17.0 (2024-04-04) + +### Deprecations + +- The `--[no-]analysis-summary-v2` and `--[no-]new-analysis-summary` options + that were used to enable (or disable) improved summary information printed at + the end of a `codeql database analyze` invocation are no longer supported. + Improved summary information is now enabled for all invocations. +- Support for overwriting default CodeQL SARIF run properties using the + `--sarif-run-property` command line option has been removed. This removes the + ability to overwrite the `semmle.formatSpecifier`, `metricResults`, and + `codeqlConfigSummary` properties in the SARIF run file. + +### Improvements + +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now performs better in low-memory situations. (Put another way, it now needs + less RAM to achieve the same performance as before.) + +- The worst-case performance of transitive closure computation (using + the `+` or `*` postfix operators or the `fastTC` higher-order + primitive in QL) has been greatly improved. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.2. + ## Release 2.16.6 (2024-03-26) ### Bugs fixed @@ -75,19 +103,22 @@ ### New Features -- A new extractor option has been added to the Python extractor. - Pass one of `--extractor-option python_executable_name=py` - or `--extractor-option python_executable_name=python` - or `--extractor-option python_executable_name=python3` - to `codeql database create` (or `codeql database trace-command` or, - for indirect tracing, `codeql database init`) to override the default - Python executable search and selection behavior of the Python - extractor. For example, on Windows machines, the Python extractor - will expect to find `py.exe` on the system `PATH` by default. - Setting this extractor option or environment variable allows - overriding this behavior to look for a different name. +- A new extractor option has been added to the Python extractor: + `python_executable_name`. You can use this option to override the default + process the extractor uses to find and select a Python executable. Pass one of + `--extractor-option python_executable_name=py` or `--extractor-option + python_executable_name=python` or `--extractor-option + python_executable_name=python3` to commands that run the extractor, for + example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. If the Python executable has a different name, you + can set the new extractor option to override this value and look for + `python.exe` or `python3.exe`. - More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + For more information about using the extractor option with the CodeQL CLI, see + [Extractor + options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -177,7 +208,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 9a440a9881f4f831d4a60fae9b7cdaa300f71dcb Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Wed, 24 Apr 2024 16:35:27 +0100 Subject: [PATCH 18/54] Update CHANGELOG.md --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d8a6f86..3ab0aac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,26 @@ you know what to do). --> +## Release 2.17.1 (2024-04-24) + +### Deprecations + +- The `--mode` option and `-m` alias to `codeql database create`, + `codeql database cleanup`, and `codeql dataset cleanup` has been + deprecated. Instead, use the new `--cache-cleanup` option, which has + identical behavior. + +### Improvements + +- Improved the diagnostic message produced when no code is processed + when creating a database. If a build mode was specified using + `--build-mode`, the message is now tailored to your build mode. + +### Miscellaneous + +- The `scc` tool used by the CodeQL CLI to calculate source code baseline + information has been updated to version [3.2.0](https://github.com/boyter/scc/releases/tag/v3.2.0). + ## Release 2.17.0 (2024-04-04) ### Deprecations From 6f3cb28e2eab62434cd376944bcfadf70e1f4371 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 7 May 2024 12:16:02 +0100 Subject: [PATCH 19/54] Update changelog for 2.17.2 --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ab0aac..df9ef16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,34 @@ you know what to do). --> +## Release 2.17.2 (2024-05-07) + +### Known issues + +- The beta support for analyzing Swift in this release and all + previous releases requires `g++-13` when running on Linux. Users + analyzing Swift using the `ubuntu-latest`, `ubuntu-22.04`, or + `ubuntu-20.04` runner images for GitHub Actions should update their + workflows to install `g++-13`. For more information, see [the runner + images + announcement](https://github.com/actions/runner-images/issues/9679). + +### Improvements + +- When uploading a SARIF file to GitHub using `codeql github + upload-results`, the CodeQL CLI now waits for the file to be + processed by GitHub. If any errors occurred during processing of the + analysis results, the command will log these and return a non-zero + exit code. To disable this behaviour, pass the + `--no-wait-for-processing` flag. + + By default, the command will wait for the SARIF file to be processed + for a maximum of 2 minutes, however this is configurable with the + `--wait-for-processing-timeout` option. +- The build tracer is no longer enabled when using the [`none` build + mode](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes) + to analyze a compiled language, thus improving performance. + ## Release 2.17.1 (2024-04-24) ### Deprecations From 0b00cc963e1fbe20fdc4c9dbea1661f497fdb97a Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 17 May 2024 12:57:07 -0400 Subject: [PATCH 20/54] Update CHANGELOG.md --- CHANGELOG.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df9ef16..3a19eb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,21 @@ you know what to do). --> +## Release 2.17.3 (2024-05-17) + +### Improvements + +- The language server that our IDE integration is built on now defaults + to fine-grained dependency tracking for incremental error-checking + after file changes. This slightly improves the latency of refreshing + errors after local source code edits and will enable significant + speedups in the future. +- We now properly handle globs (such as `folder/**/*.py`) in `paths` configuration + to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan). +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now supports allocating 2^32 IDs during the import process. The previous limit + was 2^31 IDs. + ## Release 2.17.2 (2024-05-07) ### Known issues @@ -256,7 +271,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 5bdc1ca4dde0776cbc72994f0158349628dfc2b3 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 17 May 2024 12:58:29 -0400 Subject: [PATCH 21/54] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a19eb6..f549eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -271,7 +271,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From df51e673119ee0578d9cb4b2c01661f0c7c5e2ff Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 3 Jun 2024 09:01:05 -0700 Subject: [PATCH 22/54] Update CHANGELOG.md for release 2.17.4 --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f549eb2..29c516c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.17.4 (2024-06-03) + +- There are no user-facing changes in this release. + ## Release 2.17.3 (2024-05-17) ### Improvements From a10394a868747eba71cc617087f600f3878e03f5 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 12 Jun 2024 17:58:19 +0100 Subject: [PATCH 23/54] Update CHANGELOG.md for 2.17.5 --- CHANGELOG.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29c516c..6657c0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,63 @@ you know what to do). --> +## Release 2.17.5 (2024-06-12) + +### Breaking changes + +- All the commands that output SARIF will output a minified version to reduce the size. + The `codeql database analyze`, `codeql database interpret-results`, `codeql generate query-help`, and `codeql bqrs interpret` commands support the option `--no-sarif-minify` to output a pretty printed SARIF file. + +- A number of breaking changes have been made to the `semmle-extractor-options` + functionality available for C and C++ CodeQL tests. + + - The Arm, Intel, and CodeWarrior compilers are no longer supported and the + `--armcc`, `--intel`, `--codewarrior` flags are now ignored, as are all the + flags that only applied to those compilers. + - The `--threads` and `-main-file-name` options, which did not have any effect + on tests, are now ignored. Any specification of these options as part of + `semmle-extractor-options` should be removed. + - Support for `--linker`, all flags that would only invoke the preprocessor, + and the `/clr` flag have been removed, as those flags would never produce any + usable test output. + - Support for the `--include_path_environment` flag has been removed. All include + paths should directly be specified as part of `semmle-extractor-options`. + - Microsoft C/C++ compiler response files specified via `@some_file_name` are + now ignored. Instead, all options should directly be specified as part of + `semmle-extractor-options`. + - Support for Microsoft `#import` preprocessor directive has been removed, as + support depends on the availability of the Microsoft C/C++ compiler, and + availability cannot be guaranteed on all platforms while executing tests. + - Support for the Microsoft `/EHa`, `/EHs`, `/GX`, `/GZ`, `/Tc`, `/Tp`, and `/Zl` + flags, and all `/RTC` flags have been removed. Any specification of these + options as part of `semmle-extractor-options` should be removed. + - Support for the Apple-specific `-F` and `-iframework` flags has been removed. + The `-F` flag can still be used by replacing `-F ` by + `--edg -F --edg `. Any occurrence of `-iframework ` should be + replaced by `--edg --sys_framework --edg `. + - Support for the `/TC`, `/TP`, and `-x` flags has been removed. Please ensure + all C, respectively C++, source files have a `.c`, respectively `.cpp`, + extension. + - The `--build_error_dir`, `-db`, `--edg_base_dir`, `--error_limit`, + `--src_archive`, `--trapfolder`, and `--variadic_macros` flags are now ignored. + + The above changes do not affect the creation of databases through the CodeQL CLI, + or when calling the C/C++ extractor directly with the `--mimic` or `--linker` flags. + Similar functionality continues to be supported in those scenarios, except for + CodeWarrior and the `--edg_base_dir`, `--include_path_environment`, `/Tc`, and `/Tp` + flags, which were never supported. + +### Improvements + +- `codeql generate log-summary` now reports completed pipeline runs that + are part of an incomplete recursive predicate. + +### Miscellaneous + +- The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal + documentation generation commands has been updated to version + [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). + ## Release 2.17.4 (2024-06-03) - There are no user-facing changes in this release. From ea99a263394282a49dc3a4ecacf55e738857df88 Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Tue, 18 Jun 2024 13:06:02 +0100 Subject: [PATCH 24/54] Quick fix to the 2.17.4 changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6657c0d..b44b006 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,6 +77,8 @@ ## Release 2.17.4 (2024-06-03) - There are no user-facing changes in this release. +- CodeQL package management is now generally available, and all GitHub-produced + CodeQL packages have had their version numbers increased to 1.0.0. ## Release 2.17.3 (2024-05-17) From d00e3aa63972f359ff26ced7f2cb6717d46e7a89 Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Tue, 18 Jun 2024 13:29:18 +0100 Subject: [PATCH 25/54] Update CHANGELOG.md --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b44b006..dc07eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,7 +76,8 @@ ## Release 2.17.4 (2024-06-03) -- There are no user-facing changes in this release. +### New features + - CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0. From ae80a68683e54c5005e12666fab8cf3a7f5e2e91 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 27 Jun 2024 19:26:34 +0200 Subject: [PATCH 26/54] Update CHANGELOG.md --- CHANGELOG.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc07eb3..8bfe960 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,17 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.17.6 (2024-06-27) + +### New features + +- Beta support is now available for analyzing C# codebases without needing a working build. To use + this, pass the `--build-mode none` option to `codeql database create`. + +### Improvements + +- The `--model-packs` option is now publicly available. This option allows commands like `codeql database analyze` + to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis. ## Release 2.17.5 (2024-06-12) @@ -335,7 +346,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 95cac33e9326a179b5d421872eac329b0b05eddb Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Thu, 11 Jul 2024 11:22:04 +0200 Subject: [PATCH 27/54] Add entry for CLI v2.18.0 --- CHANGELOG.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bfe960..0d02c47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,72 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.18.0 (2024-07-11) + +### Breaking changes + +- A number of breaking changes have been made to the C and C++ CodeQL + environment: + - The environment no longer defines any GNU-specific builtin macros. + If these macros are still needed, please define them via + `semmle-extractor-options`. + - The `--force-recompute` option is no longer directly supported by + `semmle-extractor-options`. Instead, `--edg --force-recompute` + should be specified. + - The `--gnu_version` and `--microsoft_version` options that can be + specified via `semmle-extractor-options` are now synonyms, and only + one should be specified as part of `semmle-extractor-options`. + Furthermore, is also no longer possible to specify these options + via the following syntax. + + - `--edg --gnu_version --edg `, and + - `--edg --microsoft_version --edg ` + + The shorter `--gnu_version ` and + `--microsoft_version ` should be used. +- The `--build_error_dir` and `--predefined_macros` command line options + have been removed from the C/C++ extractor. It has never been possible + to pass these options through the CLI, but some customers with advanced + setups may have been passing them through internal undocumented interfaces. + Passing the option `--build_error_dir` did not have any effect, and it + is safe to remove the option. The `--predefined_macros` option should + have been unnecessary, as long as the extractor was invoked with the + `--mimic` option. + +### Regressions + +- Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release. + +### Improvements + +- Introduced the `--include-logs` option to the `codeql database bundle` + command. This new feature allows users to include logs in the generated + database bundle, allowing for a more complete treatment of the bundle, and + bringing the tool capabilities up-to-speed with the documentation. +- `codeql database init` and `codeql database create` now support the + `--force-overwrite` option. When this option is specified, the command will + delete the specified database directory even if it does not look like a + database directory. This option is only recommended for automation. For + directcommand line commands, it is recommended to use the `--overwrite` + option, which includes extra protection and will refuse to delete a + directory that does not look like a database directory. +- Extract `.xsaccess`, `*.xsjs` and `*.xsjslib` files for SAP HANA XS as + Javascript. +- We have updated many compiler error messages and warnings to improve their + readability and standardize their grammar. + Where necessary, please use the `--learn` option for the `codeql test run` + command. + +### Bugs fixed + +- Where a MacOS unsigned binary cannot be signed, CodeQL will now continue + trying to trace compiler invocations created by that process and its + children. In particular this means that Bazel builds on MacOS are now + traceable. +- Fixed a bug where test discovery would fail if there is a syntax error in a + qlpack file. Now, a warning message will be printed and discovery will + continue. + ## Release 2.17.6 (2024-06-27) ### New features From 1ff7b456c6f3386336f4b8da93caa2192d5d7ff1 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 25 Jul 2024 13:55:08 -0700 Subject: [PATCH 28/54] Update CHANGELOG.md for release 2.18.1 --- CHANGELOG.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d02c47..9644937 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,57 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.18.1 (2024-07-25) + +# Security Updates + +- Fixes CVE-2024-41807, an arbitrary file overwrite that can be triggered when + using untrusted third-party queries from a git repository. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-x4gx-f2xv-6wj9) + for more information. +- The following dependencies have been updated. These updates include + security fixes in the respective libraries that prevent + out-of-bounds accesses or denial-of-service in scenarios where + untrusted files are processed. These scenarios are not likely to be + encountered in most uses of CodeQL and code scanning, and only + apply to advanced use cases where precompiled query packs, + database ZIP files, or database TRAP files are obtained from + untrusted sources and then processed on a trusted machine. + - airlift/aircompressor is updated to version 0.27. + - Apache Ant is updated to version 1.10.11. + - Apache Commons Compress is updated to version 1.26.0. + - Apache Commons IO is updated to version 2.15.1. + - Apache Commons Lang3 is updated to version 3.14.0. + - jsoup is updated to version 1.15.3. + - Logback is updated to version 1.2.13. + - Snappy is updated to version 0.5. + +### New features + +- The *experimental* type `QlBuiltins::BigInt` of arbitrary-precision integers + has been introduced. To opt in to this API, compile your queries with + `--allow-experimental=bigint`. Big integers can be constructed using the + `.toBigInt()` methods of `int` and `string`. The built-in operations are: + - comparisons: `=`, `!=`, `<`, `<=`, `>`, `>=`, + - conversions: `.toString()`, `.toInt()`, + - arithmetic: binary `+`, `-`, `*`, `/`, `%`, unary `-`, + - bitwise operations: `.bitAnd(BigInt)`, `.bitOr(BigInt)`, + `.bitXor(BigInt)`, `.bitShiftLeft(int)`, `.bitShiftRightSigned(int)`, + `.bitNot()`, + - aggregates: `min`, `max`, (`strict`)`sum`, (`strict`)`count`, `avg`, + `rank`, `unique`, `any`. + - other: `.pow(int)`, `.abs()`, `.gcd(BigInt)`, `.minimum(BigInt)`, + `.maximum(BigInt)`. +- `codeql test run` now supports postprocessing of test results. When .qlref + files specify a path to a `postprocess` query, then this is evaluated after + the test query to transform the test outputs prior to concatenating them into + the `actual` results. + +### Improvements + +- The 30% QL query compilation slowdown noted in 2.18.0 has been fixed. + ## Release 2.18.0 (2024-07-11) ### Breaking changes @@ -412,7 +463,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From d7a1c17cba45b80adb479801425fe06452b8b1a0 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Fri, 26 Jul 2024 09:02:49 -0700 Subject: [PATCH 29/54] Update file overwrite CVE reference --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9644937..e593e5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,10 +19,11 @@ ## Release 2.18.1 (2024-07-25) -# Security Updates +### Security Updates -- Fixes CVE-2024-41807, an arbitrary file overwrite that can be triggered when - using untrusted third-party queries from a git repository. See the +- Resolves CVE-2023-4759, an arbitrary file overwrite in Eclipse JGit + that can be triggered when using untrusted third-party queries from a + git repository. See the [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-x4gx-f2xv-6wj9) for more information. - The following dependencies have been updated. These updates include From 40a98af252ec97eea890070477622cfe6d115731 Mon Sep 17 00:00:00 2001 From: Jeroen Ketema Date: Fri, 26 Jul 2024 23:25:21 +0200 Subject: [PATCH 30/54] Fix change note to clarify that C/C++ breaking changes related to tests only --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e593e5d..1ca422e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -73,10 +73,10 @@ ### Breaking changes - A number of breaking changes have been made to the C and C++ CodeQL - environment: - - The environment no longer defines any GNU-specific builtin macros. - If these macros are still needed, please define them via - `semmle-extractor-options`. + test environment as used by `codeql test run`: + - The test environment no longer defines any GNU-specific builtin + macros. If these macros are still needed by a test, please define + them via `semmle-extractor-options`. - The `--force-recompute` option is no longer directly supported by `semmle-extractor-options`. Instead, `--edg --force-recompute` should be specified. From 232a266fb3efeea4fb16b84f2038474c659de258 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 13 Aug 2024 15:00:18 +0100 Subject: [PATCH 31/54] Update CHANGELOG for 2.18.2 --- CHANGELOG.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ca422e..1ee5fbb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,16 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.18.2 (2024-08-13) + +### Deprecations + +- Swift analysis on Ubuntu is no longer supported. Please migrate to macOS if this affects you. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.3. ## Release 2.18.1 (2024-07-25) @@ -464,7 +474,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From b601ce8efdee96a5ea7be7d42bf2a198d10cff7f Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 13 Aug 2024 15:01:52 +0100 Subject: [PATCH 32/54] Reintroduce Non-breaking space --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ee5fbb..5e8b14e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -474,7 +474,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 0e94992f4fbc75184f10caed298c6b93e9182c43 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Wed, 28 Aug 2024 13:46:29 +0100 Subject: [PATCH 33/54] Update changelog for 2.18.3 --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e8b14e..ba87a75 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,11 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.18.3 (2024-08-28) + +- There are no user-facing changes in this release. + ## Release 2.18.2 (2024-08-13) ### Deprecations From 961a1cbafabbf89cc0f3c06924e5361d998b0dcc Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 12 Sep 2024 10:13:15 -0400 Subject: [PATCH 34/54] Update changelog for 2.18.4 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ba87a75..fa3e9ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.18.4 (2024-09-12) + +### New Features + +- C# support for `build-mode: none` is now out of beta, and generally available. +- Go 1.23 is now supported. + ## Release 2.18.3 (2024-08-28) - There are no user-facing changes in this release. From 79b4eacbe967fd3022bfb232feb866ef18efac2a Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 18 Sep 2024 13:50:16 +0100 Subject: [PATCH 35/54] Update CHANGELOG.md for 2.19.0 release --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa3e9ee..b309716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,19 @@ you know what to do). --> +## Release 2.19.0 (2024-09-18) + +### Improvements + +- `codeql database analyze` and `codeql database interpret-results` now support + the `--sarif-run-property` option. You can provide this option when using a SARIF + output format to add a key-value pair to the property bag of the run object. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.4. + ## Release 2.18.4 (2024-09-12) ### New Features From 53387737ddf48c8d0f52d3549afe6db37f819cec Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Fri, 4 Oct 2024 13:49:36 -0700 Subject: [PATCH 36/54] Add CLI v2.19.2 changelog entry --- CHANGELOG.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b309716..a120dd2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,22 @@ you know what to do). --> +## Release 2.19.1 (2024-10-04) + +### New Features + +- The command `codeql generate query-help` now supports Markdown help files. + The Markdown help format is commonly used in custom CodeQL query packs. This new + feature allows us to generate SARIF reporting descriptors for CodeQL queries that + include Markdown help directly from a query Markdown help file. + +- Added a new command, `codeql resolve packs`. This command shows each step in the + pack search process, including what packs were found in each step. With the + `--show-hidden-packs` option, it can also show details on which packs were hidden + by packs found earlier in the search sequence. `codeql resolve packs` is intended + as a replacement for most uses of `codeql resolve qlpacks`, whose output is both + less detailed and less accurate. + ## Release 2.19.0 (2024-09-18) ### Improvements @@ -236,7 +252,7 @@ - The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal documentation generation commands has been updated to version - [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). + [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). ## Release 2.17.4 (2024-06-03) @@ -400,7 +416,7 @@ `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for - example: `codeql database create`. + example: `codeql database create`. On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. If the Python executable has a different name, you From 2e5c33400002650f7159508ab7b75eeb2b582383 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Mon, 21 Oct 2024 19:16:43 +0200 Subject: [PATCH 37/54] Update CHANGELOG.md --- CHANGELOG.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a120dd2..84e3a71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.19.2 (2024-10-21) + +### Potentially breaking changes + +- The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable `CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB` to `1`. + +### Miscellaneous + +- The database relation `sourceLocationPrefix` is changed for databases created with + `codeql test run`. Instead of containing the path of the enclosing qlpack, it now + contains the actual path of the test, similar to if one had run `codeql database create` + on the test folder. For example, for a test such as + `/cpp/ql/test/query-tests/Security/CWE/CWE-611/XXE.qlref` we now populate + `sourceLocationPrefix` with `/cpp/ql/test/query-tests/Security/CWE/CWE-611/` + instead of `/cpp/ql/test/`. This change typically impacts calls to + `File.getRelativePath()`, and may as a result change the expected test output. + ## Release 2.19.1 (2024-10-04) ### New Features @@ -515,7 +532,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From feb1da6eeee2bd20c5e843483cdeb90366a2e46f Mon Sep 17 00:00:00 2001 From: Jonas Jensen Date: Tue, 22 Oct 2024 14:55:07 +0200 Subject: [PATCH 38/54] 2.19.2 changelog entry for evaluator bugfix --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 84e3a71..28360e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,10 @@ - The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable `CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB` to `1`. +### Bugs fixed + +- The 2.19.1 release contained a bug in the query evaluator that under rare conditions could lead to wrong alerts or resource exhaustion. Although we have never seen the problem outside of internal testing, we encourage users on 2.19.1 to upgrade to 2.19.2. + ### Miscellaneous - The database relation `sourceLocationPrefix` is changed for databases created with From fd9e254cb7a7240e4539cb30570e2981eb62b95f Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 7 Nov 2024 09:10:41 -0800 Subject: [PATCH 39/54] Update CHANGELOG.md for 2.19.3 release --- CHANGELOG.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28360e9..d2e161f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,48 @@ you know what to do). --> +## Release 2.19.3 (2024-11-07) + +### Bugs fixed + +- Fixed a bug where using `codeql database import` to combine multiple non-empty + databases may produce a corrupted database. (The bug does not affect using + `codeql database finalize --additional-dbs` to combine multiple databases.) + +- Fixed a bug where uses of a `QlBuiltins::ExtensionId` variable that was not + bound to a value could be incorrectly accepted in some cases. In many cases, + this would result in a crash. + +- CodeQL would sometimes refuse to run with more than around 1,500 GB of RAM + available, complaining that having so much memory was "unrealistic". The + amount of memory CodeQL is able to make any meaningful use of still tops out + at about that value, but it will now gracefully accept that so large + computers do in fact exist. + +- Fixed a bug in command-line parsing where a misspelled option could sometimes + be misinterpreted as, e.g., the name of a query to run. Now every command-line + argument that begins with a dash is assumed to be intended as an option + (unless it comes after the `--` separator), and an appropriate error is + emitted if that is not a recognized one. + + The build command in `codeql database trace-command` is exempted from this for + historical reasons, but we strongly recommend putting a `--` before the entire + build command there, in case a future `codeql` version starts recognizing + options that you intended to be part of the build command. + +### Miscellaneous + +- The CodeQL Bundle is now available as an artifact that is compressed using + [Zstandard](https://en.wikipedia.org/wiki/Zstd). This artifact is + smaller and faster to decompress than the original, gzip-compressed bundle. The CodeQL bundle + is a tar archive containing tools, scripts, and various CodeQL-specific files. + + If you are currently using the CodeQL Bundle, you may want to consider switching to the + Zstandard variant of the bundle. You can download the new form of the CodeQL Bundle from the + [codeql-action releases page](https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.19.3) + by selecting the appropriate bundle with the `.zst` extension. The gzip-compressed bundles will + continue to be available for backwards compatibility. + ## Release 2.19.2 (2024-10-21) ### Potentially breaking changes @@ -536,7 +578,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 5205c9a4af4524e937df3c8a6282afcf16662359 Mon Sep 17 00:00:00 2001 From: Alex Eyers-Taylor Date: Mon, 2 Dec 2024 13:28:12 +0000 Subject: [PATCH 40/54] Add changelof for 2.19.4 --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d2e161f..f80170c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,26 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.19.4 (2024-12-02) + +### Improvements + +- CodeQL now supports passing values containing the equals character (`=`) to extractor options via + the `--extractor-option` flag. This allows cases like `--extractor-option opt=key=value`, which + sets the extractor option `opt` to hold the value `key=value`, whereas previously that would have + been rejected with an error. +- The `codeql pack bundle` command now sets the numeric user and group IDs of entries in the generated + `tar` archive to `0`. This avoids failures like `IllegalArgumentException: user id '7111111' is too big ( > 2097151 )` + when the numeric user ID is too large. + +### Bugs fixed + +- On MacOS, `arch -arm64` commands no longer fail when they are executed via `codeql database create --command`, + via `codeql database trace-command`, or are run after `codeql database init --begin-tracing`. Note + that build commands invoked this way still will not normally be traced, so this is useful only for + running ancillary commands which are incidental to building your code. +- Fixed a bug where `codeql test run` would not preserve test + databases on disk after a test failed. ## Release 2.19.3 (2024-11-07) From 53c6f9592963d1a73bb2ea188d5687b61b6098a6 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Mon, 9 Dec 2024 17:15:25 +0000 Subject: [PATCH 41/54] Update changelog for 2.20.0 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f80170c..58b1ca9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,20 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.0 (2024-12-09) + +### New features + +- The [`QlBuiltins::BigInt` type](https://codeql.github.com/docs/ql-language-reference/modules/#bigint) of + arbitrary precision integers is generally available and no longer hidden behind the + `--allow-experimental=bigint` CLI feature flag. + +### Miscellaneous + +- Backslashes are now escaped when writing output in the Graphviz DOT format (`--format=dot`). +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version + 21.0.5. + ## Release 2.19.4 (2024-12-02) ### Improvements From d83aedf3de84418107b73c51215d699906c605aa Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 10 Dec 2024 14:20:15 +0000 Subject: [PATCH 42/54] Add known issue for Windows exe properties version number --- CHANGELOG.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58b1ca9..f64a0b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,12 @@ --> ## Release 2.20.0 (2024-12-09) +### Known issues + +- The Windows executable for this release is labeled with an incorrect version number + within its properties: the version number should be 2.20.0 rather than 2.19.4. + `codeql version` reports the correct version number. + ### New features - The [`QlBuiltins::BigInt` type](https://codeql.github.com/docs/ql-language-reference/modules/#bigint) of @@ -612,7 +618,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From bc44bf68debfbbed6ace465d6636291545cf6b29 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 9 Jan 2025 16:50:38 -0500 Subject: [PATCH 43/54] Update changelog for 2.20.1 --- CHANGELOG.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f64a0b8..2a1dc7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,19 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.1 (2025-01-09) + +### Improvements + +- Automatic installation of dependencies for C++ autobuild is now supported on Ubuntu 24.04. + +- The CLI will now warn if it detects that it is installed in a + location where it is likely to cause performance issues. This + includes: user home, desktop, downloads, or the file system root. + + You can avoid this warning by setting the `CODEQL_ALLOW_INSTALLATION_ANYWHERE` + environment variable to `true`. + ## Release 2.20.0 (2024-12-09) ### Known issues @@ -618,7 +631,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 1fa874136402ce0bd5ba66d4c0463ef78bd92d4b Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 22 Jan 2025 14:44:29 +0000 Subject: [PATCH 44/54] Update changelog for 2.20.2 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a1dc7b..af98a2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,23 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.2 (2025-01-22) + +### Improvements + +- `codeql database create` and `codeql database finalize` now write relations to disk in a new, compressed format. As a result, databases will generally take up less space on disk, whether zipped or unzipped. Note that databases in this format can only be read and analyzed using CodeQL version 2.20.1 onwards. Attempting to analyze such a database with CodeQL version 2.20.0 or older will fail, with an error message like the following: + ``` + UnsortedExtensionalError: Tuples that were assumed to be in order are not: [123456777, 777654321, 123456777]<[777654321, 123456777, 777654321] + ``` + +### Enhancements + +- Added the `.bitLength()` method to `QlBuiltins::BigInt`. + +### Bugs Fixed + +- Fixed a bug where CodeQL would crash on rare occasions while merging SARIF files before uploading results. + ## Release 2.20.1 (2025-01-09) ### Improvements From 39038de5c1fe27dd47c17f81bcfc5b74eb1f8a6d Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 24 Jan 2025 10:48:58 -0500 Subject: [PATCH 45/54] Changelog for 2.20.3 --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index af98a2b..e9edcbd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,19 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.20.3 (2025-01-24) + +### Security Updates + +- Resolves a security vulnerability where CodeQL databases or logs produced by the CodeQL CLI may contain the environment variables from the time of + database creation. This includes any secrets stored in an environment variables. For more information, see the + [CodeQL CLI security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gqh3-9prg-j95m). + + All users of CodeQL should follow the advice in the CodeQL advisory mentioned above or upgrade to this version or a later version of CodeQL. + + If you are using the CodeQL Action, also see the related [CodeQL Action security advisory](https://github.com/github/codeql-action/security/advisories/GHSA-vqf5-2xx6-9wfm). + ## Release 2.20.2 (2025-01-22) ### Improvements From 9a8c3477ba83a7925ceda48739e6bd3ee3063ce2 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 6 Feb 2025 12:06:35 +0100 Subject: [PATCH 46/54] Update CHANGELOG.md for 2.20.4 --- CHANGELOG.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9edcbd..5e29033 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,21 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.4 (2025-02-06) + +### New features + +- Using the `actions` language (for analysis of GitHub Actions workflows) no longer requires + the `CODEQL_ENABLE_EXPERIMENTAL_FEATURES` environment variable to be set. Support for analysis + of GitHub Actions workflows remains in public preview. + +### Bugs fixed + +- Fixed a bug where CodeQL for Java would fail with an SSL exception while trying to download `maven`. + +### Miscellaneous + +- The build of the [logback-core](https://logback.qos.ch/) library that is used for logging in the CodeQL CLI has been updated to version 1.3.15. ## Release 2.20.3 (2025-01-24) @@ -661,7 +676,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 90a3e2a1b82757157110c9657fedee362c5d5575 Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Thu, 20 Feb 2025 14:30:37 +0000 Subject: [PATCH 47/54] Add 2.20.5 to changelog --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e29033..6192d4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,21 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.5 (2025-02-20) + +### Breaking changes + +- Removed support for `QlBuiltins::BigInt`s in the `avg()` aggregate. + +- A number of breaking changes have been made to the C and C++ CodeQL test environment as used by `codeql test run`: + - The `-Xclang-only=` option is no longer supported by `semmle-extractor-options`. Instead, when either `--clang` or `--clang_version` is specified the option should be replaced by `` only, otherwise the option should be omitted. + - The `--sys_include ` and `--preinclude ` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --edg ` should be specified. + - The `-idirafter ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --sys_include --edg ` should be specified. + - The `-imacros ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude_macros --edg ` should be specified. + - The `/FI ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude --edg ` should be specified. + - The `-Wreserved-user-defined-literal`, `-Wno-reserved-user-defined-literal`, `-fwritable-strings`, `/Zc:rvalueCast`, `/Zc:rvalueCast-`, and `/Zc:wchar_t-` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --reserved_user_defined_literal`, `--edg --no-reserved_user_defined_literal`, `--edg --no_const_string_literals`, `--edg --no_preserve_lvalues_with_same_type_casts`, `--edg --preserve_lvalues_with_same_type_casts`, and `--edg --no_wchar_t_keyword` should be specified, respectively. + - The `/Fo ` option is no longer supported by `semmle-extractor-options`. The option should be omitted. + ## Release 2.20.4 (2025-02-06) ### New features From a30bd7afd1834ee18b5b09a9f406cd540c10e3da Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 6 Mar 2025 07:08:35 -0800 Subject: [PATCH 48/54] Update CHANGELOG.md for 2.20.6 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6192d4e..45b7404 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,17 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.20.6 (2025-03-06) + +### Miscellaneous + +- The CodeQL XML extractor is now able to parse documents in a wider array of + character sets. + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.6. + ## Release 2.20.5 (2025-02-20) ### Breaking changes From 87f935dfb392d8fde2421d837f2485fdcfb553ed Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Tue, 18 Mar 2025 12:37:27 +0000 Subject: [PATCH 49/54] Update CHANGELOG.md --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45b7404..37710f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.20.7 (2025-03-18) + +- There are no user-facing changes in this release. + ## Release 2.20.6 (2025-03-06) ### Miscellaneous From c22b8bd41dd90bce1a3c38253da287dfb814c59f Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 3 Apr 2025 15:43:41 +0200 Subject: [PATCH 50/54] Update CHANGELOG.md --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 37710f2..305cd8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,14 @@ you know what to do). --> +## Release 2.21.0 (2025-04-03) + +### Miscellaneous + +- On macOS the `CODEQL_TRACER_RELOCATION_EXCLUDE` environment variable can now be used to exclude certain paths from the + tracer relocation and tracing process. This environment variable accepts newline-separated regex patterns of binaries + to be excluded. + ## Release 2.20.7 (2025-03-18) - There are no user-facing changes in this release. From 518072067ef64dca25185591bf3cddd25acb3aa0 Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Tue, 22 Apr 2025 12:33:47 +0100 Subject: [PATCH 51/54] Changelog: Update release notes for version 2.21.1 Add details about bug fixes in CodeQL analysis for GitHub Actions. Clarify behavior with `paths-ignore` and `paths` configurations to improve performance on large codebases. --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 305cd8a..c5e2252 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,26 @@ you know what to do). --> +## Release 2.21.1 (2025-04-22) + +### Bugs fixed + +- Fixed a bug in CodeQL analysis for GitHub Actions in the presence + of a code scanning configuration file containing `paths-ignore` + exclusion patterns but not `paths` inclusion patterns. + Previously, such a configuration incorrectly led to all YAML, HTML, + JSON, and JS source files being extracted, + except for those filtered by `paths-ignore`. + This in turn led to performance issues on large codebases. + Now, only workflow and Action metadata YAML files relevant to the + GitHub Actions analysis will be extracted, + except for those filtered by `paths-ignore`. + This matches the default behavior when no configuration file + is provided. + The handling of `paths` inclusion patterns is unchanged: + if provided, only those paths will be considered, + except for those filtered by `paths-ignore`. + ## Release 2.21.0 (2025-04-03) ### Miscellaneous From d1e534e60eafaca3770361ddcfc6b53a06068d91 Mon Sep 17 00:00:00 2001 From: Nick Rolfe Date: Thu, 1 May 2025 13:34:25 +0100 Subject: [PATCH 52/54] Update changelog for 2.21.2 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5e2252..2a9962a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.21.2 (2025-05-01) + +### Bugs fixed + +- `codeql generate log-summary` now correctly includes `dependencies` + maps in predicate events for `COMPUTED_EXTENSIONAL` predicates. + ## Release 2.21.1 (2025-04-22) ### Bugs fixed From 9a367b9f8833da440fdde706789c18cb11f21624 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 15 May 2025 12:41:58 +0100 Subject: [PATCH 53/54] Update changelog for 2.21.3 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a9962a..9f8780a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.21.3 (2025-05-15) + +### Miscellaneous + +- Windows binaries for the CodeQL CLI are now built with `/guard:cf`, enabling [Control Flow Guard](https://learn.microsoft.com/en-us/windows/win32/secbp/control-flow-guard). + ## Release 2.21.2 (2025-05-01) ### Bugs fixed From f1c6d0410fc75afe3b3c146bee902cbd83e19fb8 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Mon, 2 Jun 2025 13:43:16 +0200 Subject: [PATCH 54/54] Update CHANGELOG.md --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f8780a..df6571f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.21.4 (2025-06-02) + +### Deprecations + +- The `clang_vector_types`, `clang_attributes`, and `flax-vector-conversions` command + line options have been removed from the C/C++ extractor. These options were introduced + as workarounds to frontend limitations in earlier versions of the extractor and are + no longer needed when calling the extractor directly. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.7. + ## Release 2.21.3 (2025-05-15) ### Miscellaneous