diff --git a/CHANGELOG.md b/CHANGELOG.md index 059feae..9f8780a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,38 +16,784 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.21.3 (2025-05-15) + +### Miscellaneous + +- Windows binaries for the CodeQL CLI are now built with `/guard:cf`, enabling [Control Flow Guard](https://learn.microsoft.com/en-us/windows/win32/secbp/control-flow-guard). + +## Release 2.21.2 (2025-05-01) + +### Bugs fixed + +- `codeql generate log-summary` now correctly includes `dependencies` + maps in predicate events for `COMPUTED_EXTENSIONAL` predicates. + +## Release 2.21.1 (2025-04-22) + +### Bugs fixed + +- Fixed a bug in CodeQL analysis for GitHub Actions in the presence + of a code scanning configuration file containing `paths-ignore` + exclusion patterns but not `paths` inclusion patterns. + Previously, such a configuration incorrectly led to all YAML, HTML, + JSON, and JS source files being extracted, + except for those filtered by `paths-ignore`. + This in turn led to performance issues on large codebases. + Now, only workflow and Action metadata YAML files relevant to the + GitHub Actions analysis will be extracted, + except for those filtered by `paths-ignore`. + This matches the default behavior when no configuration file + is provided. + The handling of `paths` inclusion patterns is unchanged: + if provided, only those paths will be considered, + except for those filtered by `paths-ignore`. + +## Release 2.21.0 (2025-04-03) + +### Miscellaneous + +- On macOS the `CODEQL_TRACER_RELOCATION_EXCLUDE` environment variable can now be used to exclude certain paths from the + tracer relocation and tracing process. This environment variable accepts newline-separated regex patterns of binaries + to be excluded. + +## Release 2.20.7 (2025-03-18) + +- There are no user-facing changes in this release. + +## Release 2.20.6 (2025-03-06) + +### Miscellaneous + +- The CodeQL XML extractor is now able to parse documents in a wider array of + character sets. + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.6. + +## Release 2.20.5 (2025-02-20) + +### Breaking changes + +- Removed support for `QlBuiltins::BigInt`s in the `avg()` aggregate. + +- A number of breaking changes have been made to the C and C++ CodeQL test environment as used by `codeql test run`: + - The `-Xclang-only=` option is no longer supported by `semmle-extractor-options`. Instead, when either `--clang` or `--clang_version` is specified the option should be replaced by `` only, otherwise the option should be omitted. + - The `--sys_include ` and `--preinclude ` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --edg ` should be specified. + - The `-idirafter ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --sys_include --edg ` should be specified. + - The `-imacros ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude_macros --edg ` should be specified. + - The `/FI ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude --edg ` should be specified. + - The `-Wreserved-user-defined-literal`, `-Wno-reserved-user-defined-literal`, `-fwritable-strings`, `/Zc:rvalueCast`, `/Zc:rvalueCast-`, and `/Zc:wchar_t-` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --reserved_user_defined_literal`, `--edg --no-reserved_user_defined_literal`, `--edg --no_const_string_literals`, `--edg --no_preserve_lvalues_with_same_type_casts`, `--edg --preserve_lvalues_with_same_type_casts`, and `--edg --no_wchar_t_keyword` should be specified, respectively. + - The `/Fo ` option is no longer supported by `semmle-extractor-options`. The option should be omitted. + +## Release 2.20.4 (2025-02-06) + +### New features + +- Using the `actions` language (for analysis of GitHub Actions workflows) no longer requires + the `CODEQL_ENABLE_EXPERIMENTAL_FEATURES` environment variable to be set. Support for analysis + of GitHub Actions workflows remains in public preview. + +### Bugs fixed + +- Fixed a bug where CodeQL for Java would fail with an SSL exception while trying to download `maven`. + +### Miscellaneous + +- The build of the [logback-core](https://logback.qos.ch/) library that is used for logging in the CodeQL CLI has been updated to version 1.3.15. + +## Release 2.20.3 (2025-01-24) + +### Security Updates + +- Resolves a security vulnerability where CodeQL databases or logs produced by the CodeQL CLI may contain the environment variables from the time of + database creation. This includes any secrets stored in an environment variables. For more information, see the + [CodeQL CLI security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gqh3-9prg-j95m). + + All users of CodeQL should follow the advice in the CodeQL advisory mentioned above or upgrade to this version or a later version of CodeQL. + + If you are using the CodeQL Action, also see the related [CodeQL Action security advisory](https://github.com/github/codeql-action/security/advisories/GHSA-vqf5-2xx6-9wfm). + +## Release 2.20.2 (2025-01-22) + +### Improvements + +- `codeql database create` and `codeql database finalize` now write relations to disk in a new, compressed format. As a result, databases will generally take up less space on disk, whether zipped or unzipped. Note that databases in this format can only be read and analyzed using CodeQL version 2.20.1 onwards. Attempting to analyze such a database with CodeQL version 2.20.0 or older will fail, with an error message like the following: + ``` + UnsortedExtensionalError: Tuples that were assumed to be in order are not: [123456777, 777654321, 123456777]<[777654321, 123456777, 777654321] + ``` + +### Enhancements + +- Added the `.bitLength()` method to `QlBuiltins::BigInt`. + +### Bugs Fixed + +- Fixed a bug where CodeQL would crash on rare occasions while merging SARIF files before uploading results. + +## Release 2.20.1 (2025-01-09) + +### Improvements + +- Automatic installation of dependencies for C++ autobuild is now supported on Ubuntu 24.04. + +- The CLI will now warn if it detects that it is installed in a + location where it is likely to cause performance issues. This + includes: user home, desktop, downloads, or the file system root. + + You can avoid this warning by setting the `CODEQL_ALLOW_INSTALLATION_ANYWHERE` + environment variable to `true`. + +## Release 2.20.0 (2024-12-09) + +### Known issues + +- The Windows executable for this release is labeled with an incorrect version number + within its properties: the version number should be 2.20.0 rather than 2.19.4. + `codeql version` reports the correct version number. + +### New features + +- The [`QlBuiltins::BigInt` type](https://codeql.github.com/docs/ql-language-reference/modules/#bigint) of + arbitrary precision integers is generally available and no longer hidden behind the + `--allow-experimental=bigint` CLI feature flag. + +### Miscellaneous + +- Backslashes are now escaped when writing output in the Graphviz DOT format (`--format=dot`). +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version + 21.0.5. + +## Release 2.19.4 (2024-12-02) + +### Improvements + +- CodeQL now supports passing values containing the equals character (`=`) to extractor options via + the `--extractor-option` flag. This allows cases like `--extractor-option opt=key=value`, which + sets the extractor option `opt` to hold the value `key=value`, whereas previously that would have + been rejected with an error. +- The `codeql pack bundle` command now sets the numeric user and group IDs of entries in the generated + `tar` archive to `0`. This avoids failures like `IllegalArgumentException: user id '7111111' is too big ( > 2097151 )` + when the numeric user ID is too large. + +### Bugs fixed + +- On MacOS, `arch -arm64` commands no longer fail when they are executed via `codeql database create --command`, + via `codeql database trace-command`, or are run after `codeql database init --begin-tracing`. Note + that build commands invoked this way still will not normally be traced, so this is useful only for + running ancillary commands which are incidental to building your code. +- Fixed a bug where `codeql test run` would not preserve test + databases on disk after a test failed. + +## Release 2.19.3 (2024-11-07) + +### Bugs fixed + +- Fixed a bug where using `codeql database import` to combine multiple non-empty + databases may produce a corrupted database. (The bug does not affect using + `codeql database finalize --additional-dbs` to combine multiple databases.) + +- Fixed a bug where uses of a `QlBuiltins::ExtensionId` variable that was not + bound to a value could be incorrectly accepted in some cases. In many cases, + this would result in a crash. + +- CodeQL would sometimes refuse to run with more than around 1,500 GB of RAM + available, complaining that having so much memory was "unrealistic". The + amount of memory CodeQL is able to make any meaningful use of still tops out + at about that value, but it will now gracefully accept that so large + computers do in fact exist. + +- Fixed a bug in command-line parsing where a misspelled option could sometimes + be misinterpreted as, e.g., the name of a query to run. Now every command-line + argument that begins with a dash is assumed to be intended as an option + (unless it comes after the `--` separator), and an appropriate error is + emitted if that is not a recognized one. + + The build command in `codeql database trace-command` is exempted from this for + historical reasons, but we strongly recommend putting a `--` before the entire + build command there, in case a future `codeql` version starts recognizing + options that you intended to be part of the build command. + +### Miscellaneous + +- The CodeQL Bundle is now available as an artifact that is compressed using + [Zstandard](https://en.wikipedia.org/wiki/Zstd). This artifact is + smaller and faster to decompress than the original, gzip-compressed bundle. The CodeQL bundle + is a tar archive containing tools, scripts, and various CodeQL-specific files. + + If you are currently using the CodeQL Bundle, you may want to consider switching to the + Zstandard variant of the bundle. You can download the new form of the CodeQL Bundle from the + [codeql-action releases page](https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.19.3) + by selecting the appropriate bundle with the `.zst` extension. The gzip-compressed bundles will + continue to be available for backwards compatibility. + +## Release 2.19.2 (2024-10-21) + +### Potentially breaking changes + +- The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable `CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB` to `1`. + +### Bugs fixed + +- The 2.19.1 release contained a bug in the query evaluator that under rare conditions could lead to wrong alerts or resource exhaustion. Although we have never seen the problem outside of internal testing, we encourage users on 2.19.1 to upgrade to 2.19.2. + +### Miscellaneous + +- The database relation `sourceLocationPrefix` is changed for databases created with + `codeql test run`. Instead of containing the path of the enclosing qlpack, it now + contains the actual path of the test, similar to if one had run `codeql database create` + on the test folder. For example, for a test such as + `/cpp/ql/test/query-tests/Security/CWE/CWE-611/XXE.qlref` we now populate + `sourceLocationPrefix` with `/cpp/ql/test/query-tests/Security/CWE/CWE-611/` + instead of `/cpp/ql/test/`. This change typically impacts calls to + `File.getRelativePath()`, and may as a result change the expected test output. + +## Release 2.19.1 (2024-10-04) + +### New Features + +- The command `codeql generate query-help` now supports Markdown help files. + The Markdown help format is commonly used in custom CodeQL query packs. This new + feature allows us to generate SARIF reporting descriptors for CodeQL queries that + include Markdown help directly from a query Markdown help file. + +- Added a new command, `codeql resolve packs`. This command shows each step in the + pack search process, including what packs were found in each step. With the + `--show-hidden-packs` option, it can also show details on which packs were hidden + by packs found earlier in the search sequence. `codeql resolve packs` is intended + as a replacement for most uses of `codeql resolve qlpacks`, whose output is both + less detailed and less accurate. + +## Release 2.19.0 (2024-09-18) + +### Improvements + +- `codeql database analyze` and `codeql database interpret-results` now support + the `--sarif-run-property` option. You can provide this option when using a SARIF + output format to add a key-value pair to the property bag of the run object. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.4. + +## Release 2.18.4 (2024-09-12) + +### New Features + +- C# support for `build-mode: none` is now out of beta, and generally available. +- Go 1.23 is now supported. + +## Release 2.18.3 (2024-08-28) + +- There are no user-facing changes in this release. + +## Release 2.18.2 (2024-08-13) + +### Deprecations + +- Swift analysis on Ubuntu is no longer supported. Please migrate to macOS if this affects you. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.3. + +## Release 2.18.1 (2024-07-25) + +### Security Updates + +- Resolves CVE-2023-4759, an arbitrary file overwrite in Eclipse JGit + that can be triggered when using untrusted third-party queries from a + git repository. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-x4gx-f2xv-6wj9) + for more information. +- The following dependencies have been updated. These updates include + security fixes in the respective libraries that prevent + out-of-bounds accesses or denial-of-service in scenarios where + untrusted files are processed. These scenarios are not likely to be + encountered in most uses of CodeQL and code scanning, and only + apply to advanced use cases where precompiled query packs, + database ZIP files, or database TRAP files are obtained from + untrusted sources and then processed on a trusted machine. + - airlift/aircompressor is updated to version 0.27. + - Apache Ant is updated to version 1.10.11. + - Apache Commons Compress is updated to version 1.26.0. + - Apache Commons IO is updated to version 2.15.1. + - Apache Commons Lang3 is updated to version 3.14.0. + - jsoup is updated to version 1.15.3. + - Logback is updated to version 1.2.13. + - Snappy is updated to version 0.5. + +### New features + +- The *experimental* type `QlBuiltins::BigInt` of arbitrary-precision integers + has been introduced. To opt in to this API, compile your queries with + `--allow-experimental=bigint`. Big integers can be constructed using the + `.toBigInt()` methods of `int` and `string`. The built-in operations are: + - comparisons: `=`, `!=`, `<`, `<=`, `>`, `>=`, + - conversions: `.toString()`, `.toInt()`, + - arithmetic: binary `+`, `-`, `*`, `/`, `%`, unary `-`, + - bitwise operations: `.bitAnd(BigInt)`, `.bitOr(BigInt)`, + `.bitXor(BigInt)`, `.bitShiftLeft(int)`, `.bitShiftRightSigned(int)`, + `.bitNot()`, + - aggregates: `min`, `max`, (`strict`)`sum`, (`strict`)`count`, `avg`, + `rank`, `unique`, `any`. + - other: `.pow(int)`, `.abs()`, `.gcd(BigInt)`, `.minimum(BigInt)`, + `.maximum(BigInt)`. +- `codeql test run` now supports postprocessing of test results. When .qlref + files specify a path to a `postprocess` query, then this is evaluated after + the test query to transform the test outputs prior to concatenating them into + the `actual` results. + +### Improvements + +- The 30% QL query compilation slowdown noted in 2.18.0 has been fixed. + +## Release 2.18.0 (2024-07-11) + +### Breaking changes + +- A number of breaking changes have been made to the C and C++ CodeQL + test environment as used by `codeql test run`: + - The test environment no longer defines any GNU-specific builtin + macros. If these macros are still needed by a test, please define + them via `semmle-extractor-options`. + - The `--force-recompute` option is no longer directly supported by + `semmle-extractor-options`. Instead, `--edg --force-recompute` + should be specified. + - The `--gnu_version` and `--microsoft_version` options that can be + specified via `semmle-extractor-options` are now synonyms, and only + one should be specified as part of `semmle-extractor-options`. + Furthermore, is also no longer possible to specify these options + via the following syntax. + + - `--edg --gnu_version --edg `, and + - `--edg --microsoft_version --edg ` + + The shorter `--gnu_version ` and + `--microsoft_version ` should be used. +- The `--build_error_dir` and `--predefined_macros` command line options + have been removed from the C/C++ extractor. It has never been possible + to pass these options through the CLI, but some customers with advanced + setups may have been passing them through internal undocumented interfaces. + Passing the option `--build_error_dir` did not have any effect, and it + is safe to remove the option. The `--predefined_macros` option should + have been unnecessary, as long as the extractor was invoked with the + `--mimic` option. + +### Regressions + +- Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release. + +### Improvements + +- Introduced the `--include-logs` option to the `codeql database bundle` + command. This new feature allows users to include logs in the generated + database bundle, allowing for a more complete treatment of the bundle, and + bringing the tool capabilities up-to-speed with the documentation. +- `codeql database init` and `codeql database create` now support the + `--force-overwrite` option. When this option is specified, the command will + delete the specified database directory even if it does not look like a + database directory. This option is only recommended for automation. For + directcommand line commands, it is recommended to use the `--overwrite` + option, which includes extra protection and will refuse to delete a + directory that does not look like a database directory. +- Extract `.xsaccess`, `*.xsjs` and `*.xsjslib` files for SAP HANA XS as + Javascript. +- We have updated many compiler error messages and warnings to improve their + readability and standardize their grammar. + Where necessary, please use the `--learn` option for the `codeql test run` + command. + +### Bugs fixed + +- Where a MacOS unsigned binary cannot be signed, CodeQL will now continue + trying to trace compiler invocations created by that process and its + children. In particular this means that Bazel builds on MacOS are now + traceable. +- Fixed a bug where test discovery would fail if there is a syntax error in a + qlpack file. Now, a warning message will be printed and discovery will + continue. + +## Release 2.17.6 (2024-06-27) + +### New features + +- Beta support is now available for analyzing C# codebases without needing a working build. To use + this, pass the `--build-mode none` option to `codeql database create`. + +### Improvements + +- The `--model-packs` option is now publicly available. This option allows commands like `codeql database analyze` + to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis. + +## Release 2.17.5 (2024-06-12) + +### Breaking changes + +- All the commands that output SARIF will output a minified version to reduce the size. + The `codeql database analyze`, `codeql database interpret-results`, `codeql generate query-help`, and `codeql bqrs interpret` commands support the option `--no-sarif-minify` to output a pretty printed SARIF file. + +- A number of breaking changes have been made to the `semmle-extractor-options` + functionality available for C and C++ CodeQL tests. + + - The Arm, Intel, and CodeWarrior compilers are no longer supported and the + `--armcc`, `--intel`, `--codewarrior` flags are now ignored, as are all the + flags that only applied to those compilers. + - The `--threads` and `-main-file-name` options, which did not have any effect + on tests, are now ignored. Any specification of these options as part of + `semmle-extractor-options` should be removed. + - Support for `--linker`, all flags that would only invoke the preprocessor, + and the `/clr` flag have been removed, as those flags would never produce any + usable test output. + - Support for the `--include_path_environment` flag has been removed. All include + paths should directly be specified as part of `semmle-extractor-options`. + - Microsoft C/C++ compiler response files specified via `@some_file_name` are + now ignored. Instead, all options should directly be specified as part of + `semmle-extractor-options`. + - Support for Microsoft `#import` preprocessor directive has been removed, as + support depends on the availability of the Microsoft C/C++ compiler, and + availability cannot be guaranteed on all platforms while executing tests. + - Support for the Microsoft `/EHa`, `/EHs`, `/GX`, `/GZ`, `/Tc`, `/Tp`, and `/Zl` + flags, and all `/RTC` flags have been removed. Any specification of these + options as part of `semmle-extractor-options` should be removed. + - Support for the Apple-specific `-F` and `-iframework` flags has been removed. + The `-F` flag can still be used by replacing `-F ` by + `--edg -F --edg `. Any occurrence of `-iframework ` should be + replaced by `--edg --sys_framework --edg `. + - Support for the `/TC`, `/TP`, and `-x` flags has been removed. Please ensure + all C, respectively C++, source files have a `.c`, respectively `.cpp`, + extension. + - The `--build_error_dir`, `-db`, `--edg_base_dir`, `--error_limit`, + `--src_archive`, `--trapfolder`, and `--variadic_macros` flags are now ignored. + + The above changes do not affect the creation of databases through the CodeQL CLI, + or when calling the C/C++ extractor directly with the `--mimic` or `--linker` flags. + Similar functionality continues to be supported in those scenarios, except for + CodeWarrior and the `--edg_base_dir`, `--include_path_environment`, `/Tc`, and `/Tp` + flags, which were never supported. + +### Improvements + +- `codeql generate log-summary` now reports completed pipeline runs that + are part of an incomplete recursive predicate. + +### Miscellaneous + +- The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal + documentation generation commands has been updated to version + [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). + +## Release 2.17.4 (2024-06-03) + +### New features + +- CodeQL package management is now generally available, and all GitHub-produced + CodeQL packages have had their version numbers increased to 1.0.0. + +## Release 2.17.3 (2024-05-17) + +### Improvements + +- The language server that our IDE integration is built on now defaults + to fine-grained dependency tracking for incremental error-checking + after file changes. This slightly improves the latency of refreshing + errors after local source code edits and will enable significant + speedups in the future. +- We now properly handle globs (such as `folder/**/*.py`) in `paths` configuration + to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan). +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now supports allocating 2^32 IDs during the import process. The previous limit + was 2^31 IDs. + +## Release 2.17.2 (2024-05-07) + +### Known issues + +- The beta support for analyzing Swift in this release and all + previous releases requires `g++-13` when running on Linux. Users + analyzing Swift using the `ubuntu-latest`, `ubuntu-22.04`, or + `ubuntu-20.04` runner images for GitHub Actions should update their + workflows to install `g++-13`. For more information, see [the runner + images + announcement](https://github.com/actions/runner-images/issues/9679). + +### Improvements + +- When uploading a SARIF file to GitHub using `codeql github + upload-results`, the CodeQL CLI now waits for the file to be + processed by GitHub. If any errors occurred during processing of the + analysis results, the command will log these and return a non-zero + exit code. To disable this behaviour, pass the + `--no-wait-for-processing` flag. + + By default, the command will wait for the SARIF file to be processed + for a maximum of 2 minutes, however this is configurable with the + `--wait-for-processing-timeout` option. +- The build tracer is no longer enabled when using the [`none` build + mode](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes) + to analyze a compiled language, thus improving performance. + +## Release 2.17.1 (2024-04-24) + +### Deprecations + +- The `--mode` option and `-m` alias to `codeql database create`, + `codeql database cleanup`, and `codeql dataset cleanup` has been + deprecated. Instead, use the new `--cache-cleanup` option, which has + identical behavior. + +### Improvements + +- Improved the diagnostic message produced when no code is processed + when creating a database. If a build mode was specified using + `--build-mode`, the message is now tailored to your build mode. + +### Miscellaneous + +- The `scc` tool used by the CodeQL CLI to calculate source code baseline + information has been updated to version [3.2.0](https://github.com/boyter/scc/releases/tag/v3.2.0). + +## Release 2.17.0 (2024-04-04) + +### Deprecations + +- The `--[no-]analysis-summary-v2` and `--[no-]new-analysis-summary` options + that were used to enable (or disable) improved summary information printed at + the end of a `codeql database analyze` invocation are no longer supported. + Improved summary information is now enabled for all invocations. +- Support for overwriting default CodeQL SARIF run properties using the + `--sarif-run-property` command line option has been removed. This removes the + ability to overwrite the `semmle.formatSpecifier`, `metricResults`, and + `codeqlConfigSummary` properties in the SARIF run file. + +### Improvements + +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now performs better in low-memory situations. (Put another way, it now needs + less RAM to achieve the same performance as before.) + +- The worst-case performance of transitive closure computation (using + the `+` or `*` postfix operators or the `fastTC` higher-order + primitive in QL) has been greatly improved. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.2. + +## Release 2.16.6 (2024-03-26) + +### Bugs fixed + +- Fixes a bug where extractor logs would be output at a lower than expected + verbosity level when using the `codeql database create` command. + +## Release 2.16.5 (2024-03-21) + +### New features + +- Beta support has been added for analyzing Java codebases without needing a working build. To enable + this, pass the `--build-mode none` option to `codeql database create`. + +## Release 2.16.4 (2024-03-11) + +### Potentially breaking changes + +- A number of internal command line options (`--builtin_functions_file`, `--clang_builtin_functions`, + `--disable-objc-default-synthesize-properties`, `--list_builtin_functions`, `--memory-limit-bytes`, + `--mimic_config`, and `--objc`) has been removed from the C/C++ extractor. It has never been + possible to pass these options through the CLI itself, but some customers with advanced setups may + have been passing them through internal undocumented interfaces. All of the removed options were + already no-ops, and will now generate errors. + + The `--verbosity` command line option has also been removed. The option was an alias for + `--codeql-verbosity`, which should be used instead. + +### Improvements + +- The frontend of the C/C++ extractor has been updated, improving the + extractor's reliability and increasing its ability to extract source code. + +### Bugs fixed + +- When parsing user-authored YAML files such as `codeql-pack.yml`, + `qlpack.yml`, `codeql-workspace.yml`, and any YAML file defining a data + extension, unquoted string values starting with a `*` character are now + correctly interpreted as YAML aliases. Previously, they were interpreted + as strings, but with the first character skipped. + + If you see a parse error similar to `while scanning an alias... unexpected` + `character found *(42)`,it likely means that you need to add quotes around + the indicated string value. The most common cause is unquoted glob patterns + that start with `*`, such as `include: **/*.yml`, which will need to be + quoted as `include: "**/*.yml"`. + +## Release 2.16.3 (2024-02-22) + +### Security patches + +- Fixes CVE-2024-25129, a limited data exfiltration vulnerability that + could be triggered by untrusted databases or QL packs. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gf8p-v3g3-3wph) + for more information. + +### New Features + +- A new extractor option has been added to the Python extractor: + `python_executable_name`. You can use this option to override the default + process the extractor uses to find and select a Python executable. Pass one of + `--extractor-option python_executable_name=py` or `--extractor-option + python_executable_name=python` or `--extractor-option + python_executable_name=python3` to commands that run the extractor, for + example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. If the Python executable has a different name, you + can set the new extractor option to override this value and look for + `python.exe` or `python3.exe`. + + For more information about using the extractor option with the CodeQL CLI, see + [Extractor + options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + +### Bugs fixed + +- Fixed a bug where CodeQL may produce an invalid database when it exhausts + all available ID numbers. Now it detects the condition and reports an + error instead. + +## Release 2.16.2 (2024-02-12) + +- There are no user-facing changes in this release. + +## Release 2.16.1 (2024-01-25) + +### Improvements + +- When executing the `codeql database init` command, the CodeQL runner + executable path is now stored in the `CODEQL_RUNNER` environment variable. + Users of indirect tracing on MacOS with System Integrity Protection enabled + who previously had trouble with indirect tracing should prefix their build + command with this path. For example, `$CODEQL_RUNNER build.sh`. + +### QL language improvements + +- Name clashes between weak aliases (i.e. aliases that are not final aliases of + non-final entities) of the same target no longer cause ambiguity errors. + +## Release 2.16.0 (2024-01-16) + +### New Features + +- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. + +### Improvements + +- The `Experimental` flag has been removed from all packaging and related commands. +- The RA pretty-printer omits names of internal RA nodes and pretty-prints + binary unions with nested internal unions as n-ary unions. VS Code extension + v1.11.0 or newer is required to compute join order badness metrics in VS Code + for the new RA format. + + +### Potentially breaking changes + +- The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting `CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1`. +- The `--ram` option to `codeql database run-queries` and other + commands that execute queries is now interpreted more strictly. + Previously it was mostly a rough hint for how much memory to use, + and the actual memory footprint of the CodeQL process could be + hundreds of megabytes higher. From this release, CodeQL tries harder + to keep its _total_ memory consumption during evaluation below the + given limit. + + The new behavior yields more predictable memory use, but since it + works by allocating less RAM, it can lead to more use of _disk_ + storage for intermediate results compared to earlier releases with + the same `--ram` value, and consequently a slight performance + loss. In rare cases, for large databases, analysis may fail with a + Java `OutOfMemoryError`. + + The cure for this is to increase `--ram` to be closer to the amount + of memory actually available for CodeQL. As a rule of thumb, it will + usually be possible to increase the value of `--ram` by 700 MB or + more, without actually using more resources than release 2.15.x + would with the old setting. An exact amount cannot stated, however, + since the actual memory footprint in earlier releases depended on + factors such as the size of the databases that were not fully taken + into account. + + If you use the CodeQL Action, you do not need to do anything unless + you have manually overridden the Action's RAM setting. The Action + will automatically select a `--ram` setting that matches the version + of the CLI it uses. + +## Release 2.15.5 (2023-12-20) + +### New features + +- A new extractor option has been added to the JavaScript/TypeScript extractor. + Set the environment variable `CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES` + to `true` to skip the extraction of types in TypeScript files. + Use this to speed up extraction if your codebase has a high volume of + TypeScript type information that causes a noticeable bottleneck for + TypeScript extraction. The majority of analysis results should be preserved + even when no types are extracted. + +### Bugs fixed + +- Fixed an issue where CodeQL would sometimes incorrectly report that no files + were scanned when running on Windows. + This affected the human-readable summary produced by `codeql database analyze` + and `codeql database interpret-results`, but did not impact the file coverage + information produced in the SARIF output and displayed on the tool status page. +- When analyzing Swift codebases, CodeQL build tracing will now ignore the + `codesign` tool. This prevents errors in build commands or workflows on macOS + that include both CodeQL and code signing. + +## Release 2.15.4 (2023-12-11) + +### New features + +- Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns. + +### Improvements + +- Parallelism in the evaluator has been improved, resulting in faster analysis when + running with many threads, particularly for large databases. + ## Release 2.15.3 (2023-11-22) ### New features -- A new compilation flag (`--fail-on-ambiguous-relation-name`) has been added to specify - that compilation should fail if the compiler generates an ambiguous relation name. -- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the - linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of - database completeness) for C++ projects where this part of database creation consumes - too much memory. This option is available in the commands `database create`, - `database finalize`, `database import`, `dataset import`, `test extract`, and - `test run`. -- The CodeQL language server now provides basic support for Rename, and you can - now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The - current Rename support is less a refactoring tool and more a labor-saving - device. You may have to perform some manual edits after using Rename, but it - should still be faster and less work than renaming a symbol manually. - `codeql database analyze` now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag `--sarif-include-query-help`, which provides the options `always` (which includes query help for all queries), `custom_queries_only` (the default) and `never` (which does not include query help for any query). The existing flag `--sarif-add-query-help` has been deprecated and will be removed in a future release. +- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the + linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of + database completeness) for C++ projects where this part of database creation consumes + too much memory. This option is available in the commands `database create`, + `database finalize`, `database import`, `dataset import`, `test extract`, and + `test run`. +- The CodeQL language server now provides basic support for Rename, and you can now use + the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename + support is less a refactoring tool and more a labor-saving device. You may have to + perform some manual edits after using Rename, but it should still be faster and less + work than renaming a symbol manually. ### Improvements -- The Find References feature in the CodeQL language server now supports all - CodeQL identifiers and offers improved performance compared to CodeQL CLI - 2.14 releases. -- The compiler generates shorter human-readable DIL and RA relation names. Due - to use of an extended character set, full VS Code support for short relation - names requires VS Code extension 1.9.4 or newer. +- The Find References feature in the CodeQL language server now supports all CodeQL + identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. +- The compiler generates shorter human-readable DIL and RA relation names. Due to use + of an extended character set, full VS Code support for short relation names requires + VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. @@ -56,14 +802,12 @@ - Fixed an internal error in the compiler when arguments to the `codePointCount` string primitive were not bound. -- Fixed a bug where `codeql database finalize` would fail if a - database under construction was moved between machines between - `codeql database init` and `codeql database finalize`. This should - now work, as long as both commands are run by the same _release_ of - the CodeQL CLI and the extractors used are the ones bundled with the - CLI. -- Fixed a bug where `codeql database run-queries` would fail in some - circumstances when the database path included an `@`. +- Fixed a bug where `codeql database finalize` would fail if a database under construction + was moved between machines between `codeql database init` and `codeql database finalize`. + This should now work, as long as both commands are run by the same _release_ of the + CodeQL CLI and the extractors used are the ones bundled with the CLI. +- Fixed a bug where `codeql database run-queries` would fail in some circumstances when + the database path included an `@`. ## Release 2.15.2 (2023-11-13) @@ -79,7 +823,7 @@ - `codeql database analyze` and `codeql database interpret-results` can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the `GITHUB_SERVER_URL` and `GITHUB_REPOSITORY` environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server - version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. + version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. - CodeQL now supports distinguishing file coverage information between related languages C and C++, Java and Kotlin, and JavaScript and TypeScript. By default, file coverage information for each @@ -428,7 +1172,7 @@ member predicates that had stronger binding sets than their root definitions. - Fixed a bug where a query could not be run from VS Code - when there were packs nested within sibling directories + when there were packs nested within sibling directories of the query. ## Release 2.13.2 @@ -454,7 +1198,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6. @@ -530,7 +1274,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. @@ -538,7 +1282,7 @@ This release was skipped. causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. - Customers who wish to use CodeQL 2.12.6 or 2.13.0 can + Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing `--no-sarif-include-diagnostics` to any invocations of `codeql database analyze` or `codeql database interpret-results`. @@ -704,8 +1448,8 @@ This release was skipped. `codeql database create` now accounts for [`paths` and `paths-ignore` configuration](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#specifying-directories-to-scan). - In the VS Code extension, recursive calls will be marked with inlay - hints. These can be disabled with the global inlay hints setting - (`editor.inlayHints.enabled`). If you just want to disable them for + hints. These can be disabled with the global inlay hints setting + (`editor.inlayHints.enabled`). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id is `ql`). See [Language Specific Editor Settings](https://code.visualstudio.com/docs/getstarted/settings#_language-specific-editor-settings) in the VS Code documentation for more information.