8000 [WIP] Caching resolutions in buildInfo and reusing them by sheetalkamat · Pull Request #50007 · microsoft/TypeScript · GitHub
[go: up one dir, main page]
Skip to content

[WIP] Caching resolutions in buildInfo and reusing them #50007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 31 commits into from
Closed

[WIP] Caching resolutions in buildInfo and reusing them #50007

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
3172629
Add new option to cacheResolution (No actual functionality yet)
sheetalkamat Jun 30, 2022
e1600ab
Add baselining of modules and type refs
sheetalkamat Dec 9, 2022
666eece
Add tests
sheetalkamat Jul 1, 2022
1eeea26
Store resolved module and type reference resolution cache in buildinfo
sheetalkamat Jul 7, 2022
76100d7
Read reusable module cache information from the buildinfo
sheetalkamat Jul 8, 2022
4bc74b9
Add reusing cache stub
sheetalkamat Jul 8, 2022
23cfc79
Buildinfo resolutions actually reused
sheetalkamat Jul 18, 2022
0132dc0
Handle project reference redirects for the module and type reference …
sheetalkamat Jul 22, 2022
d55ab98
Test for module resolutions from different directories
sheetalkamat Jul 22, 2022
ce07d0e
Handle resolutions that would be same in ancestor directory and can b…
sheetalkamat Jul 23, 2022
959206d
Do not store failed lookups with cacheResolution option if it is reso…
sheetalkamat Jul 25, 2022
0446430
Add tests where module resolution caches should reuse the resolutions…
sheetalkamat Jul 25, 2022
207226a
Reusing resolutions in tsserver scenario
sheetalkamat Jul 25, 2022
591960e
Set old program build info as a location to look for from module reso…
sheetalkamat Jul 25, 2022
71ebed9
Modify resolution cache to update on program creation completion
sheetalkamat Nov 29, 2022
4d127fa
Remove files that are not in program from cache of unresolved imports
sheetalkamat Jul 27, 2022
feb3220
Tests for unresolved imports from multiple places
sheetalkamat Jul 27, 2022
84cda33
More tests for cache reuse directory structure
sheetalkamat Jul 27, 2022
2219b5d
Start using cache as perDirectory lookup
sheetalkamat Nov 16, 2022
6261b6c
Add tests where cache resoluition is incorrectly used because of not …
sheetalkamat Jul 27, 2022
e1ed9ea
Store package json hash in buildinfo
sheetalkamat Nov 29, 2022
7de4780
Use the hashes to verify the affecting location before using the reso…
sheetalkamat Jul 28, 2022
5c11252
Add tests for package json edits
sheetalkamat Aug 2, 2022
e21738c
Store package.json for the directories in buildinfo
sheetalkamat Aug 2, 2022
be5ecb2
During cacheResolutions dont watch failed lookups and dont look at th…
sheetalkamat Aug 3, 2022
78e215f
Dont store package json path if its found in same directory
sheetalkamat Aug 4, 2022
1e123ec
Cache packagejson scopes per directory
sheetalkamat Dec 9, 2022
81a1af9
Dont store isExternalLibraryImport in the buildInfo
sheetalkamat Dec 9, 2022
3115572
Store resolvedFileName as fileId instead of structure if thats the on…
sheetalkamat Dec 10, 2022
2bb56da
Always respect preserveSymlinks
sheetalkamat Dec 10, 2022
b179c7c
Dont store originalPath as separate, instead store originalPath || re…
sheetalkamat Dec 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Reusing resolutions in tsserver scenario
  • Loading branch information
@sheetalkamat
sheetalkamat committed Dec 9, 2022
commit 207226a9c8da0a70c5e55328c83f6469ee0997b6
3 changes: 2 additions & 1 deletion src/compiler/builder.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1941,7 +1941,8 @@ export interface ProgramBuildInfoPathDecoder {
getCanonicalFileName: GetCanonicalFileName;
}

function getProgramBuildInfoFilePathDecoder(fileNames: readonly string[], buildInfoPath: string, currentDirectory: string, getCanonicalFileName: GetCanonicalFileName): ProgramBuildInfoPathDecoder {
/** @internal */
export function getProgramBuildInfoFilePathDecoder(fileNames: readonly string[], buildInfoPath: string, currentDirectory: string, getCanonicalFileName: GetCanonicalFileName): ProgramBuildInfoPathDecoder {
const buildInfoDirectory = getDirectoryPath(getNormalizedAbsolutePath(buildInfoPath, currentDirectory));
let filePaths: Path[] | undefined;
let fileAbsolutePaths: string[] | undefined;
Expand Down
10 changes: 8 additions & 2 deletions src/compiler/watchPublic.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ export interface ReadBuildProgramHost {
/** @internal */
getBuildInfo?(fileName: string, configFilePath: string | undefined): BuildInfo | undefined;
}
export function readBuilderProgram(compilerOptions: CompilerOptions, host: ReadBuildProgramHost) {
/** @internal */
export function readBuildInfoForProgram(compilerOptions: CompilerOptions, host: ReadBuildProgramHost) {
const buildInfoPath = getTsBuildInfoEmitOutputFilePath(compilerOptions);
if (!buildInfoPath) return undefined;
let buildInfo;
Expand All @@ -115,7 +116,12 @@ export function readBuilderProgram(compilerOptions: CompilerOptions, host: ReadB
buildInfo = getBuildInfo(buildInfoPath, content);
}
if (!buildInfo || buildInfo.version !== version || !buildInfo.program) return undefined;
return createBuilderProgramUsingProgramBuildInfo(buildInfo, buildInfoPath, host, compilerOptions.configFilePath);
return { buildInfo, buildInfoPath };
}

export function readBuilderProgram(compilerOptions: CompilerOptions, host: ReadBuildProgramHost) {
const result = readBuildInfoForProgram(compilerOptions, host);
return result && createBuilderProgramUsingProgramBuildInfo(result.buildInfo, result.buildInfoPath, host, compilerOptions.configFilePath);
}

export function createIncrementalCompilerHost(options: CompilerOptions, system = sys): CompilerHost {
Expand Down
38 changes: 31 additions & 7 deletions src/services/services.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,12 @@ import {
Completions,
computePositionOfLineAndCharacter,
computeSuggestionDiagnostics,
convertToOptionsWithAbsolutePaths,
createDocumentRegistry,
createGetCanonicalFileName,
createMultiMap,
createOldBuildInfoProgram,
createProgram,
CreateProgramOptions,
createSourceFile,
CreateSourceFileOptions,
createTextSpanFromBounds,
Expand Down Expand Up @@ -108,6 +109,7 @@ import {
getNonAssignedNameOfDeclaration,
getNormalizedAbsolutePath,
getObjectFlags,
getProgramBuildInfoFilePathDecoder,
getScriptKind,
getSetExternalModuleIndicator,
getSnapshotText,
Expand Down Expand Up @@ -183,8 +185,8 @@ import {
isTagName,
isTextWhiteSpaceLike,
isThisTypeParameter,
JSDoc,
JsDoc,
JSDoc,
JSDocContainer,
JSDocTagInfo,
JsonSourceFile,
Expand Down Expand Up @@ -225,6 +227,7 @@ import {
ObjectFlags,
ObjectLiteralElement,
ObjectLiteralExpression,
OldBuildInfoProgramConstructor,
OperationCanceledException,
OrganizeImports,
OrganizeImportsArgs,
Expand All @@ -243,6 +246,7 @@ import {
PropertyName,
Push,
QuickInfo,
readBuildInfoForProgram,
refactor,
RefactorContext,
RefactorEditInfo,
Expand Down Expand Up @@ -1601,6 +1605,28 @@ export function createLanguageService(
return sourceFile;
}

function getOldProgram(options: CompilerOptions, compilerHost: CompilerHost): Program | OldBuildInfoProgramConstructor | undefined {
if (program) return program;
if (!options.cacheResolutions) return undefined;
const buildInfoResult = readBuildInfoForProgram(options, compilerHost);
if (!buildInfoResult?.buildInfo.program!.cacheResolutions) return undefined;

const buildInfoFilePathDecoder = getProgramBuildInfoFilePathDecoder(buildInfoResult.buildInfo.program.fileNames, buildInfoResult.buildInfoPath, host.getCurrentDirectory(), createGetCanonicalFileName(compilerHost.useCaseSensitiveFileNames()));
const compilerOptions = buildInfoResult.buildInfo.program.options ?
convertToOptionsWithAbsolutePaths(buildInfoResult.buildInfo.program.options, buildInfoFilePathDecoder.toAbsolutePath) :
{};
compilerOptions.configFilePath = options.configFilePath;
return host => createOldBuildInfoProgram(
host,
options,
/*cacheResolutions*/ undefined,
{
cache: buildInfoResult.buildInfo.program!.cacheResolutions!,
getProgramBuildInfoFilePathDecoder: () => buildInfoFilePathDecoder
},
);
}

function synchronizeHostData(): void {
Debug.assert(languageServiceMode !== LanguageServiceMode.Syntactic);
// perform fast check if host supports it
Expand Down Expand Up @@ -1710,15 +1736,13 @@ export function createLanguageService(
// instance. If we cancel midway through, we may end up in an inconsistent state where
// the program points to old source files that have been invalidated because of
// incremental parsing.

const options: CreateProgramOptions = {
program = createProgram({
rootNames: rootFileNames,
options: newSettings,
host: compilerHost,
oldProgram: program,
oldProgram: getOldProgram(newSettings, compilerHost),
projectReferences
};
program = createProgram(options);
});

// 'getOrCreateSourceFile' depends on caching but should be used past this point.
// After this point, the cache needs to be cleared to allow all collected snapshots to be released
Expand Down
88 changes: 35 additions & 53 deletions tests/baselines/reference/tsserver/cacheResolutions/dts-has-fewer-resolutions-than-ts.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,49 +702,31 @@ Info 14 [00:01:28.000] Config: /user/username/projects/sample1/core/tsconfig.j
Info 15 [00:01:29.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/tsconfig.json 2000 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Config file
Info 16 [00:01:30.000] DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core 1 undefined Config: /user/username/projects/sample1/core/tsconfig.json WatchType: Wild card directory
Info 17 [00:01:31.000] Elapsed:: *ms DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core 1 undefined Config: /user/username/projects/sample1/core/tsconfig.json WatchType: Wild card directory
Info 18 [00:01:32.000] ======== Resolving module '../logic' from '/user/username/projects/sample1/tests/index.ts'. ========
Info 19 [00:01:33.000] Module resolution kind is not specified, using 'NodeJs'.
Info 20 [00:01:34.000] Loading module as file / folder, candidate module location '/user/username/projects/sample1/logic', target file types: TypeScript, Declaration.
Info 21 [00:01:35.000] File '/user/username/projects/sample1/logic.ts' does not exist.
Info 22 [00:01:36.000] File '/user/username/projects/sample1/logic.tsx' does not exist.
Info 23 [00:01:37.000] File '/user/username/projects/sample1/logic.d.ts' does not exist.
Info 24 [00:01:38.000] File '/user/username/projects/sample1/logic/package.json' does not exist.
Info 25 [00:01:39.000] File '/user/username/projects/sample1/logic/index.ts' exist - use it as a name resolution result.
Info 26 [00:01:40.000] ======== Module name '../logic' was successfully resolved to '/user/username/projects/sample1/logic/index.ts'. ========
Info 27 [00:01:41.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/logic/index.ts 500 undefined WatchType: Closed Script info
Info 28 [00:01:42.000] ======== Resolving module '../core/myClass' from '/user/username/projects/sample1/logic/index.ts'. ========
Info 29 [00:01:43.000] Using compiler options of project reference redirect '/user/username/projects/sample1/logic/tsconfig.json'.
Info 30 [00:01:44.000] Module resolution kind is not specified, using 'NodeJs'.
Info 31 [00:01:45.000] Loading module as file / folder, candidate module location '/user/username/projects/sample1/core/myClass', target file types: TypeScript, Declaration.
Info 32 [00:01:46.000] File '/user/username/projects/sample1/core/myClass.ts' exist - use it as a name resolution result.
Info 33 [00:01:47.000] ======== Module name '../core/myClass' was successfully resolved to '/user/username/projects/sample1/core/myClass.ts'. ========
Info 34 [00:01:48.000] ======== Resolving module '../core' from '/user/username/projects/sample1/logic/index.ts'. ========
Info 35 [00:01:49.000] Using compiler options of project reference redirect '/user/username/projects/sample1/logic/tsconfig.json'.
Info 36 [00:01:50.000] Module resolution kind is not specified, using 'NodeJs'.
Info 37 [00:01:51.000] Loading module as file / folder, candidate module location '/user/username/projects/sample1/core', target file types: TypeScript, Declaration.
Info 38 [00:01:52.000] File '/user/username/projects/sample1/core.ts' does not exist.
Info 39 [00:01:53.000] File '/user/username/projects/sample1/core.tsx' does not exist.
Info 40 [00:01:54.000] File '/user/username/projects/sample1/core.d.ts' does not exist.
Info 41 [00:01:55.000] File '/user/username/projects/sample1/core/package.json' does not exist.
Info 42 [00:01:56.000] File '/user/username/projects/sample1/core/index.ts' exist - use it as a name resolution result.
Info 43 [00:01:57.000] ======== Module name '../core' was successfully resolved to '/user/username/projects/sample1/core/index.ts'. ========
Info 44 [00:01:58.000] ======== Resolving module '../core/anotherClass' from '/user/username/projects/sample1/logic/index.ts'. ========
Info 45 [00:01:59.000] Using compiler options of project reference redirect '/user/username/projects/sample1/logic/tsconfig.json'.
Info 46 [00:02:00.000] Module resolution kind is not specified, using 'NodeJs'.
Info 47 [00:02:01.000] Loading module as file / folder, candidate module location '/user/username/projects/sample1/core/anotherClass', target file types: TypeScript, Declaration.
Info 48 [00:02:02.000] File '/user/username/projects/sample1/core/anotherClass.ts' exist - use it as a name resolution result.
Info 49 [00:02:03.000] ======== Module name '../core/anotherClass' was successfully resolved to '/user/username/projects/sample1/core/anotherClass.ts'. ========
Info 50 [00:02:04.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/myClass.ts 500 undefined WatchType: Closed Script info
Info 51 [00:02:05.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/index.ts 500 undefined WatchType: Closed Script info
Info 52 [00:02:06.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/anotherClass.ts 500 undefined WatchType: Closed Script info
Info 53 [00:02:07.000] FileWatcher:: Added:: WatchInfo: /a/lib/lib.d.ts 500 undefined WatchType: Closed Script info
Info 54 [00:02:08.000] DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/tests/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 55 [00:02:09.000] Elapsed:: *ms DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/tests/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 56 [00:02:10.000] DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 57 [00:02:11.000] Elapsed:: *ms DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 58 [00:02:12.000] Finishing updateGraphWorker: Project: /user/username/projects/sample1/tests/tsconfig.json Version: 1 structureChanged: true structureIsReused:: Not Elapsed:: *ms
Info 59 [00:02:13.000] Project '/user/username/projects/sample1/tests/tsconfig.json' (Configured)
Info 60 [00:02:14.000] Files (6)
Info 18 [00:01:32.000] Reusing resolution of module '../logic' from '/user/username/projects/sample1/tests/index.ts' found in cache from location '/user/username/projects/sample1/tests', it was successfully resolved to '/user/username/projects/sample1/logic/index.ts'.
Info 19 [00:01:33.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/logic/index.ts 500 undefined WatchType: Closed Script info
Info 20 [00:01:34.000] Reusing resolution of module '../core/myClass' from '/user/username/projects/sample1/logic/index.ts' found in cache from location '/user/username/projects/sample1/logic', it was successfully resolved to '/user/username/projects/sample1/core/myClass.ts'.
Info 21 [00:01:35.000] Reusing resolution of module '../core/anotherClass' from '/user/username/projects/sample1/logic/index.ts' found in cache from location '/user/username/projects/sample1/logic', it was successfully resolved to '/user/username/projects/sample1/core/anotherClass.ts'.
Info 22 [00:01:36.000] ======== Resolving module '../core' from '/user/username/projects/sample1/logic/index.ts'. ========
Info 23 [00:01:37.000] Using compiler options of project reference redirect '/user/username/projects/sample1/logic/tsconfig.json'.
Info 24 [00:01:38.000] Module resolution kind is not specified, using 'NodeJs'.
Info 25 [00:01:39.000] Loading module as file / folder, candidate module location '/user/username/projects/sample1/core', target file types: TypeScript, Declaration.
Info 26 [00:01:40.000] File '/user/username/projects/sample1/core.ts' does not exist.
Info 27 [00:01:41.000] File '/user/username/projects/sample1/core.tsx' does not exist.
Info 28 [00:01:42.000] File '/user/username/projects/sample1/core.d.ts' does not exist.
Info 29 [00:01:43.000] File '/user/username/projects/sample1/core/package.json' does not exist.
Info 30 [00:01:44.000] File '/user/username/projects/sample1/core/index.ts' exist - use it as a name resolution result.
Info 31 [00:01:45.000] ======== Module name '../core' was successfully resolved to '/user/username/projects/sample1/core/index.ts'. ========
Info 32 [00:01:46.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/myClass.ts 500 undefined WatchType: Closed Script info
Info 33 [00:01:47.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/index.ts 500 undefined WatchType: Closed Script info
Info 34 [00:01:48.000] FileWatcher:: Added:: WatchInfo: /user/username/projects/sample1/core/anotherClass.ts 500 undefined WatchType: Closed Script info
Info 35 [00:01:49.000] FileWatcher:: Added:: WatchInfo: /a/lib/lib.d.ts 500 undefined WatchType: Closed Script info
Info 36 [00:01:50.000] DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/tests/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 37 [00:01:51.000] Elapsed:: *ms DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/tests/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 38 [00:01:52.000] DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 39 [00:01:53.000] Elapsed:: *ms DirectoryWatcher:: Added:: WatchInfo: /user/username/projects/sample1/node_modules/@types 1 undefined Project: /user/username/projects/sample1/tests/tsconfig.json WatchType: Type roots
Info 40 [00:01:54.000] Finishing updateGraphWorker: Project: /user/username/projects/sample1/tests/tsconfig.json Version: 1 structureChanged: true structureIsReused:: SafeModuleCache Elapsed:: *ms
Info 41 [00:01:55.000] Project '/user/username/projects/sample1/tests/tsconfig.json' (Configured)
Info 42 [00:01:56.000] Files (6)
/a/lib/lib.d.ts
/user/username/projects/sample1/core/myClass.ts
/user/username/projects/sample1/core/index.ts
Expand All @@ -766,16 +748,16 @@ Info 60 [00:02:14.000] Files (6)
index.ts
Matched by default include pattern '**/*'

Info 61 [00:02:15.000] -----------------------------------------------
Info 62 [00:02:16.000] Search path: /user/username/projects/sample1/tests
Info 63 [00:02:17.000] For info: /user/username/projects/sample1/tests/tsconfig.json :: No config files found.
Info 64 [00:02:18.000] Project '/user/username/projects/sample1/tests/tsconfig.json' (Configured)
Info 64 [00:02:19.000] Files (6)
Info 43 [00:01:57.000] -----------------------------------------------
Info 44 [00:01:58.000] Search path: /user/username/projects/sample1/tests
Info 45 [00:01:59.000] For info: /user/username/projects/sample1/tests/tsconfig.json :: No config files found.
Info 46 [00:02:00.000] Project '/user/username/projects/sample1/tests/tsconfig.json' (Configured)
Info 46 [00:02:01.000] Files (6)

Info 64 [00:02:20.000] -----------------------------------------------
Info 64 [00:02:21.000] Open files:
Info 64 [00:02:22.000] FileName: /user/username/projects/sample1/tests/index.ts ProjectRootPath: undefined
Info 64 [00:02:23.000] Projects: /user/username/projects/sample1/tests/tsconfig.json
Info 46 [00:02:02.000] -----------------------------------------------
Info 46 [00:02:03.000] Open files:
Info 46 [00:02:04.000] FileName: /user/username/projects/sample1/tests/index.ts ProjectRootPath: undefined
Info 46 [00:02:05.000] Projects: /user/username/projects/sample1/tests/tsconfig.json
After request

PolledWatches::
Expand Down Expand Up @@ -810,7 +792,7 @@ FsWatchesRecursive::
/user/username/projects/sample1/core:
{}

Info 64 [00:02:24.000] response:
Info 46 [00:02:06.000] response:
{
"responseRequired": false
}
Loading
0