-
Notifications
You must be signed in to change notification settings - Fork 7.6k
Replace string-compare-based test for copying to same file with more … #3441
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@jeffbi I updated the description as this seems the fix for #1930, but please correct it if I'm wrong. The last comment of #1930 shows:
@SteveL-MSFT, do you mean that bug had already been fixed by @iSazonov? |
@daxian-dbw Yes, it is a fix for #1930. I had that at the end of the pull-request title, but it looks like it got clipped off. I've updated the description with more information. I believe @SteveL-MSFT's comment referred to fixing the wording on an earlier comment. |
It was just a typo. |
I tested Test-Path /etc/sysctl.conf
Test-Path /etc/sysctl.CONF on WSL and Test-Path is case-sensitive. Do we really need additional code here? |
@daxian-dbw yeah, the 'fixed' statement referred to a typo in the comment @iSazonov caught, nit that the issue was fixed |
@iSazonov The goal of the original code was to ensure that the cmdlet did not attempt to copy a file onto itself, and it did so by doing a case-insensitive compare of the two normalized paths. That worked fine on FAT volumes, on NTFS volumes under Windows in its default mode, and on HFS+ volumes on OS X. In your example, it would consider Then consider links. If file A is a file on a volume, and file B is a hard or symbolic link to file A, copying A to B would actually be copying A onto itself, which the cmdlet does not allow but cannot be detected by merely comparing two path strings. The new code determines whether two paths refer to the same file (the goal of the original code) regardless of what the paths appear to be. It works on HFS+, ext, FAT, and NTFS file systems. For NTFS file systems, it does the right thing under Windows, where the volume is case-insensitive, and on Mac/Linux where the volume is case-sensitive. And it works for hard and symbolic links on those file systems that support them. |
My question was only whether we can use existing code without new low level code? Name : powershell
Length : 0
CreationTime : 3/21/2017 11:18:52 AM
LastWriteTime : 3/21/2017 11:18:52 AM
LastAccessTime : 3/21/2017 11:18:52 AM
Mode : -a---l
LinkType : SymbolicLink
Target : C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\debug\powershell.exe
VersionInfo : File: C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\debug\powershell Here So here we can make the code more simple. We should create a new API to compare path and place it in engen\Utils.cs or PathUtils.cs |
This works differently in Linux.
Here, the symbolic link points to its target, but the hard link does not. Also, once the hard link was created, the original file became a hard link as well. Linux (or the ext file system) does not distinguish between a hard link and the "original" file. A slightly off-topic question: Note that the symbolic link's attributes property says "ReparsePoint". That makes sense for Windows, but does it for Linux? |
@jeffbi Sorry for mesh in my previous comment.
I believe we should fix this locally - behavior on all platforms must be the same. So my suggestion is don't block the PR, continue the code review and open new Issue (if you confirm that we have inconsistency in file provider with soft/hard links on different platforms). |
Is this being reviewed, or is it awaiting something from me? |
I originally thought that we already have full compliance with links on all platforms as associated problems has been closed both locally and in CoreFX. As you pointed out it is not. I propose to open a new issue and collect the inconsistencies there. Anyway we need to fix it now or later. I need your help with this on Unix. @daxian-dbw Could you please review my thoughts? |
@iSazonov I don't think we can "fix" My low-level code does not attempt to find all paths to an inode, just whether two paths point to the same inode. There is an open request issue in corefx (dotnet/corefx#10120) to implement this in corefx. In PowerShell 5.1 on my Windows 10 box, the I have come across a few other issues that are related to links, one in New-Item and two that directly affect #621 (the latter two are, I believe, corefx issues: see dotnet/corefx#17843 and dotnet/corefx#17844). Do we want to create a single issue as a catch-all for link-related items? |
@jeffbi Great ** 2! 😄 Thanks!
Sorry for the delay of the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add tests.
src/libpsl-native/src/issamefile.cpp
Outdated
@@ -0,0 +1,47 @@ | |||
//! @file isdirectory.cpp | |||
//! @author Andrew Schwartzmeyer <andschwa@microsoft.com> | |||
//! @brief returns if the path is a directory |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please renew the comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
src/libpsl-native/src/issamefile.cpp
Outdated
//! char* is marshaled as an LPStr, which on Linux is UTF-8. | ||
//! @endparblock | ||
//! | ||
//! @param[in] path_one |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
path_two
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
src/libpsl-native/src/issamefile.cpp
Outdated
//! char* is marshaled as an LPStr, which on Linux is UTF-8. | ||
//! @endparblock | ||
//! | ||
//! @retval true if directory, false otherwise |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please correct the comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
src/libpsl-native/src/issamefile.cpp
Outdated
//! | ||
//! @retval true if directory, false otherwise | ||
//! | ||
bool IsSameFile(const char* path_one, const char* path_two) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe it works for both files and directories. Parameters is "path" too. So maybe rename to IsSamePath
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that this could use a better name, since it does work for both files and directories, but I'm not sure that IsSamePath
is the best choice. I avoided IsSamePath
because two different paths can arrive at the same destination. I went with IsSameFile
just because the docs for the stat
function and the GetFileInformationByHandle
both just use the term "file" even though they both work for files and directories.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe good comments (on high level API too) and/or IsSameTarget
or something like that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to IsSameFileSystemItem
#endif | ||
} | ||
|
||
internal static bool WinIsSameFile(string pathOne, string pathTwo) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same as for IsSameFile
- maybe rename to WinIsSamePath
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to IsSameFileSystemItem
var access = FileDesiredAccess.GenericRead; | ||
var share = FileShareMode.Read; | ||
var creation = FileCreationDisposition.OpenExisting; | ||
var attributes = FileAttributes.BackupSemantics; // consider FileAttributes.PosixSemantics as well |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove the comment or make it full and clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
var attributes = FileAttributes.BackupSemantics; // consider FileAttributes.PosixSemantics as well | ||
var handleOne = CreateFile(pathOne, access, share, IntPtr.Zero, creation, attributes, IntPtr.Zero); | ||
|
||
using (SafeFileHandle sfOne = new SafeFileHandle(handleOne, true)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use NativeMethods.CreateFile
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really, but that code could have been a lot better. I've updated it.
{ | ||
String error = StringUtil.Format(FileSystemProviderStrings.CopyError, path); | ||
Exception e = new IOException(error); | ||
WriteError(new ErrorRecord(e, "CopyError", ErrorCategory.WriteError, path)); | ||
return; | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unneeded formatting. Please renew the comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
{ | ||
String error = StringUtil.Format(FileSystemProviderStrings.CopyError, destinationPath); | ||
Exception e = new IOException(error); | ||
WriteError(new ErrorRecord(e, "CopyError", ErrorCategory.WriteError, destinationPath)); | ||
return; | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unneeded formatting. Please renew the comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
@daxian-dbw This has been waiting for approval/merge of PR #3509 because the Pester tests should be using New-Item to create directory symbolic links. I can get around that by shelling out to cmd.exe to run the mklink program, but that would mean revisiting the test script once the PR has been merged. Perhaps I could open a new issue to update the test. Would this be acceptable? |
@daxian-dbw I think there are discussions required for #3509 regarding the proper way to create symlinks in various circumstances under Windows, and that that's why approval/merging of that PR is taking a while. |
…accurate, cross-platform test (#1930) Changes per code review * Cleaned up comments in .cpp file * Renamed IsSameFile to IsSameFileSystemItem * Removed comment about PosixSemantics * Added PosixSemantics to CreateFile call * Provide additional information in exception when paths point to same destination. * Added tests
$principal = New-Object System.Security.Principal.WindowsPrincipal($WinId) | ||
$admin = [System.Security.Principal.WindowsBuiltInRole]::Administrator | ||
|
||
return $principal.IsInRole($admin) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usually we set -Tags @('CI', 'RequireAdminOnWindows')
.
catch | ||
{ | ||
# we expected the copy to fail and it did. make sure it failed the right way | ||
$_.FullyQualifiedErrorId | Should Be "CopyError,Microsoft.PowerShell.Commands.CopyItemCommand" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use ShouldBeErrorId
$exc = {
... } | ShouldBeErrorId ...
$exc.Exception | Should BeOfType System.IO.IOException
...
{ | ||
# we expected the copy to succeed. make sure it did | ||
Test-Path $testCase.Destination | Should Be $true | ||
Remove-Item -Path $testCase.Destination -Force -ErrorAction SilentlyContinue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If previous Should Be
is failed the cleanup code will be unreacheable.
Please move the cleanup code to AfterEach
.
# Test the ability to avoid an item copying onto itself | ||
function TestSelfCopy($testCase) | ||
{ | ||
It "$($testCase.Name)" -Skip:($testCase.SkipIf) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With moving to 'RequireAdminOnWindows'
I believe we can use It "" -TestCase $testCases
directly without TestSelfCopy
@@ -629,14 +801,14 @@ Describe "Extended FileSystem Path/Location Cmdlet Provider Tests" -Tags "Featur | |||
$result = Split-Path -Path $level1_0Full -Leaf | |||
$result | Should Be $level1_0 | |||
} | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It distracts attention. We should make separate PR for formatting.
#pragma once | ||
|
||
#include "pal.h" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extra line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Discarded.
@@ -0,0 +1,48 @@ | |||
//! @file issamefilesystemitem.cpp | |||
//! @author Jeff Bienstadt <v-jebien@microsoft.com> | |||
//! @brief returns if two paths ultimately point to the same filesystem object |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
returns what?
Below the same.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please correct this.
|
||
#include "getstat.h" | ||
#include "issamefilesystemitem.h" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extra line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Discarded.
@iSazonov In response to your review:
|
We don't test Windows behavior, we test PowerShell behavior. So the tests is the same for elevated and non-elevated sessions. In other words if you perform these tests with Unix tests is not skipped if
Based on my comments we can use this.
My suggesion is
I agree with using current pattern. |
var access = FileAccess.Read; | ||
var share = FileShare.Read; | ||
var creation = FileMode.Open; | ||
var attributes = FileAttributes.BackupSemantics | FileAttributes.PosixSemantics; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please clarify - do we really need PosixSemantics here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's for if/when we find ourselves in running in a Windows environment in which case-sensitivity has been turned on. Since the purpose of the function is to see if two files/directories are the same, it made sense to me to give it every advantage in making a correct assessment.
@iSazonov I've re-worked the structure of the tests. |
The Travis CI build failure puzzles me. It appears to have succeeded on Linux but failed on OS X. It appears that the offending line is
and it looks to me like it's ignoring the |
@jeffbi That error seems to indicate a bug in the product code. The error string is from FileSystemProviderStrings.resx:
It looks to me an exception with this resource string got thrown. Searching where this resource string is used in code might lead you somewhere. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -246,19 +246,16 @@ Describe "Basic FileSystem Provider Tests" -Tags "CI" { | |||
} | |||
} | |||
|
|||
Describe "Copy-Item can avoid copying an item onto itself" -Tags "CI" { | |||
|
|||
Describe "Copy-Item can avoid copying an item onto itself" -Tags "CI", "RequireAdminOnWindows" { | |||
BeforeAll { | |||
# In Windows, only Administrators can create symbolic links. | |||
# In Unix, anyone can. | |||
function CanMakeSymlink |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unneeded function because RequireAdminOnWindows
ensures elevated rights.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Function removed, along with use of $canSymLink
@@ -280,7 +277,8 @@ Describe "Copy-Item can avoid copying an item onto itself" -Tags "CI" { | |||
# Names of files, directories, and links we'll be using to test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is obvious and useless. Please remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
@@ -291,7 +289,8 @@ Describe "Copy-Item can avoid copying an item onto itself" -Tags "CI" { | |||
$symdToOther = "$subDir/symd-to-other" | |||
$junctionToOther = "$subDir/junction-to-other" | |||
|
|||
# Set up our directories and links | |||
# Set up our files, directories, links |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is obvious and useless. Please remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
SelfCopyExpected = $true | ||
else | ||
{ | ||
Copy-Item -Path $sourcePath -Destination $destinationPath -ErrorAction SilentlyContinue | ShouldBeErrorId "CopyError,Microsoft.PowerShell.Commands.CopyItemCommand" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We cannot use -ErrorAction SilentlyContinue
with ShouldBeErrorId and we should use script block:
{ Copy-Item -Path $sourcePath -Destination $destinationPath -ErrorAction Stop } | ShouldBeErrorId "CopyError,Microsoft.PowerShell.Commands.CopyItemCommand"
{ | ||
Copy-Item -Path $sourcePath -Destination $destinationPath -ErrorAction SilentlyContinue | ShouldBeErrorId "CopyError,Microsoft.PowerShell.Commands.CopyItemCommand" | ||
$_.Exception | Should BeOfType System.IO.IOException | ||
$_.Exception.Data[$selfCopyKey] | Should Not Be $null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The template is:
$exc = {
{ Copy-Item -Path $sourcePath -Destination $destinationPath -ErrorAction Stop } | ShouldBeErrorId "CopyError,Microsoft.PowerShell.Commands.CopyItemCommand"
}
$exc.Exception.Data[$selfCopyKey] | Should Not Be $null
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed the test code to use $Error[0]
rather than $_
. I'll gladly update to use your template when ShouldBeErrorId
returns something other than Boolean
.
} | ||
Context "Copy-Item avoids copying an item onto itself" { | ||
BeforeAll { | ||
# Set up our test cases |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same about comment.
@daxian-dbw Thanks for the suggestion. I'll take a look tomorrow. It's odd because I've run the Pester tests on all three platforms, and run the |
# For now, we'll assume the tests are running the platform's | ||
# native filesystem, in its default mode | ||
$isCaseSensitive = $IsLinux | ||
$canSymlink = CanMakeSymlink | ||
$canDirSymLink = $IsWindows -And (CanMakeSymlink) | ||
$canDirSymLink = $IsWindows |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not used and can be removed.
# The source and destination paths must exist. | ||
# If either do not, it's because they could not be created in the BeforeAll block | ||
# This is not a test failure, but rather a precondition failure. | ||
if ((-Not (Test-Path -Path $Source)) -or (-Not (Test-Path -Path $Destinat 4E34 ion))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please clarify 'ShouldBeErrorId' below don't fail if the items don't exist?
If failed please remove the "if-return".
If not failed we should replace the if-return
with Should
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed if
for source file existence.
@iSazonov Can you look this over please? |
) | ||
|
||
# Junctions and directory symbolic links are Windows and NTFS only | ||
if ($IsWindows) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, we shouldn't skip tests silently 😕
Maybe define the test body (below in "It" block) as function and make two "It" blocks (second "skip on non-Windows")?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Revise comments in C++ code.
C++ comments updated |
@jeffbi I don't see last commit (with C++) |
Thanks! I see now. LGTM. |
@iSazonov Thanks for the review! 👍 |
@jeffbi Do you specifically choose a work with the most sophisticated tests? 😄 |
Fix #1930.
Rather than relying on case-insensitive string compares of source and destination paths, use operating system calls to determine whether two paths refer to the same file. This solves not only the case-insensitivity issue but also allows the cmdlet to operate properly if the destination is a hard or symbolic link to the source.
The Windows side is implemented in C#. The Unix side is implemented partially in native code.