8000 RepositoryStatus sometimes crashes with an AccessViolationException · Issue #1513 · libgit2/libgit2sharp · GitHub
[go: up one dir, main page]
Skip to content

RepositoryStatus sometimes crashes with an AccessViolationException #1513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jcansdale opened this issue Nov 20, 2017 · 13 comments
Open

RepositoryStatus sometimes crashes with an AccessViolationException #1513

jcansdale opened this issue Nov 20, 2017 · 13 comments

Comments

@jcansdale
Copy link
Contributor
jcansdale commented Nov 20, 2017 •
edited
Loading

We've noticed that the GitHub Extension for Visual Studio has been crashing with an AccessViolationException. This often happens when a user previews a PR and the extension calls RetrieveStatus. This exception occurs randomly and is thrown by the git_status_list_new method.

This exception is most easily reproduced using 0.24 or 0.25.0-preview-0033, but it does also happen with 0.23.1 (but I haven't seen it using the repro below).

How to reproduce

  1. Clone https://github.com/github/VisualStudio.git to C:\source\github.com\github\VisualStudio\.
  2. Compile and run the following code in Release configuration using LibGit2Sharp v0.24:
        static void Main(string[] args)
        {
            var path = @"C:\source\github.com\github\VisualStudio";
            for (int count = 0; count < 1000; count++)
            {
                var status = new Repository(path).RetrieveStatus();
                Console.WriteLine(count + ": " + status.IsDirty);
            }
        }

I tried to create a standalone app, but wasn't able to trigger it with a very simple repo. I'm using the github\VisualStudio one because I'm most familiar it, but it does this with many others as well.

What to expect

The most common exception is as follows:

System.AccessViolationException was unhandled
  HResult=-2147467261
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=LibGit2Sharp
  StackTrace:
       at LibGit2Sharp.Core.NativeMethods.git_status_list_new(git_status_list*& git_status_list, git_repository* repo, GitStatusOptions options)
       at LibGit2Sharp.Core.Proxy.git_status_list_new(RepositoryHandle repo, GitStatusOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\Core\Proxy.cs:line 2931
       at LibGit2Sharp.RepositoryStatus..ctor(Repository repo, StatusOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\RepositoryStatus.cs:line 60
       at LibGit2Sharp.RepositoryExtensions.RetrieveStatus(IRepository repository) in C:\projects\libgit2sharp\LibGit2Sharp\RepositoryExtensions.cs:line 696
       at Libgit2Repro.Program.Main(String[] args) in c:\users\passp\onedrive\documents\visual studio 2015\Projects\Libgit2Repro\Libgit2Repro\Program.cs:line 20
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()

But sometimes NotFoundExceptions are thrown as well.

image

Related issues / PRs
@bording
Copy link
Member
bording commented Nov 21, 2017

@jcansdale How many times do you have to run the program to trigger the crash? What framework are you using when you experience the crash?

Using 0.25.0-preview-0033, I've run it several times with both net461 and netcoreapp2.0, and I haven't seen it crash yet.

However, I wasn't able to fully clone https://github.com/github/VisualStudio.git because it has a private submodule that I don't have access to:

fatal: clone of 'git@github.com:github/VisualStudioBuildScripts' into submodule path 'C:/source/github.com/github/VisualStudio/script' failed

That might be why I'm not able to see it crash. Do you know of another repository that exhibits the same problem that I can fully clone?

@grokys
Copy link
grokys commented Nov 21, 2017

@bording ah yes, sorry that submodule is private - it contains our private keys etc. The build instructions cover it, you need to do a git submodule deinit script.

@grokys
Copy link
grokys commented Nov 21, 2017

Regarding reproing the crash, are you running in Release mode? The crash doesn't happen in Debug.

@carlosmn
Copy link
Member

It'd be interesting to see what the C stack trace looks like, but I certainly suspect the repository is being disposed and thus freed while we're in the middle of trying to run the status. It might need a sizeable repository in order for it to take long enough for GC to take effect.

@jcansdale
Copy link
Contributor Author

@bording,

How many times do you have to run the program to trigger the crash? What framework are you using when you experience the crash?

Often I would run it once and it would crash within 10 iterations. Just now I struggled a bit to get the crash. It seems to crash pretty soon or not at all. Maybe try restarting the app a few times?

It did finally crash:

image

I've got a crash .dmp file from this if that would be useful?

@bording
Copy link
Member
bording commented Nov 21, 2017

I'm definitely running in release mode.

I went ahead and re-cloned the repo and removed the script submodule, butI have not been able to see a single crash. I've run the app many times now, and it's working fine.

@grokys @jcansdale I'm assuming you've got the script module actually checked out when you're running into the problem? Have you tried it with a clone of the repo without the script submodule?

If the presence of script submodule isn't relevant to causing the crash, then there must be some other variable that isn't yet accounted for.

What OS are you running on? What version of the .NET Framework do you have installed, and which one are you targeting? Are running the app as 32 or 64 bit?

I'm not sure any of that would make a difference, but I'm trying to think of what else it could be.

@jcansdale The dmp file might be helpful, if you've got a way to share it.

@carlosmn
Copy link
Member

Here's a stack trace of one of the times it crashed. The top points to https://github.com/libgit2/libgit2/blob/a5cf255b471ad7113247d552d5695db0cb720882/src/attrcache.c#L182 which is a call to the inlined https://github.com/libgit2/libgit2/blob/a5cf255b471ad7113247d552d5695db0cb720882/src/attrcache.c#L27-L36 which just looks up the entry in a dictionary.

It's still trying to look up address 0x14 which keeps looking like we cleaned up the repo, or at least the cache midway and we're trying to access the struct at NULL.

The crash is reproducing by opening the Avalonia PR page in GHfVS and clicking around a few different ones.

>    ntdll.dll!77e0b2e3()    Unknown
     [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]    
     git2-a5cf255.dll!attr_cache_lookup(git_attr_file * * out_file, git_attr_file_entry * * out_entry, git_repository * repo, git_attr_session * attr_session, git_attr_file_source source, const char * base, const char * filename) Line 182    C
     git2-a5cf255.dll!git_attr_cache__get(git_attr_file * * out, git_repository * repo, git_attr_session * attr_session, git_attr_file_source source, const char * base, const char * filename, int(__stdcall*)(git_repository *, git_attr_file *, const char *) parser) Line 215    C
     git2-a5cf255.dll!git_ignore__push_dir(git_ignores * ign, const char * dir) Line 352    C
     git2-a5cf255.dll!filesystem_iterator_frame_push_ignores(filesystem_iterator * iter, filesystem_iterator_entry * frame_entry, filesystem_iterator_frame * new_frame) Line 1170    C
     git2-a5cf255.dll!filesystem_iterator_frame_push(filesystem_iterator * iter, filesystem_iterator_entry * frame_entry) Line 1353    C
     git2-a5cf255.dll!filesystem_iterator_advance_into(const git_index_entry * * out, git_iterator * i) Line 1576    C
     git2-a5cf255.dll!handle_unmatched_new_item(git_diff_generated * diff, diff_in_progress * info) Line 1055    C
     git2-a5cf255.dll!git_diff__from_iterators(git_diff * * out, git_repository * repo, git_iterator * old_iter, git_iterator * new_iter, const git_diff_options * opts) Line 1235    C
     git2-a5cf255.dll!git_diff_index_to_workdir(git_diff * * out, git_repository * repo, git_index * index, const git_diff_options * opts) Line 1377    C
     git2-a5cf255.dll!git_status_list_new(git_status_list * * out, git_repository * repo, const git_status_options * opts) Line 346    C

@jcansdale jcansdale mentioned this issue Nov 23, 2017
Merged
@hybridherbst
Copy link
hybridherbst commented Oct 20, 2019 •
edited
Loading

I've been following the trail for all those libgit2sharp crashes I'm experiencing (I'm trying to use it from inside Unity, with random crashes all the time, most notably on domain reload but also in random situtations), which led me here.

Has there ever been a consensus on what causes this? Is the workaround (two years later) to still use 0.23? The same is still happening on 0.26.1 with libgit2 2.0.298, and also latest preview versions.

@jcansdale
Copy link
Contributor Author

Iirc, it was caused by objects being collected before native methods return. This was counter intuative as a developer used to managed code. The fix was to always use using blocks for objects that are disposable (most libgit2sharp types).

I hope that helps. It was a while ago we last faced this issue.

@hybridherbst
Copy link
hybridherbst commented Oct 20, 2019 •
edited
Loading

I was able to pinpoint a couple more cases where that happened and have something that feels "kinda stable" right now. Same as you, I went to "using() everywhere" (I was caching the repository object before).

Additionally, the bug that took me the longest to find was that inside the using(), I was using LINQ queries and forgot about deferred execution - when execution was actually happening at some random other point in time, the repository object was long gone and I got those native crashes. Forcing execution inside the using() block with ToList() solved that part of the issue.

@jcansdale
Copy link
Contributor Author

I was caching the repository object before

Yup, I think we were caching repository objects as well. It gets particularly confusing when Debug builds work, but Release builds don't!

@hybridherbst
Copy link

And stepping through with debugger also always works - seems to prevent the repository object GC as well...

@DaveSenn
Copy link
DaveSenn commented Jun 9, 2021 •
edited
Loading

I've the same problem with 0.26 and 0.27 preview:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at LibGit2Sharp.Core.NativeMethods.git_status_list_new(LibGit2Sharp.Core.git_status_list* ByRef, LibGit2Sharp.Core.git_repository*, LibGit2Sharp.Core.GitStatusOptions)
--------------------------------
   at LibGit2Sharp.Core.Proxy.git_status_list_new(LibGit2Sharp.Core.Handles.RepositoryHandle, LibGit2Sharp.Core.GitStatusOptions)
   at LibGit2Sharp.RepositoryStatus..ctor(LibGit2Sharp.Repository, LibGit2Sharp.StatusOptions)
   at LibGit2Sharp.Repository.RetrieveStatus(LibGit2Sharp.StatusOptions

Just seen #1886 ... my issue also started after an update of VS2019 to 16.10.1. (This update also contained the latest .NET 5 SDK 5.0.7)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@carlosmn @bording @grokys @hybridherbst @DaveSenn @jcansdale
0