8000 The configured user limit (128) on the number of inotify instances has been reached... · Issue #411 · sourcegit-scm/sourcegit · GitHub
[go: up one dir, main page]

Skip to content

The configured user limit (128) on the number of inotify instances has been reached... #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ScruffyDerHausmeister opened this issue Aug 25, 2024 · 13 comments
Assignees
Labels
enhancement New feature or request os-linux This issue only exists on Linux

Comments

@ScruffyDerHausmeister
Copy link
ScruffyDerHausmeister commented Aug 25, 2024

I've got an error when opening a repository on linux (Zorin OS / Ubuntu).

Crash::: System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
crash_2024-08-25_20-48-56.log

@gadfly3173
Copy link
Contributor
gadfly3173 commented Aug 26, 2024

You can add these to /etc/sysctl.conf .

fs.inotify.max_user_instances=1024
fs.inotify.max_user_watches=1048576

This error occurs when listening for a large number of files on Linux.
Edit: To solve this problem from the software level, it may need to catch the IOException thrown when starting the Watcher and set the Watcher to polling mode.

@ScruffyDerHausmeister
Copy link
Author

It'd be nice to find a solution without changing systemfiles which never were needed to be changed.

@love-linger
Copy link
Collaborator

Using lsof -c sourcegit command to show the file descriptors opened by SourceGit on Ubuntu 20.04 (WSL 2).

Note

When a repository is opened in SourceGit, we will create two FileSystemWatcher - one for repository's git dir and another for repository's worktree.

The main difference:

image

@love-linger
Copy link
Collaborator

Another test

image

It seems like that when we create a new FileSystemWatcher, a a_inode FD will be opened.

@love-linger
Copy link
Collaborator

According to the test results, the FDs did not increase beyond the expected situation.

@love-linger love-linger self-assigned this Aug 26, 2024
@gadfly3173
Copy link
Contributor
gadfly3173 commented Aug 26, 2024

According to the test results, the FDs did not increase beyond the expected situation.

The upper limit that is usually reached is fs.inotify.max_user_instances. When I opened 4 IDEAs, 1 Rider, 2 Tomcats, and 1 SourceGit (opened 6 repositories), I could see that the number of instances reached 105. Opening another SourceGit will bring the number to 117.

   INOTIFY   INSTANCES
   WATCHES      PER   
    COUNT     PROCESS   PID USER         COMMAND
------------------------------------------------------------
   15703        13       40694 homolo      /opt/sourcegit/sourcegit
   14907         1        3981 homolo      /snap/intellij-idea-ultimate/524/bin/fsnotifier
    3414         1      140759 homolo      /snap/rider/489/bin/fsnotifier
      60         3       98184 homolo      /usr/libexec/dde-file-manager -d
      48         3       40197 homolo      /usr/bin/dde-select-dialog-x11
      21         3        1172 homolo      /usr/bin/dde-desktop
      19         4       22666 homolo      /opt/google/chrome/chrome

========ignore=============

INotify instances per user (e.g. limits specified by fs.inotify.max_user_instances): 

INSTANCES    USER
-----------  ------------------
105          homolo

It can be observed that IDEA only creates two instances. I think SourceGit can also consider reducing the number of inotify instances created.

   14907         1        3981 homolo      /snap/intellij-idea-ultimate/524/bin/fsnotifier
       9         1        3848 homolo      /snap/intellij-idea-ultimate/524/bin/idea

Edit: the script is https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers

@love-linger
Copy link
Collaborator
test.mp4

@gadfly3173 I've test with this script https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers

It shows when a repository is opened, 2 instances (needed by 2 FileSystemWatcher) are created.

Could you help me to test that what's the result when multiple instance of IDEA opened with different project? (We need to listen different paths not just one project)

@gadfly3173
Copy link
Contributor
gadfly3173 commented Aug 26, 2024

Could you help me to test that what's the result when multiple instance of IDEA opened with different project? (We need to listen different paths not just one project)您能否帮我测试一下,当使用不同项目打开多个 IDEA 实例时,结果是什么?(我们需要倾听不同的路径,而不仅仅是一个项目)

The previously mentioned IDEA creates only two instances is the situation when four different projects are opened.

   14907         1        3981 homolo      /snap/intellij-idea-ultimate/524/bin/fsnotifier
       9         1        3848 homolo      /snap/intellij-idea-ultimate/524/bin/idea

reduce to 2 project:

   12190         1        3981 homolo      /snap/intellij-idea-ultimate/524/bin/fsnotifier
       9         1        3848 homolo      /snap/intellij-idea-ultimate/524/bin/idea

Rider with SourceGit repository:

    3418         1      160173 homolo      /snap/rider/489/bin/fsnotifier
       9         1      160024 homolo      /snap/rider/489/bin/rider

@aikawayataro
Copy link
Contributor
aikawayataro commented Aug 26, 2024

I think we can use single instance of inotify subsystem on Linux if we use P/Invoke and call inotify_add_watch with single subsystem
This will not reduce the number of watches, though

EDIT:
Or use fanotify which should work better for Source Git case

@gadfly3173
Copy link
Contributor
gadfly3173 commented Aug 26, 2024

Here is Gitkraken with 6 opened repositories. And I think this is a dotnet-runtime issue: dotnet/runtime#62869

    2289         3      168727 homolo      /persistent/data/Gitkraken/gitkraken --type=renderer --crashpad-handler-pid=1686
       6         4      168590 homolo      /persistent/data/Gitkraken/gitkraken
       2         1      168648 homolo      /persistent/data/Gitkraken/gitkraken --type=utility --utility-sub-type=network.m

@love-linger
Copy link
Collaborator

I took a look at how to listen for filesystem changes in Java and found that starting Java 7, it provides 'java.nio.file.WatchService' and it allows register multiple paths. For example:

WatchService watchService = FileSystems.getDefault().newWatchService();
Path path1 = Paths.get("path1");
path1.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
Path path2 = Paths.get("path2");
path2.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);

But in C#, the FileSystemWatcher.Path is a simple string. We need to create multiple instance of FileSystemWatcher to listen multiple paths.

@gadfly3173
Copy link
Contributor
gadfly3173 commented Aug 26, 2024

After observing the behavior of Gitkraken, I found that it would prompt File watching failed to start for this repository.. Then the repository would be opened normally, but the file status would not be automatically updated until I switched to this tab again.

@love-linger love-linger added enhancement New feature or request os-linux This issue only exists on Linux labels Aug 26, 2024
@love-linger
Copy link
Collaborator

I've pushed a commit that deals with this in the way Gitkraken does and these two method still works without a Watcher instance:

public void MarkBranchesDirtyManually()
{
    if (_watcher == null)
    {
        Task.Run(() =>
        {
            RefreshBranches();
            RefreshCommits();
        });

        Task.Run(RefreshWorkingCopyChanges);
        Task.Run(RefreshWorktrees);
    }                
    else
    {
        _watcher.MarkBranchDirtyManually();
    }
}

public void MarkWorkingCopyDirtyManually()
{
    if (_watcher == null)
        Task.Run(RefreshWorkingCopyChanges);
    else
        _watcher.MarkWorkingCopyDirtyManually();
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request os-linux This issue only exists on Linux
Projects
None yet
Development

No branches or pull requests

4 participants
0