8000 Shallow Clone Support by lya001 · Pull Request #6396 · libgit2/libgit2 · GitHub
[go: up one dir, main page]

Skip to content

Shallow Clone Support #6396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 81 commits into from
May 9, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
0668384
repo: basic graft API
tiennou Aug 2, 2018
919501a
repo: load grafts on open
tiennou Aug 2, 2018
3c17f22
repo: load shallow roots
tiennou Oct 10, 2019
a8b1d51
repo: graft shallow roots on open
tiennou Aug 2, 2018
a35cc40
commit: apply grafts when parsing
tiennou Aug 2, 2018
d54c008
tests: graft commits
tiennou Aug 2, 2018
22f201b
grafts: make the structure self-contained and opaque
pks-t Oct 3, 2019
14a309a
repository: convert grafts parsing to use parse context
pks-t Oct 3, 2019
05e286f
grafts: move parsing into grafts module
pks-t May 12, 2020
a11026e
repository: reuse grafts for shallow roots
pks-t Oct 24, 2019
70867f7
repository: disentangle shallow and normal grafts
pks-t Oct 3, 2019
fd2398b
grafts: move refresh logic into grafts code
pks-t Oct 3, 2019
a4803c3
grafts: fix memory leak if replacing pre-existing graft
pks-t Oct 10, 2019
79af067
repository: do not expose grafting mechanism
pks-t Oct 24, 2019
e07aa9c
Merge branch 'main' into pr/pks-t/5254
lya001 Jun 24, 2022
13bd14d
add feature flag for shallow clone support
Jun 27, 2022
7a93625
add test for shallow feature flag
Jun 27, 2022
89494f6
add shallow.h
Jun 27, 2022
562246b
move declaration of feature flag to graft.h/graft.c
Jun 27, 2022
6bab22f
move feature flag tests to tests/grafts/shallow.c
Jun 27, 2022
ad56355
use shallow feature flag in shallow clone support source code
Jun 27, 2022
c4cd9a5
correct naming of feature flag
Jun 27, 2022
5918975
disable shallow clone support by default
Jun 28, 2022
397753f
enable shallow clone support in tests when necessary
Jun 28, 2022
9d1507d
correct use of feature flag
Jun 28, 2022
06eacb9
fix graft assertion
Jun 28, 2022
70a332a
disable shallow clone feature flag in test cleanup
Jun 28, 2022
afa79ca
Merge branch 'pr/tiennou/4747' into transportPR
Jul 4, 2022
10e2573
attempt to build
Jul 5, 2022
3e64f15
rewrite shallow_root
Jul 7, 2022
c652f3d
enable cloning of shallow repo
Jul 19, 2022
f19ffc8
add test for shallow repo depth 1
Jul 19, 2022
7f46bfa
unset GIT_RAND_GETENTROPY to avoid linux GLIBC issues (#3)
lya001 Jul 26, 2022
52ba17f
Merge branch 'pr/pks-t/5254' into shallow-clone-local
Jul 26, 2022
83f71b1
fix build errors
Jul 26, 2022
cfc2ae6
eliminate build warnings
Jul 27, 2022
c01b784
improve error handling
Jul 28, 2022
14d2a60
fix load_grafts
Jul 28, 2022
62cc77a
refactor commit parent assignment with graft
Jul 28, 2022
a544a91
rename function assign_parents_from_graft
Jul 28, 2022
3d7a609
Merge pull request #4 from lya001/shallow-clone-local
lya001 Jul 29, 2022
09acf69
Merge branch 'mw_dev' into shallow-clone-local
lya001 Jul 29, 2022
a491917
Merge pull request #5 from mathworks/shallow-clone-local
lya001 Jul 29, 2022
68bbcef
Merge branch 'transportPR' into shallow-clone-network
Jul 29, 2022
73d25f0
remove build errors
Jul 29, 2022
598ec30
eliminate build warnings
Jul 29, 2022
179aac7
fix clone::shallow test behaviour
Jul 29, 2022
e7294e8
fix memory leaks about packets
Aug 1, 2022
2d33fe7
refactor git_fetch_option.depth and usage
Aug 2, 2022
4536477
fix memory leak
Aug 2, 2022
e93d081
attempt to fix nego.shallowarray memory leak
Aug 2, 2022
da04d3f
fix grafts and shallowarray memory leaks
Aug 2, 2022
8ef492f
fix build warning
Aug 2, 2022
829555a
edit tests for shallow clones
Aug 2, 2022
09b3d33
fix memory leak
Aug 9, 2022
df5eb32
support fetch unshallow option on shallow repos
Aug 9, 2022
49e641b
remove unused api
Aug 11, 2022
8b521f0
document unshallow behaviour in fetch.c
Aug 11, 2022
7c2b1f4
Merge pull request #6 from lya001/shallow-clone-network
lya001 Aug 11, 2022
a3bfd28
Use GIT_OID_SHA1_HEXSIZE
lya001 Aug 31, 2022
01cb90b
restore GIT_RAND_GETENTROPY
lya001 Aug 31, 2022
b20f013
restore getentropy
lya001 Aug 31, 2022
f1f9b45
fix test failures
lya001 Aug 31, 2022
6c46b58
include oid.h in grafts.c
lya001 Aug 31, 2022
4cf19ad
refactor smart_pkt
lya001 Aug 31, 2022
d0eba8a
fix shallowarray test
lya001 Aug 31, 2022
89c1b01
fix free error
lya001 Aug 31, 2022
34de5c8
fix seg faults
lya001 Sep 5, 2022
7122fcd
fix depth initialisation
lya001 Sep 5, 2022
47f36a9
fix error handling
lya001 Sep 5, 2022
d23a790
remove unused statements
lya001 Sep 5, 2022
4f2f91a
fix shallow array search
lya001 Sep 6, 2022
a9793ac
refactor grafts tests
lya001 Sep 6, 2022
570ef74
Merge commit 'd066d0d95c43e97df6624292f3f527f9372ca8fe'
lrm29 Mar 10, 2023
1cc2979
Fix merge error
lrm29 Mar 10, 2023
2da3e8c
Remove stray comma
lrm29 Mar 10, 2023
d935773
Remove unused git_transport_flags_t
lrm29 Mar 10, 2023
79ed94e
Apply suggestions from code review
lrm29 Mar 10, 2023
e3bf6db
Merge branch 'shallow-clone-network' of https://github.com/mathworks/…
lrm29 Mar 10, 2023
5b71133
Update src/libgit2/fetch.c
lrm29 Mar 10, 2023
e288f87
Remove unnecessary include.
lrm29 Mar 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
repository: reuse grafts for shallow roots
The shallow roots are in fact another user of the grafting mechanism,
and in essence they do use the same file format for grafted commits.
Thus, instead of hand-coding the parsing logic a second time, we can
just reuse the `git_grafts` structure for shallow commits, as well.
  • Loading branch information
pks-t committed Jun 27, 2020
commit a11026ecbb3575b02638ae060b20e77227e2291c
3 changes: 2 additions & 1 deletion include/git2/repository.h
Original file line number Diff line number Diff line change
Expand Up @@ -878,7 +878,8 @@ GIT_EXTERN(int) git_repository_is_shallow(git_repository *repo);
/**
* Determine the shallow roots of the repository
*
* This oidarray is owned by the library. Do not free it.
* The resulting OID array needs to be free'd by calling
* `git_oidarray_free`.
*
* @param out An array of shallow oids.
* @param repo The repository
Expand Down
21 changes: 21 additions & 0 deletions src/grafts.c
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

#include "grafts.h"

#include "oidarray.h"
#include "parse.h"

struct git_grafts {
Expand Down Expand Up @@ -155,6 +156,26 @@ int git_grafts_get(git_commit_graft **out, git_grafts *grafts, const git_oid *oi
return 0;
}

int git_grafts_get_oids(git_oidarray *out, git_grafts *grafts)
{
git_array_oid_t oids = GIT_ARRAY_INIT;
const git_oid *oid;
size_t i = 0;
int error;

assert(out && grafts);

while ((error = git_oidmap_iterate(NULL, grafts->commits, &i, &oid)) == 0) {
git_oid *cpy = git_array_alloc(oids);
GIT_ERROR_CHECK_ALLOC(cpy);
git_oid_cpy(cpy, oid);
}

git_oidarray__from_array(out, &oids);

return 0;
}

size_t git_grafts_size(git_grafts *grafts)
{
return git_oidmap_size(grafts->commits);
Expand Down
1 change: 1 addition & 0 deletions src/grafts.h
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ int git_grafts_parse(git_grafts *grafts, const char *content, size_t contentlen)
int git_grafts_add(git_grafts *grafts, const git_oid *oid, git_array_oid_t parents);
int git_grafts_remove(git_grafts *grafts, const git_oid *oid);
int git_grafts_get(git_commit_graft **out, git_grafts *grafts, const git_oid *oid);
int git_grafts_get_oids(git_oidarray *out, git_grafts *grafts);
size_t git_grafts_size(git_grafts *grafts);

#endif
103 changes: 31 additions & 72 deletions src/repository.c
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ int git_repository__cleanup(git_repository *repo)
git_cache_clear(&repo->objects);
git_attr_cache_flush(repo);
git_grafts_free(repo->grafts);
git_array_clear(repo->shallow_oids);
git_grafts_free(repo->shallow_grafts);

set_config(repo, NULL);
set_index(repo, NULL);
Expand Down Expand Up @@ -613,22 +613,22 @@ static int load_grafts(git_repository *repo)

static int load_shallow(git_repository *repo)
{
int error = 0;
git_array_oid_t roots = GIT_ARRAY_INIT;
git_array_oid_t parents = GIT_ARRAY_INIT;
git_oidarray roots;
int error;
size_t i;
git_oid *graft_oid;

/* Graft shallow roots */
if ((error = git_repository__shallow_roots(&roots, repo)) < 0) {
return error;
}
if ((error = git_repository_shallow_roots(&roots, repo)) < 0)
goto out;

git_array_foreach(roots, i, g 67ED raft_oid) {
if ((error = git_grafts_add(repo->grafts, graft_oid, parents)) < 0)
return error;
}
return 0;
for (i = 0; i < roots.count; i++)
if ((error = git_grafts_add(repo->grafts, &roots.ids[i], parents)) < 0)
goto out;

out:
git_oidarray_free(&roots);
return error;
}

int git_repository_open_bare(
Expand Down Expand Up @@ -2960,79 +2960,38 @@ int git_repository_state_cleanup(git_repository *repo)
return git_repository__cleanup_files(repo, state_files, ARRAY_SIZE(state_files));
}

int git_repository__shallow_roots(git_array_oid_t *out, git_repository *repo)
int git_repository_shallow_roots(git_oidarray *out, git_repository *repo)
{
git_buf path = GIT_BUF_INIT;
git_buf contents = GIT_BUF_INIT;
int error, updated, line_num = 1;
char *line;
char *buffer;

assert(out && repo);

if ((error = git_buf_joinpath(&path, repo->gitdir, "shallow")) < 0)
return error;

error = git_futils_readbuffer_updated(&contents, git_buf_cstr(&path), &repo->shallow_checksum, &updated);
git_buf_dispose(&path);
git_buf path = GIT_BUF_INIT, contents = GIT_BUF_INIT;
int error, updated = 0;

if (error < 0 && error != GIT_ENOTFOUND)
return error;

/* cancel out GIT_ENOTFOUND */
git_error_clear();
error = 0;

if (!updated) {
*out = repo->shallow_oids;
goto cleanup;
}
assert(out && repo);

git_array_clear(repo->shallow_oids);
memset(out, 0, sizeof(*out));

buffer = contents.ptr;
while ((line = git__strsep(&buffer, "\n")) != NULL) {
git_oid *oid = git_array_alloc(repo->shallow_oids);
if (!repo->shallow_grafts && (error = git_grafts_new(&repo->shallow_grafts)) < 0)
goto error;

error = git_oid_fromstr(oid, line);
if (error < 0) {
git_error_set(GIT_ERROR_REPOSITORY, "Invalid OID at line %d", line_num);
git_array_clear(repo->shallow_oids);
error = -1;
goto cleanup;
}
++line_num;
if ((error = git_buf_joinpath(&path, repo->gitdir, "shallow")) < 0 ||
(error = git_futils_readbuffer_updated(&contents, git_buf_cstr(&path),
&repo->shallow_checksum, &updated)) < 0) {
if (error == GIT_ENOTFOUND)
error = 0;
goto error;
}

if (*buffer) {
git_error_set(GIT_ERROR_REPOSITORY, "No EOL at line %d", line_num);
git_array_clear(repo->shallow_oids);
error = -1;
goto cleanup;
}
if (updated && (error = git_grafts_parse(repo->shallow_grafts, contents.ptr, contents.size)) < 0)
goto error;

*out = repo->shallow_oids;
if ((error = git_grafts_get_oids(out, repo->shallow_grafts)) < 0)
goto error;

cleanup:
error:
git_buf_dispose(&path);
git_buf_dispose(&contents);

return error;
}

int git_repository_shallow_roots(git_oidarray *out, git_repository *repo)
{
int ret;
git_array_oid_t array = GIT_ARRAY_INIT;

assert(out);

ret = git_repository__shallow_roots(&array, repo);

git_oidarray__from_array(out, &array);

return ret;
}

int git_repository_is_shallow(git_repository *repo)
{
git_buf path = GIT_BUF_INIT;
Expand Down
4 changes: 1 addition & 3 deletions src/repository.h
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,8 @@ struct git_repository {
git_grafts *grafts;
git_oid graft_checksum;

git_grafts *shallow_grafts;
git_oid shallow_checksum;
git_array_oid_t shallow_oids;

git_atomic attr_session_key;

Expand Down Expand Up @@ -263,6 +263,4 @@ extern size_t git_repository__reserved_names_posix_len;
bool git_repository__reserved_names(
git_buf **out, size_t *outlen, git_repository *repo, bool include_ntfs);

int git_repository__shallow_roots(git_array_oid_t *out, git_repository *repo);

#endif
21 changes: 12 additions & 9 deletions tests/grafts/shallow.c
Original file line number Diff line number Diff line change
Expand Up @@ -42,38 +42,41 @@ void test_grafts_shallow__clears_errors(void)

void test_grafts_shallow__shallow_oids(void)
{
git_oidarray oids, oids2;
git_oidarray oids;
g_repo = cl_git_sandbox_init("shallow.git");

cl_git_pass(git_repository_shallow_roots(&oids, g_repo));
cl_assert_equal_i(1, oids.count);
cl_assert_equal_oid(&g_shallow_oid, &oids.ids[0]);

cl_git_pass(git_repository_shallow_roots(&oids2, g_repo));
cl_assert_equal_p(oids.ids, oids2.ids);
git_oidarray_free(&oids);
}

void test_grafts_shallow__cache_clearing(void)
{
git_oidarray oids, oids2;
git_oidarray oids;
git_oid tmp_oid;

git_oid_fromstr(&tmp_oid, "0000000000000000000000000000000000000000");
cl_git_pass(git_oid_fromstr(&tmp_oid, "0000000000000000000000000000000000000000"));
g_repo = cl_git_sandbox_init("shallow.git");

cl_git_pass(git_repository_shallow_roots(&oids, g_repo));
cl_assert_equal_i(1, oids.count);
cl_assert_equal_oid(&g_shallow_oid, &oids.ids[0]);
git_oidarray_free(&oids);

cl_git_mkfile("shallow.git/shallow",
"be3563ae3f795b2b4353bcce3a527ad0a4f7f644\n"
"0000000000000000000000000000000000000000\n"
);

cl_git_pass(git_repository_shallow_roots(&oids2, g_repo));
cl_assert_equal_i(2, oids2.count);
cl_assert_equal_oid(&g_shallow_oid, &oids2.ids[0]);
cl_assert_equal_oid(&tmp_oid, &oids2.ids[1]);
cl_git_pass(git_repository_shallow_roots(&oids, g_repo));
cl_assert_equal_i(2, oids.count);
cl_assert((git_oid_equal(&g_shallow_oid, &oids.ids[0]) &&
git_oid_equal(&tmp_oid, &oids.ids[1])) ||
(git_oid_equal(&g_shallow_oid, &oids.ids[1]) &&
git_oid_equal(&tmp_oid, &oids.ids[0])));
git_oidarray_free(&oids);

cl_git_pass(p_unlink("shallow.git/shallow"));
cl_git_pass(git_repository_shallow_roots(&oids, g_repo));
Expand Down
0