8000 LeRobotDataset v3 by Cadene · Pull Request #969 · huggingface/lerobot · GitHub
[go: up one dir, main page]
Skip to content

LeRobotDataset v3 #969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Cadene wants to merge 72 commits into from
Closed

LeRobotDataset v3 #969

Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
38c1457
Bump CODEBASE_VERSION
Feb 10, 2025
57c9c21
Merge remote-tracking branch 'origin/main' into user/aliberts/2025_02…
Feb 10, 2025
d67ca34
Merge remote-tracking branch 'origin/main' into user/aliberts/2025_02…
Feb 11, 2025
9d6886d
Add frame level task (#693)
Feb 14, 2025
7c2bbee
Validate features during `add_frame` + Add 2D-to-5D + Add string (#720)
Feb 14, 2025
8426c64
Per-episode stats (#521)
aliberts Feb 15, 2025
aed3eb4
Merge remote-tracking branch 'origin/main' into user/aliberts/2025_02…
Feb 15, 2025
624eaf1
Merge remote-tracking branch 'origin/main' into user/aliberts/2025_02…
Feb 17, 2025
02bc4e0
support openx/rlds to lerobot
Tavish9 Feb 18, 2025
fbf2f22
Remove `local_files_only` and use `codebase_version` instead of branc…
aliberts Feb 19, 2025
76436ca
Merge remote-tracking branch 'tavish9_lerobot_openx/main' into user/r…
Cadene Feb 19, 2025
2487228
Use `HF_HOME` env variable (#753)
aliberts Feb 19, 2025
6fe42a7
Add tag
Feb 19, 2025
969ef74
Remove dataset `consolidate` (#752)
aliberts Feb 19, 2025
392a8c3
Improve doc
Feb 20, 2025
64ed525
Fix batch convert
Feb 20, 2025
b520941
Merge remote-tracking branch 'origin/user/aliberts/2025_02_10_dataset…
Cadene Feb 20, 2025
71d1f5e
WIP
Cadene Feb 20, 2025
5fbbaa1
fix No such file or directory error
Cadene Feb 20, 2025
93c80b2
rm brake
Cadene Feb 20, 2025
52fb414
workers
Cadene Feb 21, 2025
15e7a9d
before new launch from scratch
Cadene Feb 21, 2025
eda0b99
new dir
Cadene Feb 21, 2025
689c5ef
optimize shard
Cadene Feb 22, 2025
39ad2d1
let's go
Cadene Feb 22, 2025
ff0029f
aggregate works
Cadene Feb 22, 2025
e2e6f6e
Add auto_downsample_height_width
Cadene Feb 23, 2025
c36d225
Aggregate works
Cadene Feb 23, 2025
3daab2a
Add upload_large_folder
Cadene Feb 23, 2025
3666ac9
WIP UploadDataset
Cadene Mar 1, 2025
7866c1f
Merge remote-tracking branch 'origin/main' into user/rcadene/2025_02_…
Cadene Mar 1, 2025
1a5c1ef
Rename openx to droid + Improve all (not tested)
Cadene Mar 18, 2025
5d184a7
NIT
Cadene Mar 18, 2025
65738f0
Improve slurm droid
Cadene Mar 20, 2025
53ecec5
WIP v21 to v30
Cadene Mar 31, 2025
c1b28f0
Commit before episodes episodes_stats merging
Cadene Apr 9, 2025
34c5d4c
Most unit tests are passing
Cadene Apr 11, 2025
6c4d122
fix joints
Cadene Apr 11, 2025
c2a05a1
Fix (Now loading all frames is possible)
Cadene Apr 14, 2025
6b6a990
most unit tests passing (TODO: convert datasets)
Cadene Apr 16, 2025
eab5543
Merge (No verify)
Cadene Apr 17, 2025
54b5c80
Revert mistake convert_dataset_v20_to_v21.py
Cadene Apr 17, 2025
b0cca75
Progress on aggregate_datasets
Cadene Apr 19, 2025
9c0836c
Remove legacy from datasets/utils.py
Cadene Apr 19, 2025
5a6ea09
Rename tests/test_aggregate_datasets.py -> tests/datasets/test_aggreg…
Cadene Apr 19, 2025
4acf99f
pre-commit run --all-files
Cadene Apr 21, 2025
4375a05
Add push to hub for convert_dataset_v21_to_v30
Cadene Apr 21, 2025
2866d07
small fix ffmpeg encoding
Cadene Apr 21, 2025
5bd9cb1
Merge remote-tracking branch 'origin/main' into user/rcadene/2025_04_…
Cadene Apr 21, 2025
b9b880b
fix get_parquet_file_size_in_mb + DEFAULT_FILE_SIZE_IN_MB=100
Cadene Apr 21, 2025
20b74ae
fix
Cadene Apr 21, 2025
601b5fd
Merge remote-tracking branch 'origin/user/rcadene/2025_04_11_dataset_…
Cadene Apr 22, 2025
367d9bd
Fix unit tests
Cadene Apr 22, 2025
d518b03
Faster self.meta.episodes[...]
Cadene Apr 22, 2025
7c005c2
Merge remote-tracking branch 'origin/user/rcadene/2025_04_11_dataset_…
Cadene Apr 23, 2025
71715c3
fix hf_dataset.set_transform(hf_transform_to_torch)
Cadene Apr 23, 2025
253c649
Fix convert v30 with image datasets
Cadene Apr 24, 2025
e11d2e4
Aggregate: Add concatenation
Cadene May 2, 2025
588bf96
Fix aggregate (num_frames, dataset_from_index, index)
Cadene May 6, 2025
0309a9f
Speedup data loading
Cadene May 6, 2025
1ecaeab
Uploaded droid 1.0.1
Cadene May 8, 2025
e88af0e
Fix visualize_dataset with rerun
Cadene May 8, 2025
e07cb52
In tests: Add use_videos=False by default, Create mp4 file if True, t…
Cadene May 12, 2025
8d36092
WIP aggregate
Cadene May 16, 2025
f07887e
Merge remote-tracking branch 'origin/user/rcadene/2025_04_11_dataset_…
Cadene May 16, 2025
8746276
WIP after Francesco discussion
Cadene May 28, 2025
41132be
WIP after Francesco discussion
Cadene May 28, 2025
6d9374c
add: support for videos generation in datasets
Jun 6, 2025
4a570b5
fix: debug aggregation code
Jun 6, 2025
fbef584
add: tests for aggregation code
Jun 6, 2025
6721683
fix: modularize tests to improve readability
Jun 10, 2025
8415279
Fix Aggregation, Add Tests (#1264)
fracapuano Jul 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix batch convert
  • Loading branch information
Simon Alibert committed Feb 20, 2025
commit 64ed5258e67f00ae42144cbd58aaf8f85f0c345f
3 changes: 2 additions & 1 deletion lerobot/common/datasets/v21/batch_convert_dataset_v20_to_v21.py
View file
Open in desktop
4D60
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@

def batch_convert():
status = {}
LOCAL_DIR.mkdir(parents=True, exist_ok=True)
logfile = LOCAL_DIR / "conversion_log_v21.txt"
for num, repo_id in available_datasets:
for num, repo_id in enumerate(available_datasets):
print(f"\nConverting {repo_id} ({num}/{len(available_datasets)})")
print("---------------------------------------------------------")
try:
Expand Down
Loading
0