Normalize VAE-decoded image datasets for training AI artifact detection models with formal verification guarantees and RSR (Rhodium Standard Repository) compliance.
The VAE Dataset Normalizer prepares VAE-decoded image datasets for machine learning research. It provides cryptographically verified data splits, formal proofs of correctness, and contrastive learning models to detect VAE artifacts.
This project is fully compliant with the Rhodium Standard Repository (RSR) specification, ensuring reproducibility, security, and accountability.
-
SHAKE256 (d=256) cryptographic checksums for data integrity (FIPS 202)
-
Train/Test/Val/Calibration splits (70/15/10/5) - both random and stratified
-
Dublin Core metadata via CUE configuration language
-
Nickel schema for flexible configuration
-
Isabelle/HOL proofs for split property verification
-
Julia/Flux.jl utilities for training integration
-
Contrastive learning model for VAE artifact detection
-
Diff-based compression reducing storage by ~50%
# Full normalization with checksums
vae-normalizer normalize -d /path/to/dataset -o /path/to/output
# Fast mode (skip checksums)
vae-normalizer normalize -d /path/to/dataset -o /path/to/output --skip-checksums
# Custom seed and strata
vae-normalizer normalize -d /path/to/dataset -o /path/to/output --seed 12345 --strata 8# Basic verification
vae-normalizer verify -o /path/to/output
# With checksum verification
vae-normalizer verify -o /path/to/output --checksums -d /path/to/dataset# Text format
vae-normalizer stats -o /path/to/output
# JSON format
vae-normalizer stats -o /path/to/output --format jsonReduce storage by ~50% by storing VAE images as diffs from originals:
# Compress dataset (creates Original/ + Diff/ structure)
vae-normalizer compress -d /path/to/dataset -o /path/to/compressed
# Decompress (reconstruct full VAE/ directory)
vae-normalizer decompress -d /path/to/compressed -o /path/to/reconstructed-vae
# Reconstruct single image
vae-normalizer reconstruct -o /path/to/original.png -d /path/to/diff.png -o output.pngThe diff encoding uses: diff = VAE - Original + 128 (offset for signed values).
Reconstruction: VAE = Original + Diff - 128.
output/
├── splits/
│ ├── random_train.txt
│ ├── random_test.txt
│ ├── random_val.txt
│ ├── random_calibration.txt
│ ├── stratified_train.txt
│ ├── stratified_test.txt
│ ├── stratified_val.txt
│ └── stratified_calibration.txt
├── manifest.csv
└── metadata.cueExpected input:
dataset/
├── Original/
│ ├── image001.png
│ ├── image002.png
│ └── ...
└── VAE/
├── image001.png
├── image002.png
└── ...Image files are paired by filename stem (e.g., Original/foo.png pairs with VAE/foo.png).
After running compress, the dataset uses diff encoding:
compressed/
├── Original/
│ ├── image001.png
│ └── ...
└── Diff/
├── image001.png # Encodes (VAE - Original + 128)
└── ...Diff images compress well (~50% smaller) because most pixels cluster near 128 (gray) when VAE and original are similar.
The Isabelle/HOL theory VAEDataset_Splits.thy proves:
-
Disjointness: No overlap between train/test/val/calibration sets
-
Exhaustiveness: Every image appears in exactly one split
-
Ratio correctness: Split sizes within 1% of targets
-
Bijection: 1:1 correspondence between original and VAE images
To verify proofs:
isabelle build -d . -b VAEDataset_Splitsinclude("julia_utils.jl")
using .VAEDatasetUtils
# Load training split
dataset = VAEDetectorDataset(
"output/splits/random_train.txt",
"output/manifest.csv",
"/path/to/dataset"
)
# Create data loader
loader = DataLoader(dataset, batch_size=32, shuffle=true)
# Train with Flux.jl
for (x, y) in loader
# x: batch of images
# y: labels (0=original, 1=VAE)
endinclude("julia_utils.jl")
using .VAEDatasetUtils
# Load compressed dataset (VAE images reconstructed on-the-fly)
dataset = CompressedVAEDataset(
"output/splits/random_train.txt",
"/path/to/compressed/Original",
"/path/to/compressed/Diff"
)
# Same API as standard dataset
loader = DataLoader(dataset, batch_size=32, shuffle=true)
for (x, y) in loader
# VAE images are reconstructed automatically from diffs
endMetadata validated against Dublin Core via metadata_schema.cue:
dublin_core: {
title: "VAEDecodedImages-SDXL"
creator: "Joshua Jewell"
type: "Dataset"
format: "image/png"
}This repository follows the Rhodium Standard Repository (RSR) specification:
-
README.adoc, LICENSE.txt, SECURITY.md, CODE_OF_CONDUCT.md
-
CONTRIBUTING.adoc with Tri-Perimeter Contribution Framework (TPCF)
-
GOVERNANCE.adoc, MAINTAINERS.md, FUNDING.yml
-
CHANGELOG.md, ROADMAP.md, REVERSIBILITY.md
-
Memory-safe Rust implementation (no unsafe code)
-
FIPS 202 compliant cryptography (SHAKE256)
-
Chainguard Wolfi container base
-
SPDX headers on all source files
-
Supply chain security (pinned dependencies)
-
Nix flakes for reproducibility
-
Podman containers (never Docker)
-
Nickel/CUE configuration
-
Justfile task runner
This software is licensed under the Palimpsest-MPL-1.0 License.
The Palimpsest-MPL extends MPL-2.0 with provisions for:
-
Ethical use and emotional lineage preservation
-
Quantum-safe cryptographic provenance
-
Attribution and transparent modification history
-
Community benefit and cooperative economics
See palimpsest-license for full details.
See CONTRIBUTING.adoc for contribution guidelines.
See SECURITY.md for security policy and vulnerability reporting.
