A simple LLM blender, to easily combine the best available open-source large language models into a super one.
-
Clone this repository and navigate into it:
git clone https://github.com/pandego/blendit.git cd blendit -
Create and activate your Python Environment:
conda env create -f envrionment.yml conda activate blendit
-
Initiate your project with
poetry:poetry install
That's it for the Python environment!
In order to blend models you will have to setup a config.yaml file with the instructions for the blend. You can find some examples in src/config/.
-
(Optional) If you need to download a set of models to get started with the example above, you can run the following command:
bash download_models.sh
-
Run the follow command to launch the blend, and let it shake 🍸
sh run_merge.sh
If you are curious, the most important part of the script is the following:
mergekit-yaml src/config/example_task_arithmetic.yaml models/merged-models/my-first-blend \ --verbose \ --allow-crimes \ # experimental: allows blends of different model architectures --no-cuda \ # if you have a big Nvidia GPU, then set this to --cuda --copy-tokenizer \ # copies the tokenizer from the base model --out-shard-size 1B \ # splits model so it can be load in smaller RAMs (default: parse_kmb("5B")) --lazy-unpickle \ # experimental: lowers memory usage --write-model-card \ --low-cpu-memory \ --random-seed 123 \ --trust-remote-code \ --safe-serialization
You can see all the options by running
mergekit-yaml --help.
In order submit your blend to OpenLLM Leaderboard, you need to upload it to HuggingFace first.
- You can do that by running the following script and follow the instructions:
sh upload_model_to_hf.sh
#TODOs
[ ] Add way to test model locally
[x] Add way to submit it to HF for evaluation
[ ] Wrap it all up in a nice Streamlit App
[ ] Dockerize it, deploy it!