Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Summary
The current convert_hf_to_gguf.py script fails to convert HuggingFace models that require trust_remote_code=True, especially those using custom tokenizers or architectures (e.g., TikTokenTokenizer). This results in runtime errors or input prompts that break automation and scripting workflows.
Adding support for trust_remote_code (via CLI flag or built-in toggle) would enable full compatibility with a growing set of modern models.
🔥 Encountered Issues
During conversion of huihui-ai/Moonlight-16B-A3B-Instruct-abliterated, the following blockers were hit:
-
Broken
tokenizer_config.json- Trailing comma →
JSONDecodeError - Manual fix required
- Trailing comma →
-
Script lacks support for
trust_remote_code=True- Models using TikTokenTokenizer trigger:
ValueError: Please pass trust_remote_code=True - Prompt appears even when run in non-interactive mode
- No
--trust-remote-codeCLI flag exists
- Models using TikTokenTokenizer trigger:
-
Script attempts
.vocabon TikTokenTokenizer- Raises:
AttributeError: 'TikTokenTokenizer' object has no attribute 'vocab' - Workaround: use
tokenizer.model.n_vocabor hardcoded value
- Raises:
🧠 Suggested Fixes
(Flexible depending on preference)
- ✅ Add CLI flag:
--trust-remote-code - ✅ Pass the flag to
AutoTokenizer.from_pretrained()andAutoModelForCausalLM.from_pretrained() - ✅ Gracefully handle vocab fallback for tokenizers missing
.vocab
Optional:
- ❓ Auto-enable
trust_remote_codeiftokenizer_classis custom? - ❓ Add an example or mention this behavior in
READMEordocs/gguf.md
🧪 Repro Steps
git clone https://huggingface.co/huihui-ai/Moonlight-16B-A3B-Instruct-abliterated
./convert_hf_to_gguf.py --verbose --outfile moonlight.gguf --outtype bf16 huihui-ai/Moonlight-16B-A3B-Instruct-abliterated
# fails with prompt + crashes if not patched✅ Workaround (What I Did)
- Manually patched:
AutoTokenizer.from_pretrained(..., trust_remote_code=True)
- Replaced
.vocabcall withtokenizer.model.n_vocab - Then conversion succeeded
🙏 Why This Matters
Many newer models are starting to:
- Use custom tokenizers (
TikToken,JinjaTokenizer, etc.) - Require remote code execution
- Break current conversion pipelines
Fixing this once will unlock dozens of models for GGUF and llama.cpp.
Happy to PR this if helpful. Thanks again for all the work — your tooling is incredible.
Motivation
Motivation
Support for trust_remote_code=True is increasingly important as more HuggingFace models rely on custom tokenizers or architectures. Without it, convert_hf_to_gguf.py cannot load or convert these models, blocking compatibility with llama.cpp.
Possible Implementation
No response