convert.py : handle special tokens · Issue #2820 · ggml-org/llama.cpp · GitHub

8000 convert.py : handle special tokens · Issue #2820 · ggml-org/llama.cpp · GitHub

convert.py : handle special tokens #2820

Closed

Closed

convert.py : handle special tokens#2820

Labels

enhancementgood first issue

opened

on Aug 26, 2023

Here we need to start handling special tokens in convert.py:

https://github.com/ggerganov/llama.cpp/blob/e4324cbd4d367fb9ddaaef98c0929acebbf22954/convert.py#L790-L800

An example is shown in convert-llama-7b-pth-to-gguf.py:

https://github.com/ggerganov/llama.cpp/blob/e4324cbd4d367fb9ddaaef98c0929acebbf22954/convert-llama-7b-pth-to-gguf.py#L186-L239

Metadata

Assignees

No one assigned

Labels

enhancementgood first issue

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

0