[falcon] Fix Falcon for rw-1b model #2887
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cebtenzzre
reviewed
Aug 29, 2023
|
I'm taking a wild guess here and thinking that After updating gguf to account for an extra post-attention layer in the tensor map for falcon, I'm successfully able to convert the model. Let me try running it now |
|
Here is the output I see after quantizing + running the model: |
|
I have no idea why, but it looks like the tokenizer is missing some tokens. This model also seems to use alibi, so it will probably require some changes to the computation graph as well. |
|
This is really bizarre. In the conversion script, I wonder if taking the stated vocab size would at least fix this issue. I can look into the alibi stuff next |
|
That fixed the vocab issue. Now: I think this is probably due to how we expected to have reshaped the qkv tensor which we skip for this model let me play around with this |
|
Alright, latest update. I actually got the model running and outputting something (nonsense for now): This model has a bunch of extra |
convert-falcon-hf-to-gguf.py for rw models|
It might be more difficult to support these models than I initially imagined. Here is the reference implmentation: Here is how As @slaren mentioned, these models also use |
|
@ggerganov is there generally some kind of mapping of the operations in the python implementation to what we have available in ggml? Do you see anything missing? |
|
The needed operators (like alibi) should already be available in It's mostly a matter of correctly building the graphs depending on the config parameters. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See #2868
config.json:config.jsonvocab size does not match contents oftokenizer.json- going with the stated size and padding with the missing tokens)ffm-normlayer.error loading model: create_tensor: tensor 'blk.0.attn_qkv.weight' has wrong shape; expected 2048, 2176, got 2048, 6144