8000 JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length by ochafik · Pull Request #6555 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length #6555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Apr 12, 2024
Merged

JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length #6555

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2148f24
json: rename python schema converter to make import easier
Apr 8, 2024
f771a8f
server: skip null json_schema / grammar fields
Apr 8, 2024
159b883
json: deps management for primitive rules (+ allow null values)
Apr 8, 2024
a59e943
json: optimize repetitions for minItems/maxItems and regexps: `a{,3}`…
Apr 8, 2024
07163fb
8000 grammars: add troubleshooting section to readme
Apr 8, 2024
dcf5d32
json: cap length of numbers to 15 digits before/after decimal point
Apr 8, 2024
181f984
json: unify all repetition code (w/ or w/o sep)
Apr 8, 2024
de4e60e
json: support string minLength/maxLength
Apr 8, 2024
6c885dc
server+json: update server/README w/ result_format
Apr 8, 2024
3c81e94
nits
Apr 8, 2024
67a5184
json: fix type error w/ python 3.8
Apr 8, 2024
9c33ee9
json: fix server/README (json_schema in /completion vs. result_format…
Apr 9, 2024
ed13d47
json: simplify DOT `{"type": "string", "pattern": "^.$"}`
Apr 12, 2024
958bdda
Merge remote-tracking branch 'origin/master' into json-faster-repetit…
Apr 12, 2024
64e3059
json: remove recursion in opt_repetitions (avoids Python stack overflow)
Apr 12, 2024
ba90d5b
json: rm dead code
Apr 12, 2024
dfd4eb3
json: rm useless assert & ggml.h import
Apr 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
json: remove recursion in opt_repetitions (avoids Python stack overflow)
  • Loading branch information
ochafik committed Apr 12, 2024
commit 64e305901efa17f93579fd161ffb11171c1d7816
18 changes: 18 additions & 0 deletions common/json-schema-to-grammar.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "ggml.h"

using json = nlohmann::ordered_json;

Expand Down Expand Up @@ -36,6 +37,23 @@ static std::string build_repetition(const std::string & item_rule, int min_items
}

std::function<std::string(int, bool)> opt_repetitions = [&](int up_to_n, bool prefix_with_sep) -> std::string {
auto content = prefix_with_sep && !separator_rule.empty() ? separator_rule + " " + item_rule : item_rule;

if (up_to_n == 0) {
return "";
} else if (up_to_n == 1) {
return "(" + content + ")?";
} else if (!separator_rule.empty() && !prefix_with_sep) {
return "(" + content + " " + opt_repetitions(up_to_n - 1, true) + ")?";
} else {
std::string res = repeat("(" + content + " ", up_to_n);
// strip trailing space
GGML_ASSERT(!res.empty());
res = res.substr(0, res.length() - 1);
res += repeat(")?", up_to_n);
return res;
}

if (up_to_n == 0) {
return "";
}
Expand Down
18 changes: 13 additions & 5 deletions examples/json_schema_to_grammar.py
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I'm on board with the rename to use underscores -- while there are a few other files with underscores (such as pydantic_models_to_grammar.py), most seem to use hyphens (pydantic-models-to-grammar-examples.py, etc), and it seems like the old filename is possibly better?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally I wanted all filenames in the repo to use hyphens. But later I found out that Python does not work well when there are hyphens in the filenames (e.g. I think you cannot include a Python file that has hyphens). So I think it's better to eventually rename all Python files to use underscores in their filenames

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh I did all this as a prerequisite for #6389, in which i need to import the converter from Python. I also found out llama-cpp-python inlines that file in their codebase, since it's hard / not trivial to import (short of using importlib, which feels dirty).

Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,21 @@ def _build_repetition(item_rule, min_items, max_items, separator_rule=None, item
result = (f' {separator_rule} ' if separator_rule else ' ').join([item_rule] * min_items)

def opt_repetitions(up_to_n, prefix_with_sep=False):
'''
- n=4, no sep: '(a (a (a (a)?)?)?)?'
- n=4, sep=',', prefix: '("," a ("," a ("," a ("," a)?)?)?)?'
- n=4, sep=',', no prefix: '(a ("," a ("," a ("," a)?)?)?)?'
'''

content = f'{separator_rule} {item_rule}' if prefix_with_sep and separator_rule else item_rule
if up_to_n == 0:
return ''

res = separator_rule + ' ' + item_rule if separator_rule and prefix_with_sep else item_rule
if up_to_n > 1:
res += ' ' + opt_repetitions(up_to_n - 1, prefix_with_sep=True)
return f'({res})?'
elif up_to_n == 1:
return f'({content})?'
elif separator_rule and not prefix_with_sep:
return f'({content} {opt_repetitions(up_to_n - 1, prefix_with_sep=True)})?'
else:
return (f'({content} ' * up_to_n).rstrip() + (')?' * up_to_n)

if min_items > 0 and max_items != min_items:
result += ' '
Expand Down
Loading
0