server: --offline mode #13804
Merged
server: --offline mode #13804
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also support OFFLINE=1 ./tests.sh for server tests
Co-Authored-By: ochafik <ochafik@google.com>
Co-Authored-By: ochafik <ochafik@google.com>
Co-Authored-By: ochafik <ochafik@google.com>
ngxson
reviewed
May 26, 2025
common/arg.cpp
Outdated
Comment on lines
289
to
292
| if (!file_exists || !offline) | ||
| { | ||
| bool should_download = !file_exists; // by default, we should download if the file does not exist | ||
|
|
There was a problem hiding this comment.
I think we don't need to put this code block inside a long if..else like this. Instead, there is a more simple way:
if (file_exists && offline) {
LOG_INF("%s: using cached file (offline mode): %s\n", __func__, path.c_str());
return true; // skip verification/downloading
}
// keep the rest of the code as-is
common_load_model_from_url_headers headers;
bool head_request_ok = false;
bool should_download = !file_exists; // by default, we should download if the file does not exist
// get ETag to see if the remote file has changed
{There was a problem hiding this comment.
Or even more fail-safe:
if (offline) {
if (file_exists) {
// if offline mode is enabled and the file exists, we can use it
LOG_INF("%s: using cached file (offline mode): %s\n", __func__, path.c_str());
return true; // skip verification/downloading
} else {
LOG_ERR("%s: required file is not available in cache (offline mode): %s\n", __func__, path.c_str());
return false;
}
}Co-Authored-By: ochafik <ochafik@google.com>
ngxson
reviewed
May 26, 2025
| @@ -298,6 +272,9 @@ static bool common_download_file_single(const std::string & url, const std::stri | |||
| // if we cannot open the metadata file, we assume that the downloaded file is not valid (etag and last-modified are left empty, so we will download it again) | |||
| } else { | |||
| LOG_INF("%s: no previous model file found %s\n", __func__, path.c_str()); | |||
| if (offline) { | |||
| return false; | |||
There was a problem hiding this comment.
This code branch will be trigger when file_exists == false && offline == true, so technically one of the condition inside if (offline) below now become dead code.
Instead, you can move the if (offline) to above of if (file_exists). So if file exist, we completely skip reading metadata
There was a problem hiding this comment.
ah ofc, rearranged to skip, thanks!
Co-Authored-By: ochafik <ochafik@google.com>
ngxson
approved these changes
May 26, 2025
tools/server/tests/utils.py
Outdated
Comment on lines
124
to
125
| if "OFFLINE" in os.environ: | ||
| server_args.append("--offline") |
There was a problem hiding this comment.
maybe we don't need this, because you can do LLAMA_OFFLINE=1 (not sure if the env is forwarded to child process though)
Co-Authored-By: ochafik <ochafik@google.com>
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds --offline flag (env:
LLAMA_OFFLINE) to use locally cached manifests & models if available, or fail@ngxson I know you told me you've incubated something similar (here?) but since I was stuck in a plane with ~zero internet all day yesterday, I ended up hacking this together. Feel free to merge later or ignore in favour of your approach.