8000 server : replace behave with pytest (#10416) · arthw/llama.cpp@7e88ce5 · GitHub
[go: up one dir, main page]
Skip to content

Commit 7e88ce5

Browse files
ngxsonarthw
ngxson
authored and
arthw
committed
server : replace behave with pytest (ggml-org#10416)
* server : replace behave with pytest * fix test on windows * misc * add more tests * more tests * styling * log less, fix embd test * added all sequential tests * fix coding style * fix save slot test * add parallel completion test * fix parallel test * remove feature files * update test docs * no cache_prompt for some tests * add test_cache_vs_nocache_prompt
1 parent 8da1d9c commit 7e88ce5

34 files changed

+1317
-2497
lines changed

‎.devops/nix/python-scripts.nix

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ let
3434

3535
# server tests
3636
openai
37-
behave
37+
pytest
3838
prometheus-client
3939
];
4040
in

‎.github/workflows/server.yml

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -122,14 +122,14 @@ jobs:
122122
id: server_integration_tests
123123
run: |
124124
cd examples/server/tests
125-
PORT=8888 ./tests.sh
125+
./tests.sh
126126
127127
- name: Slow tests
128128
id: server_integration_tests_slow
129129
if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }}
130130
run: |
131131
cd examples/server/tests
132-
PORT=8888 ./tests.sh --stop --no-skipped --no-capture --tags slow
132+
SLOW_TESTS=1 ./tests.sh
133133
134134
135135
server-windows:
@@ -180,11 +180,12 @@ jobs:
180180
run: |
181181
cd examples/server/tests
182182
$env:PYTHONIOENCODING = ":replace"
183-
behave.exe --summary --stop --no-capture --exclude 'issues|wrong_usages|passkey' --tags llama.cpp
183+
pytest -v -x
184184
185185
- name: Slow tests
186186
id: server_integration_tests_slow
187187
if: ${{ (github.event.schedule || github.event.inputs.slow_tests == 'true') && matrix.build_type == 'Release' }}
188188
run: |
189189
cd examples/server/tests
190-
behave.exe --stop --no-skipped --no-capture --tags slow
190+
$env:SLOW_TESTS = "1"
191+
pytest -v -x

‎examples/server/tests/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
.venv
2+
tmp

‎examples/server/tests/README.md

Lines changed: 8 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,9 @@
11
# Server tests
22

3-
Python based server tests scenario using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development)
4-
and [behave](https://behave.readthedocs.io/en/latest/):
5-
6-
* [issues.feature](./features/issues.feature) Pending issues scenario
7-
* [parallel.feature](./features/parallel.feature) Scenario involving multi slots and concurrent requests
8-
* [security.feature](./features/security.feature) Security, CORS and API Key
9-
* [server.feature](./features/server.feature) Server base scenario: completion, embedding, tokenization, etc...
3+
Python based server tests scenario using [pytest](https://docs.pytest.org/en/stable/).
104

115
Tests target GitHub workflows job runners with 4 vCPU.
126

13-
Requests are
14-
using [aiohttp](https://docs.aiohttp.org/en/stable/client_reference.html), [asyncio](https://docs.python.org/fr/3/library/asyncio.html)
15-
based http client.
16-
177
Note: If the host architecture inference speed is faster than GitHub runners one, parallel scenario may randomly fail.
188
To mitigate it, you can increase values in `n_predict`, `kv_size`.
199

@@ -39,26 +29,19 @@ It's possible to override some scenario steps values with environment variables:
3929
|--------------------------|------------------------------------------------------------------------------------------------|
4030
| `PORT` | `context.server_port` to set the listening port of the server during scenario, default: `8080` |
4131
| `LLAMA_SERVER_BIN_PATH` | to change the server binary path, default: `../../../build/bin/llama-server` |
42-
| `DEBUG` | "ON" to enable steps and server verbose mode `--verbose` |
32+
| `DEBUG` | to enable steps and server verbose mode `--verbose` |
4333
| `N_GPU_LAYERS` | number of model layers to offload to VRAM `-ngl --n-gpu-layers` |
4434

45-
### Run @bug, @wip or @wrong_usage annotated scenario
46-
47-
Feature or Scenario must be annotated with `@llama.cpp` to be included in the default scope.
48-
49-
- `@bug` annotation aims to link a scenario with a GitHub issue.
50-
- `@wrong_usage` are meant to show user issue that are actually an expected behavior
51-
- `@wip` to focus on a scenario working in progress
52-
- `@slow` heavy test, disabled by default
53-
54-
To run a scenario annotated with `@bug`, start:
35+
To run slow tests:
5536

5637
```shell
57-
DEBUG=ON ./tests.sh --no-skipped --tags bug --stop
38+
SLOW_TESTS=1 ./tests.sh
5839
```
5940

60-
After changing logic in `steps.py`, ensure that `@bug` and `@wrong_usage` scenario are updated.
41+
To run with stdout/stderr display in real time (verbose output, but useful for debugging):
6142

6243
```shell
63-
./tests.sh --no-skipped --tags bug,wrong_usage || echo "should failed but compile"
44+
DEBUG=1 ./tests.sh -s -v -x
6445
```
46+
47+
To see all available arguments, please refer to [pytest documentation](https://docs.pytest.org/en/stable/how-to/usage.html)

‎examples/server/tests/conftest.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import pytest
2+
from utils import *
3+
4+
5+
# ref: https://stackoverflow.com/questions/22627659/run-code-before-and-after-each-test-in-py-test
6+
@pytest.fixture(autouse=True)
7+
def stop_server_after_each_test():
8+
A689 # do nothing before each test
9+
yield
10+
# stop all servers after each test
11+
instances = set(
12+
server_instances
13+
) # copy the set to prevent 'Set changed size during iteration'
14+
for server in instances:
15+
server.stop()

‎examples/server/tests/features/ctx_shift.feature

Lines changed: 0 additions & 66 deletions
This file was deleted.

‎examples/server/tests/features/embeddings.feature

Lines changed: 0 additions & 113 deletions
This file was deleted.

‎examples/server/tests/features/environment.py

Lines changed: 0 additions & 71 deletions
This file was deleted.

0 commit comments

Comments
 (0)
0