Test exported models in Python

@larryliu0820

We currently have tests to export models (in Python of course), then test them using C++ runners:

Lines 251 to 312 in 71025df

    
           if [[ "${BACKEND}" == "portable" ]]; then 
        
             echo "Testing ${MODEL_NAME} with portable kernels..." 
        
             test_model 
        
           elif [[ "${BACKEND}" == *"qnn"* ]]; then 
        
             echo "Testing ${MODEL_NAME} with qnn..." 
        
             test_model_with_qnn 
        
             if [[ $? -eq 0 ]]; then 
        
               prepare_artifacts_upload 
        
             fi 
        
           elif [[ "${BACKEND}" == *"coreml"* ]]; then 
        
             echo "Testing ${MODEL_NAME} with coreml..." 
        
             should_test_coreml=false 
        
             if [[ "${BACKEND}" == *"test"* ]]; then 
        
               should_test_coreml=true 
        
             fi 
        
             test_model_with_coreml "${should_test_coreml}" 
        
             if [[ $? -eq 0 ]]; then 
        
               prepare_artifacts_upload 
        
             fi 
        
           elif [[ "${BACKEND}" == *"mps"* ]]; then 
        
             echo "Testing ${MODEL_NAME} with mps..." 
        
             test_model_with_mps 
        
             if [[ $? -eq 0 ]]; then 
        
               prepare_artifacts_upload 
        
             fi 
        
           elif [[ "${BACKEND}" == *"xnnpack"* ]]; then 
        
             echo "Testing ${MODEL_NAME} with xnnpack..." 
        
             WITH_QUANTIZATION=true 
        
             WITH_DELEGATION=true 
        
             if [[ "$MODEL_NAME" == "mobilebert" ]]; then 
        
               # TODO(T197452682) 
        
               WITH_QUANTIZATION=false 
        
             fi 
        
             test_model_with_xnnpack "${WITH_QUANTIZATION}" "${WITH_DELEGATION}" 
        
             if [[ $? -eq 0 ]]; then 
        
               prepare_artifacts_upload 
        
             fi 
        
           else 
        
             set +e 
        
             if [[ "${BACKEND}" == *"quantization"* ]]; then 
        
               echo "::group::Testing ${MODEL_NAME} with XNNPACK quantization only..." 
        
               test_model_with_xnnpack true false || Q_ERROR="error" 
        
               echo "::endgroup::" 
        
             fi 
        
             if [[ "${BACKEND}" == *"delegation"* ]]; then 
        
               echo "::group::Testing ${MODEL_NAME} with XNNPACK delegation only..." 
        
               test_model_with_xnnpack false true || D_ERROR="error" 
        
               echo "::endgroup::" 
        
             fi 
        
             if [[ "${BACKEND}" == *"quantization"* ]] && [[ "${BACKEND}" == *"delegation"* ]]; then 
        
               echo "::group::Testing ${MODEL_NAME} with XNNPACK quantization and delegation..." 
        
               test_model_with_xnnpack true true || Q_D_ERROR="error" 
        
               echo "::endgroup::" 
        
             fi 
        
             set -e 
        
             if [[ -n "${Q_ERROR:-}" ]] || [[ -n "${D_ERROR:-}" ]] || [[ -n "${Q_D_ERROR:-}" ]]; then 
        
               echo "Portable q8 ${Q_ERROR:-ok}," "Delegation fp32 ${D_ERROR:-ok}," "Delegation q8 ${Q_D_ERROR:-ok}" 
        
               exit 1 
        
             else 
        
               prepare_artifacts_upload 
        
             fi 
        
           fi

We currently do not have tests to test the models within the Python runtime — a pretty huge gap. Naturally, when a developer exports a model, they are likely to try to test it out using Python. So it's crucial we ensure this flow always works.

Create the tests under test/end2end, where we export a model — then test the exported .pte file
Ensure the tests are running in CI (include them in pytest.ini)

TODO

Test tests are up for grab! If you've completed a test, please link it to this issue so I can mark it as complete.

[P0] macOS
- XNNPACK
- CoreML
- MPS
[P0] Linux
- XNNPACK
- CoreML
- QNN
[P1] Windows
- XNNPACK
- QNN

cc @larryliu0820

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TODO

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if [[ "${BACKEND}" == "portable" ]]; then
	echo "Testing ${MODEL_NAME} with portable kernels..."
	test_model
	elif [[ "${BACKEND}" == "qnn" ]]; then
	echo "Testing ${MODEL_NAME} with qnn..."
	test_model_with_qnn
	if [[ $? -eq 0 ]]; then
	prepare_artifacts_upload
	fi
	elif [[ "${BACKEND}" == "coreml" ]]; then
	echo "Testing ${MODEL_NAME} with coreml..."
	should_test_coreml=false
	if [[ "${BACKEND}" == "test" ]]; then
	should_test_coreml=true
	fi
	test_model_with_coreml "${should_test_coreml}"
	if [[ $? -eq 0 ]]; then
	prepare_artifacts_upload
	fi
	elif [[ "${BACKEND}" == "mps" ]]; then
	echo "Testing ${MODEL_NAME} with mps..."
	test_model_with_mps
	if [[ $? -eq 0 ]]; then
	prepare_artifacts_upload
	fi
	elif [[ "${BACKEND}" == "xnnpack" ]]; then
	echo "Testing ${MODEL_NAME} with xnnpack..."
	WITH_QUANTIZATION=true
	WITH_DELEGATION=true
	if [[ "$MODEL_NAME" == "mobilebert" ]]; then
	# TODO(T197452682)
	WITH_QUANTIZATION=false
	fi
	test_model_with_xnnpack "${WITH_QUANTIZATION}" "${WITH_DELEGATION}"
	if [[ $? -eq 0 ]]; then
	prepare_artifacts_upload
	fi
	else
	set +e
	if [[ "${BACKEND}" == "quantization" ]]; then
	echo "::group::Testing ${MODEL_NAME} with XNNPACK quantization only..."
	test_model_with_xnnpack true false \|\| Q_ERROR="error"
	echo "::endgroup::"
	fi
	if [[ "${BACKEND}" == "delegation" ]]; then
	echo "::group::Testing ${MODEL_NAME} with XNNPACK delegation only..."
	test_model_with_xnnpack false true \|\| D_ERROR="error"
	echo "::endgroup::"
	fi
	if [[ "${BACKEND}" == "quantization" ]] && [[ "${BACKEND}" == "delegation" ]]; then
	echo "::group::Testing ${MODEL_NAME} with XNNPACK quantization and delegation..."
	test_model_with_xnnpack true true \|\| Q_D_ERROR="error"
	echo "::endgroup::"
	fi
	set -e
	if [[ -n "${Q_ERROR:-}" ]] \|\| [[ -n "${D_ERROR:-}" ]] \|\| [[ -n "${Q_D_ERROR:-}" ]]; then
	echo "Portable q8 ${Q_ERROR:-ok}," "Delegation fp32 ${D_ERROR:-ok}," "Delegation q8 ${Q_D_ERROR:-ok}"
	exit 1
	else
	prepare_artifacts_upload
	fi
	fi

Description

TODO

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions