Update QONNX parsing for 1.0 #979

jmitrevs · 2024-03-12T00:12:40Z

Description

This change updates the ONNX parser and adds support for QONNX. It replaces PR #832. It only supports ONNX that has been cleaned by the qonnx package, including converting convolutions to be channels-last and changing Gemm to MatMul and Add.

In QONNX Quant nodes can act on constants as well as the datapath. To make handling this easier, we explicitly put constants in the initial graph. There are also some helper nodes like MatMul and Conv that are introduced to support the explicit constant nodes. After the convert flow, no special ONNX nodes remain in the graph, though.

Generally Quant nodes that have power-of-2 scales and no zero-offset get converted to fixed data types either by setting the types of constants or adding a linear activation that is usually merged into preceding nodes. Non-power-of-2 scales result in ApplyAlpha nodes beings added to scale and unscale, with propagation across some layers. This can be further optimized and has generally been tested less.

This includes the changes from PR #855 with a few updates that will be backported and discussed there. Therefore, this PR needs to wait till that PR is merged, which is why I am making it draft.

Note: for the config_from_onnx_model I made the default granularity be "name" because that enables automatic precision inference, which you need for QONNX. The way that I did that is that I set config['Model']['Precision'] to the default (e.g. fixed<16,6>), but all the precisions filled by config['Model'] are auto. These can be overriden if, for example, the accumulator becomes too wide. In general, though, they are set by the infer_precision.py optimizer.

Binary networks are not yet supported.

More information can be found in this presentation:
https://www.icloud.com/keynote/025yxvgBx8IF2m3Iso6HosqPw#QONNX_Ingestion_0p1

Type of change

New feature (non-breaking change which adds functionality)
A new research paper code implementation

Tests

The pytest, test_qonnx.py, is the main test, building some models from the QONNX model zoo

Checklist

I have read the guidelines for contributing.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
I have installed and run pre-commit on the files I edited or added.
I have added tests that prove my fix is effective or that my feature works.

JanFSchulte · 2024-08-23T18:12:24Z

hls4ml/converters/onnx/core.py

@@ -37,7 +29,7 @@ def parse_gemm_layer(reader, node, inputs_map, input_shapes, graph, config):
    'Softmax',
    'Softsign',
    'Softplus',
-    'Clip',
+    # 'Clip',


Remove commented code?

JanFSchulte · 2024-08-23T18:12:33Z

hls4ml/converters/onnx/core.py

@@ -53,70 +45,89 @@ def parse_gemm_layer(reader, node, inputs_map, input_shapes, graph, config):
    'Softmax': 'Softmax',
    'Softsign': 'Activation',
    'Softplus': 'Activation',
-    'Clip': 'Clip',
+    # 'Clip': 'Clip',


Remove commented code?

JanFSchulte · 2024-08-23T18:17:44Z

hls4ml/converters/onnx/merge.py

-        )
-        output_shape[layer['axis']] = new_dim
-
-    elif layer['class_name'] == 'Add':


Just for my understanding, what is the reason that BiasAdd is no longer supported?

JanFSchulte · 2024-08-23T18:54:43Z

hls4ml/model/optimizer/passes/batchnorm_opt.py

+        """
+
+        if not (len(node.inputs) == 5 and all(node.inputs)):
+            raise ValueError(f'All {len.node.inputs} BatchNormOnnnx inputs need to be defined')


Not sure I understand the error message here, shouldn't it just read "All 5 BatchNormOnnnx inputs need to be defined" since it will also throw the error if all inputs are defined, but their number does not equal 5?

JanFSchulte · 2024-08-23T18:55:34Z

hls4ml/model/optimizer/passes/batchnorm_opt.py

+
+        gamma_node = node.get_input_node(node.inputs[1])
+        if not isinstance(gamma_node, Constant):
+            raise TypeError('Only consant gammas supported')


typo, "constant"

JanFSchulte · 2024-08-23T19:30:47Z

hls4ml/model/optimizer/passes/batchnorm_opt.py

+class FuseConsecutiveBatchNormalization(OptimizerPass):
+    """
+    OptimizerPass to merge consecutive BatchNormalization layers,
+    only if the earlier one does not have quantization specified


The code below seems to match also in the case when the current node does not have quantization specified, not just the earlier one. Does this description need to be updated?

JanFSchulte · 2024-08-23T19:31:22Z

hls4ml/model/optimizer/passes/batchnorm_opt.py

+        # # Not sure why this part is needed
+        # node_map = node.get_output_use_map()
+        # if len(node_map[node.outputs[0]]) > 1:
+        #     return False


remove commented code?

JanFSchulte · 2024-08-23T19:35:58Z

hls4ml/model/optimizer/passes/bn_fuse.py

+    Note:  Consider restricting this to ApplyAlpha.  Batch Normalization quantization seems to be ignored.
+
+    Note:  This optimizer may not be safe if weights are updateable. May need to turn off.
+    """


This seems to be the doc string from FuseConsecutiveBatchNormalization above and should probably be replaced with one appropriate to this function.

JanFSchulte · 2024-08-23T19:36:32Z

hls4ml/model/optimizer/passes/bn_fuse.py

+        # # Not sure why this part is needed
+        # node_map = node.get_output_use_map()
+        # if len(node_map[node.outputs[0]]) > 1:
+        #     return False


remove commented code.

JanFSchulte · 2024-08-23T19:41:44Z

hls4ml/model/optimizer/passes/conv_to_convxd.py

+        # The ConvxD nodes expect the weight data to be in a different format, not (M, k1.., C)
+        if node.attributes['n_dim'] == 1:
+            newtype = Conv1D
+            attributes['weight_data'] = np.transpose(weight_data, (1, 2, 0))


Out of curiosity, why do we need to transpose the weights here when the model is supposed to have been processed by qonnx-to-channels-last already?

JanFSchulte · 2024-08-27T13:13:39Z

hls4ml/model/optimizer/passes/merge_const.py

+
+
+class MergeToApplyAlpha(OptimizerPass):
+    """Convert Add, Sub, Mul, or Div Merges with consant to ApplyAlpha"""


typo "consant" -> "constant"

JanFSchulte

I left a bunch of very minor comments, otherwise this seems ready for merge from my point of view.

jmitrevs added 30 commits July 11, 2023 11:00

Add needed layer types for QONNX

0765ec4

add qonnx pytest

ff788ea

first migration of onnx parsing

cda7208

change tuples to lists

af47a0d

snapshot of adding qonnx optimizers

8f8cc0b

snapshot that runs qonnx test, but gets incorrect results

5cea82d

add quant node quantizer

d5394d4

fix broadcasting when going from Merge to ApplyAlpha

9817ed3

update linear merging

e494f43

update automatic setting of accumulators (QONNX-only for now)

ffddb5e

update qonnx tests

57c89fb

remove batch dimension from flatten in Keras

233905a

Merge branch 'main' into qonnx_0p8

aafe0ca

fix optimizer that fuses consecutive batch norms

6f11955

Merge branch 'main' into qonnx_0p8

0becc04

Merge remote-tracking branch 'vloncar/auto_precision' into qonnx-1p0

1f433fe

snapshot of work

76be67b

snapshot before removing redundant precision attributes

4d52975

snapshot

cf5c9a1

bug fixes from attempting to run

81f3e53

fix some bugs from qonnx pytest

9a74e46

fix assertion of not matching the number of inputs when replacing node

60a74bb

Merge remote-tracking branch 'vloncar/auto_precision' into qonnx-1p0

a032a5d

update some precisions inference

88a8d35

Merge remote-tracking branch 'upstream/main' into qonnx-1p0

0379db2

extract bitwidth from size 1 array in quant node

10a3c50

update automatic onnx configuration

ab8d67b

standardize on merge operators

0a863ad

snapshot of current work

bfe6a3f

Fix bug in FuseBatchNormalization

25849ef

jmitrevs added 4 commits July 17, 2024 11:04

Merge branch 'main' into qonnx-1p0

6189953

Following what seems to be done in the main branch

2909d15

update infer_precision based on changes in keras-config-auto

c9693da

loosen batchnorm merging restrictions, fix ternary handling

aaaa2fc

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Jul 19, 2024

remove some backends from slow qonnx test

a2b88f4

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Jul 19, 2024

Merge remote-tracking branch 'upstream/main' into qonnx-1p0

169d9e5

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Aug 21, 2024

move multi_dense to conv above inferming precision types

ef02b4f

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Aug 21, 2024

fix the default reuse factor

c3ffa7b

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Aug 21, 2024

JanFSchulte reviewed Aug 23, 2024

View reviewed changes

JanFSchulte reviewed Aug 27, 2024

View reviewed changes

JanFSchulte approved these changes Aug 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update QONNX parsing for 1.0 #979

Update QONNX parsing for 1.0 #979



		class MergeToApplyAlpha(OptimizerPass):
		"""Convert Add, Sub, Mul, or Div Merges with consant to ApplyAlpha"""

Update QONNX parsing for 1.0 #979

Are you sure you want to change the base?

Update QONNX parsing for 1.0 #979

Conversation

Description

Type of change

Tests

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment