[DISCUSS] Columnar data protocol: Arrow or implement a new one? #33
Description
A columnar data protocol is required in the following cases:
- Reading metadata from snapshot metadata (JSON, Avro)
- Writing metadata to snapshot metadata (JSON, Avro)
- Reading data from data files (Parquet, Avro, ORC)
- Writing data to data files (Parquet, Avro, ORC)
I'd like to use Arrow Array (backed by nanoarrow) for this purpose because
- It is the de-facto standard and accepted by many popular engines.
- nanoarrow library is lightweight and easy to use.
WDYT?