Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is it possible to apply these optimizations to Arrow?

Arrow's struct is called StructArray. Fields of StructArray have a StructType.

StructType has .bit_width and .byte_width attrs in Python and probably the other implementations too: https://arrow.apache.org/docs/python/generated/pyarrow.Struc...

Arrow supports bitfields with BooleanArray, and enums with categoricals but

"BUG: Categorical columns using the PyArrow backend requires 4x more memory" (open) https://github.com/pandas-dev/pandas/issues/58062 :

> On disk Parquet appears to store the category data as logical type String which is compressed with snappy and encoded

Arrow Flight RPC handles nested structs with enums over the wire somehow too FWIU



Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: