Home / Data Engineering & Columnar

Data Engineering & Columnar

High-throughput analytical columnar formats, embedded databases, and row/stream serialization specs used in modern lakehouses.

.arrow

Tier 1 Deep Parser little

Apache Arrow IPC Random-Access File

Zero-copy in-memory columnar format serialized to disk with 8-byte aligned `ARROW1\x00\x00` framing and FlatBuffers schema/record-batch descriptors.

Magic (0x0): 41 52 52 4F 57 31 00 00 (ARROW1..)

Inspect & View Spec → 3 QA Samples →

.avro

Tier 1 Deep Parser little

Apache Avro Object Container File

Row-oriented binary data serialization container widely used in Kafka/Confluent event streaming and Iceberg manifest lists, embedding full JSON writer schema in the file header.

Magic (0x0): 4F 62 6A 01 (Obj.)

Inspect & View Spec → 3 QA Samples →

.csv

Tier 1 Deep Parser text

Comma-Separated Values (RFC 4180)

Plain-text tabular interchange format notorious for real-world parser edge cases: optional UTF-8 BOMs (`EF BB BF`), multiline quoted cells, doubled quote escapes (`""`), and formula injection.

Inspect & View Spec → 3 QA Samples →

.feather

Tier 1 Deep Parser little

Feather v2 Columnar DataFrame

Fast, language-agnostic DataFrame persistence format for Python (Pandas/Polars) and R, standardized since v2 directly atop the Apache Arrow IPC file specification.

Magic (0x0): 41 52 52 4F 57 31 00 00 (ARROW1..)

Inspect & View Spec → 3 QA Samples →

.parquet

Tier 1 Deep Parser little

Apache Parquet Columnar Storage

De-facto standard hybrid columnar storage format for modern data lakes (Iceberg, Delta, DuckDB, Spark), placing Thrift Compact Protocol schema and statistics in the file trailer.

Magic (0x0): 50 41 52 31 (PAR1)

Inspect & View Spec → 3 QA Samples →

.sqlite

Tier 1 Deep Parser big

SQLite 3 Database File

Ubiquitous single-file ACID relational B-tree database with a big-endian 100-byte header at offset 0x00 and full SQL DDL catalog stored in Root Page 1 (`sqlite_schema`).

Magic (0x0): 53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 (SQLite format 3.)

Inspect & View Spec → 3 QA Samples →

.duckdb

Tier 2 Hex Atlas little

DuckDB Analytical Database File

Single-file vectorized OLAP database format storing compressed columnar row-groups in 4 KB / 256 KB block-managed pages.

Magic (0x8): 44 55 43 4B (DUCK)

Inspect & View Spec → 3 QA Samples →

.orc

Tier 2 Hex Atlas little

Apache ORC (Optimized Row Columnar)

Stripe-based columnar storage format originating in the Hive/Trino/Presto ecosystem with Protobuf-encoded Footer and a 1-byte PostScript length at the very last byte of the file.

Magic (0x0): 4F 52 43 (ORC)

Inspect & View Spec → 3 QA Samples →