Apache Parquet Columnar Storage
De-facto standard hybrid columnar storage format for modern data lakes (Iceberg, Delta, DuckDB, Spark), placing Thrift Compact Protocol schema and statistics in the file trailer.
Starts AND ends with 4-byte `PAR1` magic. The 4 bytes immediately preceding the trailing `PAR1` (`[size-8 .. size-4]`) store `uint32_le footer_length` for single-seek metadata reads without scanning row groups.
Drop any .parquet file (even 100 GB+) to inspect locally
Reads tiny File.slice() header/footer pages in a sandboxed Web Worker · 0 Bytes Uploaded
.parquet Magic Bytes & Header Offset Map
Leading 4-byte Apache Parquet magic (mirrored at EOF-4)
| Byte Offset | Length | Data Type | Field Name | Engineering Description |
|---|---|---|---|---|
| 0x00 - 0x03 | 4 B | char[4] | header_magic | ASCII 'PAR1' (or 'PARE' for encrypted footer) |
| 0x04 .. EOF-8-L | var | chunks | row_groups | Compressed column chunks & data/dictionary pages |
| EOF-8-L .. EOF-8 | L B | thrift_compact | file_metadata | Thrift Compact encoded FileMetaData (schema, stats, codecs) |
| EOF-8 .. EOF-5 | 4 B | uint32_le | footer_length | Byte length L of FileMetaData |
| EOF-4 .. EOF-1 | 4 B | char[4] | footer_magic | Trailing ASCII 'PAR1' confirming uncorrupted write |
.parquet Security & Polyglot Risk Matrix
Always check BOTH head[0:4] == 'PAR1' and tail[-4:] == 'PAR1' to reject interrupted multipart S3 uploads before enqueueing ETL jobs.
Read .parquet Header & Metadata Without Heavy Dependencies
import os, struct, sys
with open(sys.argv[1], "rb") as f:
head = f.read(4)
f.seek(-8, os.SEEK_END)
footer_len, tail = struct.unpack("<I4s", f.read(8))
assert head == b"PAR1" and tail == b"PAR1"
print(f"Valid Parquet: Thrift footer={footer_len} bytes")Deterministic .parquet Sample Files & Edge Cases
minimal-snappy.parquet
Real Parquet file written by pyarrow with Snappy compression: 3 rows × user_id INT64, latency_ms DOUBLE, service UTF8.
curl -fsSL https://fileformat.co/fixtures/v1/parquet/minimal-snappy.parquet -o minimal-snappy.parquet \
&& echo "8b84a4eb04fdad847b748c47f6132f1a61f5b1de219178e86fea93aca5ac90f5 minimal-snappy.parquet" | shasum -a 256 -cevents-zstd-decimal.parquet
pyarrow-written 250-row Parquet with ZSTD codec, DECIMAL(12,2) and TIMESTAMP_MICROS logical types.
curl -fsSL https://fileformat.co/fixtures/v1/parquet/events-zstd-decimal.parquet -o events-zstd-decimal.parquet \
&& echo "1b9a434953963317ee622358bccb62cce53d74422b0b8331f063f237ff8b123b events-zstd-decimal.parquet" | shasum -a 256 -ctruncated-footer.parquet
Real Parquet bytes truncated before the footer length and trailing PAR1 magic — pyarrow must raise.
curl -fsSL https://fileformat.co/fixtures/v1/parquet/truncated-footer.parquet -o truncated-footer.parquet \
&& echo "042078532f4fd53caf059416a0dd17b81c8620cf769ad5c88dafc7a1d8f30491 truncated-footer.parquet" | shasum -a 256 -cRelated Specifications & Alternatives
Zero-copy in-memory columnar format serialized to disk with 8-byte aligned `ARROW1\x00\x00` framing and FlatBuffers schema/record-batch descriptors.
Fast, language-agnostic DataFrame persistence format for Python (Pandas/Polars) and R, standardized since v2 directly atop the Apache Arrow IPC file specification.
Stripe-based columnar storage format originating in the Hive/Trino/Presto ecosystem with Protobuf-encoded Footer and a 1-byte PostScript length at the very last byte of the file.
Single-file vectorized OLAP database format storing compressed columnar row-groups in 4 KB / 256 KB block-managed pages.