For Data owners · Updated Oct 20, 2026
Data format standards
For file-based delivery through S3, GCS or SFTP, exports should follow these standards.
- File format: Parquet is preferred. CSV is an acceptable fallback.
- Compression: Snappy, for Parquet exports.
- Folder structure: Organized by dataset or table, then by date or hour.
- Incremental key: Each table should include a field that identifies new or updated records, such as an updated-at timestamp.
- Backfill first: Scowtt validates a full historical backfill with you before recurring incremental updates start.
