Data Engineering · FinTech
2026SAT Data Pipeline
XML-to-Parquet lakehouse & DuckDB analytics engine for Mexican SAT CFDI 4.0 invoices
A high-performance Python fiscal data lakehouse that extracts raw CFDI 4.0 XMLs, batches records via Apache Arrow, generates Hive-partitioned Parquet datasets, screens against SAT EFOS blacklists in O(1) time, and runs sub-second analytical SQL via DuckDB.
Key Architecture & Impact
- Streaming XML ETL: Memory-constant chunked parsing via lxml with Anexo 20 namespace handling
- Columnar storage: Zero-copy Apache Arrow batching into Hive-partitioned Parquet (year/month)
- Embedded OLAP: Sub-second analytical SQL queries over hundreds of thousands of invoices via DuckDB