Data Science & Analytics
Databricks Streaming Guardian
Guard production Databricks data pipelines — Delta Lake, Liquid Clustering, Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a new column, a DLT refresh drops data, or before running a destructive op on a streamed-from table. Trigger with "ConcurrentAppendException", "ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full refresh".
- Publisher / source
- jeremylongshore/tons-of-skills-marketplace (found on GitHub top repositories)
- Upstream record
- jeremylongshore/tons-of-skills-marketplace
- Version
- Version 1
- Published
- Publication review
- Approved for publication
- Content fingerprint
4d4fcf78677bb549672e2b49214f8d1f836e84b95f2618f2839176b86d472b47