r/databricks

Hey, i am deciding between DQX and Deequ for data quality on Databricks, or even to use them both, i think Deequ is amazing because of AnomalyCheck which let us compare batch to batch and make the data flow consistently over time which is very under appreciated, while DQX is amazing at the row level detection. How did u design your data quality on Databricks?

I was thinking using DQX for in-transit Data Quality checks for hard fails, while Deequ for AnomalyCheck for observartion/dashboards/notifications.

Mid-sized org, around 300 people. Running multiple Databricks workspaces across AWS and Azure, hundreds of Spark jobs daily.

Debugging slow jobs, skew, small files, memory spills, shuffle issues takes too much manual time. Spark UI and Databricks monitoring cover the basics but at this scale someone always has to notice something is wrong before investigation starts. That lag is becoming a real problem.
Started looking into Spark agents to automate detection and surface bottlenecks without someone manually going through every stage. The volume justifies it but not sure what's production-ready vs still experimental.

What are people using for Spark agents at this scale? Does it hold up across multiple workspaces or does it fall apart once jobs get into the hundreds daily?

Try it out in your Query Editor or on your Databricks SQL warehouse today!

Hi all,

Can you suggest any GitHub projects or YouTube videos that cover end-end databricks projects which are complex and interesting to gain experience and exposure same as prod environment?

I wrote a practical deep dive on Delta Lake that explains what actually happens behind the scenes—not just the basic theory.

Most tutorials stop at “Delta supports ACID and Time Travel,” but I wanted to understand how it really works.

In this blog, I covered:

• _delta_log and transaction logs
• Why Delta never deletes old files immediately
• Checkpoints and snapshot mechanism
• Data skipping and how Z-Ordering improves performance
• History, Restore, and Time Travel
• Merge, Update, Delete operations
• Convert Parquet to Delta
• Optimize and the small file problem
• Real PySpark examples for every concept

I tried to explain everything in a simple, practical way with real examples instead of documentation-style theory.

https://medium.com/@wnccpdfvz/why-delta-lake-is-faster-than-traditional-data-lakes-5c865f67b66b

Hi Everyone, Codebeamer is an Application Life Cycle Management tool . I have been assigned to get all the data out of this application to Databricks i.e ingesting all data in delta tables creating data model fit for analytics purpose. I have given only codeBeamer APIs to work with. Anyone has worked on similar projects, let me know your insights and suggestions and best practices. Thanks in advance.