we have dbt models pushing schema changes to prod pretty regularly but downstream reports and bi dashboards keep breaking silently. no alerts, just find out when someone complains a week later.

current setup is basic git history + dbt docs but that doesn't catch when a column rename or type change nukes a join in some forgotten looker dashboard. tried adding pre deploy checks with sql fluff but its too static, misses runtime impacts.

our team is small, 4 data engs handling 50+ models across prod/staging. leadership wants zero breakage but manually reviewing every pr is killing us.

anyone got a lightweight way to track this like dbt macros that flag downstream deps, or some schema diff tool that pings slack on breaks open source preferred since budget sucks. What've you seen work at scale without turning into a full ci nightmare?

curious how others avoid this treadmill.

In what circumstance would you use Streams instead of simply cheking for rows where the timestamp (inserted at/updated at) changed?

Ehy are streams useful, can't you do the same wirh simple delta loading?

Data engineers using Cortex Code require a single cost view that separates warehouse compute from AI token credits.

Snowflake now gives dedicated usage history for Cortex Code CLI, Cortex Code in Snowsight, and AI SQL usage, but its not enough!

usage charts are super important but we also need guardrails and financial reporting. We've built them all and much more in SeemoreData but you can also build something sufficient by yourself

this blog explains a good methodology you can build on your own that helps you set the control plane for better cortex code management

https://seemoredata.io/blog/how-to-manage-snowflake-cortex-code-cost-without-slowing-engineers-down/

hope this helps the community!

this is the linkedin post (feel free to like and share) -> linkedin post to like and share :)
as always happy to connect directly -> lets connect directly

Anyone attending this year's summit that would like to discuss code rewriting capabilities? Or any topics you are looking forward to?

We are hiring for a Subject Expert Role in Data Engineer SME (Snowflake, AWS, DBT)

Experience Required : 3-5 years

Remote | Full Time | EST Working hours | Pay: ₹14-18 LPA
About the Opportunity
We are seeking a battle-tested Data Engineer SME with deep expertise in Snowflake, data modelling, SQL/Python, and a flair for creating EdTech video content and teaching.
What We're Looking For

• 3-5 years in data engineering, with hands-on AWS, Snowflake, DBT & AWS Databricks experience
• Expert in data modelling (star schema, dimensional, etc.)
• Advanced SQL querying and optimization
• Python mastery (Pandas, ETL scripting, PySpark)
• Proficient in AWS ecosystem: S3, Glue, Lambda, Redshift, Airflow, dbt for pipelines
• Comfortable shooting EdTech videos (tutorials, demos) and leading live teaching/workshops

Why Join Us?

• Full-time remote— work-life balance
• Blend engineering with teaching: Shape future data pros through content

Looking for Indian talent with immediate to 7 days notice period .Interested folks please DM.

Hi!

I know Snowflake announced the expansion of Cortex Code today and they mentioned the SDK. This is going to be a huge unlock. Does anyone know where/how I can get access to this?

I’m a data scientist and I find it hard to use Claude Code for SQL because of the lack of DB context. so I made yet another database MCP server! only Snowflake support for now.

I had to reconnect with nature after reading native Snowflake MCP setup docs so for my server I’ve made a nice CLI wizard to set up DB connection and install the Claude Code plugin: MCP + skill - you can ask it like `/db write dbt model to rank leads`.

It also has a `plan` tool for complex questions. when you ask a blurry question, it triggers a separate text-to-SQL agent that uses 1. (kinda) RAG for your schema (along with some values) that builds during DB connection (if you agree) 2. subagents to explore your data. 3. planning. This is what Snowflake Cortex is supposed to do, but when I try it, it never finds the right tables.

Database-as-MCP sounds like a security nightmare, but I put a lot of effort into making it safer. I’d appreciate any thoughts on the secure design. by default, CLI asks for select permissions on all schemas, not just information_schema. I’m convinced that it’s impossible to write good SQL without peeking into the data. maybe it's a hot take - share your thoughts!

Everything is free and hosted by me, but rate-limited. In the future, I want to charge for planning calls above the limit. I have a bunch of ideas on how to make a smarter text-to-SQL, so I want to keep this part closed-source. I’ll open-source more though - it’s just deployed as a monolith now.

Hi,
So we had some workflows running with create or replace on tables. The tables schema change from time to time so everytime new data arrives i will just create or replace the tables also it's small data. Nevertheless it's a production workflow. To keep my code/sql as simple as possible and to improve the workflow so people can historic versions of the tables i changed this workflow to:
CREATE OR ALTER TABLE

INSERT AND OVERWRITE

What do you think anything against this solution? Just asking because this CREATE OR ALTER seems more of a use-case for initial bootstrapping of tables as i can see in snowflake doc.

have been working as snowflake developer and admin. where should i focus on to have job for next 10 years. previously worked on Data extract loading too.

will AI tools like clause automize snowflake jobs?

Currently working as a Senior Data Engineer with 5 YoE getting a chance to work for Snowflake. The new role seems less technical and more on the sales side to work on POCs which i have some experience with. I am feeling underpaid at my current role and the new position at snowflake is a huge pay bump. While its something i would still like to work on i feel i will lose my technical skills if i take it up. Would an internal switch be possible after a few years in this case? Also is it safe in this market to be working in the sales division?

Hey folks,

Our team at Estuary works with customers all the time to enable their Snowflake pipelines, and we've seen the gamut of how these ingestion tradeoffs play out, from obliterating credits to underestimating the engineering effort needed to configure and monitor everything across pipelines. Based on this experience, we put together a comparison guide to evaluate each Snowflake ingestion method; how it works, what it costs, and when it makes sense for your use case.

Ultimately, we believe that a well-designed Snowflake data stack applies different ingestion methods for different data streams, and ensures the latency actually matches what the workflow needs. If you're looking for a framework to audit your existing Snowflake pipelines, or you're designing a new pipeline and feeling overwhelmed, we hope this guide helps you out!

We’re reviewing a bunch of vendors in the data reliability software space and trying to narrow things down. Quick thoughts so far:

monte carlo: strong enterprise presence, broad coverage across warehouses and bi tools, very polished but can feel heavy and expensive.

Bigeye: legacy stack support, decent anomaly detection, seems solid for teams not on modern data stack.

elementary: tbh stands out if you're running dbt, they seem to also have Python support. it’s deeply dbt native and seem easier to operationalize. great visibility into data health, freshness, and lineage without overwhelming onboarding. the AI agents look promising, setup is straightforward, and it feels more aligned with analytics engineering workflows instead of forcing a separate platform mindset.

anomalo: heavy focus on ml based anomaly detection, good for automated insights but may require tuning, and is very enterprise heavy.

metaplane: modern UI, focuses on column level monitoring and anomalies, decent balance between automation and control.

soda: flexible and developer friendly, works well if you want more hands on control.

great expectations: more framework than platform, powerful for custom validations but requires engineering effort to scale properly.

for teams that are dbt heavy and want something opinionated but not bloated, elementary feels less intrusive and more practical compared to some of the bigger enterprise suites.

curious what others would prioritize. Full automation and enterprise coverage, or tighter integration and lower operational overhead?