r/ETL

Been deep into Base44 lately and honestly enjoying the whole vibe coding/building space.

I’ve been working on platforms involving job boards, CRM-style workflows, automations, dashboards, onboarding systems, AI-assisted forms, candidate management, subscriptions, and internal admin tools — mostly focused on making things practical and usable instead of overcomplicated.

Still learning new things every day, but if anyone here is building something on Base44 (or thinking about it) and wants another brain to bounce ideas with, troubleshoot workflows, improve UI/UX, or help structure systems, feel free to reach out.

Happy to connect with fellow builders and those curious to develop their own getting little help in the space 👌

Has anyone had an issue when trying fetch 20k+ records from BiqQuery to Postgres DB? Everything works fine if I keep it under 10k, using Table Input + SQL, but as soon as I try more records the pipeline fails. Odd Java error message. Ultimately, I am looking to move like 500k records from BQ to Postgres DB.

Most pipelines break because an upstream source changed without warning. I built a platform to catch these issues before they crash your ETL.

What it does:

• Schema Monitoring: Detects renamed columns, dropped fields, or type changes in real-time.
• Uptime Checks: Verifies your APIs and Databases are online before the pipeline runs.
• Instant Alerts: Notifies you the moment drift is detected or any problem with the source.
• Simple Setup: Connect your SQL DBs or REST APIs in under 2 minutes.

Would you use it and what features would make this a "must-have" for your workflow? Thanks!

My company is a distribution company that handle about 35k products across 20/30 different vendors. We do not produce anything. Now the board want a marketplace/e-commerce with a ton of technical details of our spelled products.

These vendors are not so well developed on websites/e-commerce and the best source of information for all of them are PDFs of their products catalogues, we all the technical data resides.

Now, im intrigue in how we could load all of the data into our future system and how we should maintain it (avoid staling information and products revisions)

How do Companies manage to load into a PIM that large amount of data entry? I mean, in our business fields products have a lot of (from 6 to 20) different attributes/dimensions, I cannot image 10 people working months just to add data. There are some smart alternatives to pure human data entry ?

I’m learning data engineering and curious what real-world problems people usually encounter while working with ETL.