Ricardo Teixeira

If your dashboards take minutes to load, you’ve already lost. Slow dashboards cost credibility. Snowflake in Australia gives BI speed, Databricks gives ML power. Which one is your team optimised for?

1

If your dashboards take minutes to load, you’ve already lost. Slow dashboards cost credibility. Snowflake in Australia gives BI speed, Databricks gives ML power. Which one is your team optimised for?

1

You can’t scale chaos. But you can scale with standards. Without clear modelling standards, one UK retailer’s data team struggled to deliver at speed. Each squad was working differently.  Deployments were inconsistent.  And as they scaled, so did the confusion, errors, and delivery delays. We implemented a standardised, automated DataOps process across all squads – creating a single source of truth for models, mappings, and lineage. The results?  Fast, efficiency, and consistent data delivery at scale.  #cicd #data #innovation

7

1 Comment

Most pre-product market fit startups don't need a big cloud datawarehouse and bloated data stack. We've been thinking about this a lot. You need centralised data. You need a single source of truth. But you probably don't need an enterprise grade stack yet. The advice used to be to go for an end-to-end tool (Amplitude, Mixpanel, etc) but the industry is moving fast: tools like dltHub and MotherDuck make for very lean data setups that still bring the governance and engineering rigour you need to scale. That's exciting! New post from Thomas in't Veld on when lean makes sense—and when to graduate to the full stack. 👉 https://lnkd.in/e69k8zXj

3

Schema changes without warning. Multiple versions across departments. Silent encoding failures that corrupt data for months. The problem isn't spreadsheets themselves—it's treating manually curated reference data like it's disposable when it's actually mission-critical infrastructure. New article on the technical realities of reference data sprawl and why integration failures are symptoms of a deeper governance problem. https://lnkd.in/ga_tA9U7 #DataEngineering #DataQuality #DataOps #ReferenceData #ETL

2

A typical day for a modern data platform engineer isn’t just writing SQL. It includes: 🧱 Managing infra-as-code for scalable pipelines 🔍 Building observability into data quality and lineage 🚀 Optimising query performance in tools like Snowflake or BigQuery 🤝 Sitting with product teams to map out schema changes and use cases These roles blend backend engineering, platform thinking, and business empathy. We help top engineers find teams that actually value this hybrid skill set. Know someone ready for their next data platform challenge? https://lnkd.in/eeMidcgn #TechYardRecruitment #DataEngineering #SnowflakeJobs

7

Aaron Phethean spoke with Nik Walker, Head of Data Engineering at Co-op, about why data engineers burn out, not from work, but from pointless work. We’ve just published the article that breaks it all down: How to cut cloud costs without starting from scratch When not to use real-time data A simple model to stop overloading your team 📝 Read the article → https://lnkd.in/eshjAH5q 🎧 Catch the full episode → https://lnkd.in/ehiMWP9m 📺 Or watch it on YouTube → https://lnkd.in/edy-_gZy

5

In regulated industries, speed isn’t a luxury, it’s a pressure. But when your data is too sensitive for the cloud, your headcount is frozen, and your teams are buried in fragmented systems… speed feels impossible. For your data scientists, clarity doesn’t come easy. And that should concern you. Before they can advise with confidence, they’re buried in chaos: - Fragmented systems, siloed reports, and data sources that were never designed to connect. - Messy variables, missing values, and hours of manual stitching just to make the data usable. - Complex models that need to be simplified into “business language” before anyone will act on them. By the time a report is produced that everyone accepts, the moment to act has already passed. That’s the cost of operating without transparency and speed. AiMod was built for that reality. - Generate insights in minutes, not weeks - giving your data teams the time to consolidate, reason, and advise with clarity - Produce transparent, regulator-ready reports with full audit trails - Deploy entirely on-premise, no data ever leaves your environment No migration required. No additional headcount. No trade-off between control and performance. Just faster answers. Stronger reasoning. Cleaner decisions. When your AI works like this -  scaling isn’t a risk. It’s a focus. Let’s talk about what secure, enterprise-grade AI looks like when it respects your perimeter. #ExplainableAI #ModelTransparency #Infomina  #AiMod #PredictiveAnalytics 

16

Green DAGs, Broken Metrics? That’s Not Reliability, That’s Risk. When conversion rates dropped 18%, the Airflow pipeline showed green. Hive was fine. Logs looked normal. The culprit? A small, silent change in a JSON schema that broke downstream logic, no errors, no alerts. This is the challenge of today’s data pipelines. “Success” isn’t a green DAG. It’s trusted outputs. Here’s how we built a resilient system: ● Enforced schema contracts with Great Expectations, versioned in Git ● Introduced hard validation gates to stop bad data early ● Integrated Slack and PagerDuty for instant alerting ● Mapped field-level lineage using Marquez to trace impact in real-time The result? ✔ 40% drop in query latency ✔ 97% of null-field errors caught within 2 minutes ✔ Debugging time cut from hours to minutes ✔ Backfills became rare ✔ Analysts trusted dashboards again Want your data to tell the truth? Validate it. Version it. Trace it. What’s your team doing to stop silent data drift before it spreads?

3

Most data platform advice falls into two camps: build everything from scratch, or wait until you absolutely need to. Both approaches waste time and create technical debt. We've just released a free eBook that cuts through the noise: Scaling Data Teams walks you through the actual decisions teams face, from 1 to 20 people. What makes it different: 👷 Real case studies from Erewhon, StashAway, Vanta, and OTTO showing what worked (and what didn't) ⚙️ Stage-specific guidance so you're not over-engineering at 2 people or firefighting at 15 🧠 Mental models for the tradeoffs between moving fast today and building for tomorrow Solo practitioners learn how to balance ad-hoc requests with reusable systems. Growing teams get frameworks for earning trust through testing and workflows. Enterprise teams learn how to transform their platform into infrastructure that the entire organization depends on. Download it today! Link in the comments

18

2 Comments

Most data teams are building AI applications with the same headcount, same timelines, and more pressure.   Cortex Code changes the development math. Native to Snowflake, context-aware, and production-ready, not just a code suggester.   Here is a breakdown what it actually enables and where it moves the needle. Abhishek Bhardwaj Ram Dwivedi Rijun Chokshi Venkat Ramasubban Sunil K.   https://lnkd.in/gX8dNeyE   #Snowflake #CortexAI #DataEngineering #EnterpriseAI #AIAgents #DataPlatform #GrowthArc #CortexCode #CoCo

13

Join us in a conversation with our Senior Director - Analytics and Data Science, as he deep dives into the revolutionary Snowflake Data Cloud to understand how it complements Data Mesh - a data architectural approach. Explore how Data Mesh reduces the time and effort it takes with the traditional approach of storing data at a central approach. Watch the full podcast here - https://hubs.li/Q03JXgV90 . . . . . #DataAnalytics #BigData #DataScience #AIInnovation #DataWarehouse

14

😴 Tired of chasing failed dbt runs at 2 AM? Orchestrating dbt across your data ecosystem shouldn’t require late-night debugging. Get 13 ready-to-run DAG code examples that show how to orchestrate dbt at scale with Apache Airflow. Learn how to: - Run dbt with BigQuery, Databricks, Snowflake and Postgres - Use DbtTaskGroup and DbtDag for fine-grained orchestration - Inject runtime variables and host dbt docs in Airflow Download the guide via the link below and get your nights back.

52

4 Comments

🎙️ Coming Soon to The Databricks Diaries Podcast Leading from the front: Richard Shaw on How Databricks Thinks About AI Readiness. In this upcoming episode, I sit down with Richard Shaw, Technical General Manager UK & Ireland, AVP Field Engineering at Databricks, for a candid conversation on how leading organisations are preparing for AI at scale. With over six years inside Databricks, Richard shares a rare behind-the-scenes perspective on: ✔️ How Databricks scaled from startup roots to a global data & AI powerhouse ✔️ Why AI readiness is as much about culture, skills, and operating models as it is technology ✔️ How Databricks partners with customers on their AI journeys ✔️ What it takes to keep teams continuously upskilled in a fast-moving AI landscape ✔️ How Databricks uses its own platform internally to move faster and innovate smarter 🚀 Dropping in Jan 2026— don’t miss it. Link to The Databricks Diaries Podcast below ⬇️ #databricks #podcast #AIreadiness

19

1 Comment

Hi Guys, I have added the final post for database change management for lakehouse. link: https://lnkd.in/eXeggssk This post covers how should be modularise files, so that we can handles hundreds of tables in a lake. Highlights: * Structured Folder Organization: By creating a dedicated folder for each table within our database, we can easily manage our changelogs and keep everything organized. *SQL File Change Sets: Each change set is stored in a SQL file, making it easily readable and deployable, even outside of Liquibase. This flexibility ensures that our changes are always accessible. *Sequential Change Management: By prefixing our change sets with numbers, we maintain a clear order of execution, allowing us to track changes effectively. *Rollback Scripts: Each change set includes a rollback script, providing a safety net for any necessary reverts. This feature is crucial for maintaining stability. *Deployment with Precision: When running the Liquibase update command, we can simply specify the root changelog, and all changes will be deployed effortlessly. Plus, with labels and contexts, we can filter deployments for specific versions, ensuring we're always in sync. This approach not only enhances our database management but also improves collaboration within our team. Excited to see how this will streamline our workflows! #DatabaseManagement #Liquibase #Changelog #SQL #DevOps #DataEngineering #SoftwareDevelopment #DatabaseVersioning #Teamwork

15

1 Comment

At #SnowflakeBUILD London, we just set a new bar for the developer experience. We’ve rolled out a data-native ecosystem that collapses the distance between your first line of code and your first production deployment. What's new: - Cortex Code: Accelerate everything from complex data engineering to agent development with built-in AI assistance. - Shared Workspaces: Your new home base. Bring code, notebooks, and dbt projects into one collaborative view. - Snowflake Notebooks v2: The power of Jupyter meets the scale of Snowflake, now with enhanced ML performance. This isn't just a feature drop. It’s about removing the friction that kills innovation. Build, test, and ship, all without leaving the AI Data Cloud. See the full developer toolkit: https://lnkd.in/gpYbf4CC

201

You can’t fix what you can’t see. Modern data pipelines are complex and with that complexity comes fragility. When data breaks, dashboards mislead, decisions go wrong, and trust erodes - data observability steps in. It’s not just about monitoring infrastructure, it’s about ensuring your data itself is accurate, timely, and reliable. ✔️ Detect pipeline issues before they impact end users ✔️ Reduce time spent firefighting broken reports ✔️ Build trust in data across teams We see data observability as a foundational layer for any analytics-driven business. Without it, even the best models and strategies risk being built on shaky ground. Because reliable data isn't a nice-to-have, it's a must-have. #DataObservability #DataOps #Analytics #Fluidata #TrustInData

13

Fetch query results without ODBC / JDBC bottlenecks. The new ADBC driver for Databricks is now in early release. Install it with dbc. Details in the comments.

86

3 Comments