César CLAVÉ

If you search ‘Top data engineering tools to learn’, Apache Spark is almost always in the top 5! here I just published a public repo for anyone exploring Kafka + PySpark + Structured Streaming in a microservices-style pipeline! Helps you get hands-on with near real-time streaming pipelines: - Ingests from Kafka, transforms with PySpark Structured Streaming - Uses PostgreSQL as intermediate storage, then sinks to BigQuery or ClickHouse - Modular & extensible for real-world pipelines Highlights - Pydantic Settings to control all app configs cleanly (env-driven, typed) - Docker Compose spins up everything you need to start with Kafka + PySpark - Step-by-step debugging guides in the README for Kafka and Spark (so you can actually see what’s going on) Repo: https://lnkd.in/gngDE4YS Would love to hear feedback from anyone working with real-time data pipelines! #Kafka #PySpark #StructuredStreaming #DataEngineering #DockerCompose #Pydantic #PostgreSQL #BigQuery #ClickHouse #OpenSource #RealTimeData #ETL #Python #SparkStreaming

12

🚀 New to Databricks? Start Here. Creating your first Databricks cluster is the foundational step in your data analytics journey—and we’ve made it easy. Our latest guide walks you through everything: ✅ What Databricks is ✅ How clusters work ✅ Step-by-step setup instructions ✅ Tips for cost control and performance Whether you're experimenting or building at scale, this is your launchpad. 👉 Read the full guide and start building smarter with Databricks. 📎 https://lnkd.in/eFaPc_CZ #Databricks #DataEngineering #BigData #CloudComputing #Dateonic #AIPlatform #DataScience

2

1 commentaire

Data observability has become essential, but how do you know what solution fits your architecture, scale, and business needs? Our new Databricks Data Observability Buyer's Guide breaks down: 🧠 The 6 most common types of observability tools (and their limitations) 🛠️ How to compare DIY setups, point solutions, native tools, and AI-native platforms 💡 What "comprehensive observability" actually looks like If you're evaluating vendors or even building in-house, this guide will save you hours and deliver clarity! Get the guide now: https://lnkd.in/gayN9ui9 #ModernDataStack #ObservabilityTools #DatabricksOptimization #DataEngineering #UnravelYourData

6

Snowflake or Databricks — which one should YOU actually learn❓ This comparison shows why the answer is not obvious. Snowflake is built for → fast analytics → BI teams → structured & semi-structured data Databricks is built for → big data at scale → Spark, ML & AI workloads → streaming & real-time processing The mistake most people make? Learning a tool without understanding the problem it solves. Analysts ≠ Engineers ≠ ML practitioners The “best” tool depends on your role. #Snowflake #Databricks #DataCareers #Analytics #DataEngineering #TechSkills

37

4 commentaires

🚀 Snowflake Architecture: Deep Diving Into Its Crucial Layers ❄️ In today’s data-driven world, Snowflake isn’t just a cloud data warehouse—it’s a game-changer in how businesses manage, scale, and secure their data. 🌐📊 🔍 Let’s peel back the layers of Snowflake’s unique architecture and understand why it stands apart: ✨ Key Layers of Snowflake Architecture 1️⃣ Database Storage Layer – Efficiently organizes structured & semi-structured data for seamless access. 📦 2️⃣ Compute Layer – Elastic, independent virtual warehouses powering performance & scalability. ⚡ 3️⃣ Cloud Services Layer – The brain that manages metadata, authentication, optimization & security. 🧠🔐 💡 Why it matters? ✔️ Infinite scalability without complexity ✔️ Pay-per-use model for cost efficiency ✔️ High performance & reliability across workloads 🌟 Snowflake’s architecture is not just about storing data—it’s about unlocking insights faster, smarter, and at scale. 👉 Ready to deep dive into how Snowflake can revolutionize your data strategy? #Snowflake #DataArchitecture #CloudComputing #BigData #Analytics https://shorturl.at/UiEYg

1

🚀 The new Databricks SQL Editor is GA — and it’s more than a UI refresh. ✅ Run multi-statement queries in parallel + compare results instantly ✅ Real-time comments & co-editing ✅ AI Quick Fix + natural-language filters ✅ Git-ready for CI/CD

2

Introducing CyberAI, built by DataNimbus in partnership with Databricks. Traditional SIEMs can’t keep up with the scale, complexity, and costs of today’s cloud-native world. Security teams need more than log management — they need data intelligence at scale. With CyberAI, you can: ✅ Ask your logs in natural language ✅ Build and enrich cases with AI assistance ✅ Collaborate seamlessly across teams ✅ Reduce false positives and accelerate investigations ✅ Lower TCO with smarter cost controls This is the power of the Databricks Data Intelligence Platform for Cybersecurity — agility, intelligence, and cost efficiency, all in one. Watch the video to see CyberAI in action. #Cybersecurity #AI #Databricks #DataIntelligence #CyberAI #SIEM

42

If you're building for the AI era, your data pipelines can't lag behind. This new eBook is your hands-on guide to modern data engineering on the Databricks Data Intelligence Platform—covering everything from ingestion and transformation to orchestration, governance, and real-time streaming. Explore how to ✅ Build reliable, scalable pipelines with Delta Live Tables ✅ Manage data securely with Unity Catalog ✅ Streamline workflows with LakeFlow ✅ Design an AI-ready architecture that scales Get the eBook - https://hubs.li/Q03qmgFm0 Leslie Holland | Durga C V K #DataEngineering #Databricks #AIReady #ModernDataStack Myridius

27

Still defining the same metrics over and over in Databricks? 😩 You're not alone. Even with powerful tools like SQL, Spark, and Delta Live Tables, metrics still get: • Defined manually • Rebuilt across projects • Optimized by hand • Duplicated in views or notebooks It’s flexible, but far from efficient. Timbr changes the game: ✅ Automatically generate measures ✅ Reuse logic across your ontology ✅ Eliminate JOINs with semantic context ✅ Speed up queries with a 4-tier cache ✅ Make metrics LLM-ready Databricks gives you the engine. Timbr gives you the model that makes it smart. 🧠 🔍 See the full side-by-side comparison → https://lnkd.in/dwAXC7pr #Databricks #SemanticLayer #DataModeling #SQL #Metrics #KnowledgeGraph #AI #BI

13

Ever wished your data pipelines ran like clockwork — fully automated, reliable, and under complete control? With Databricks Workflows, orchestration lives where your data lives: inside the Lakehouse. That means you can combine data engineering, analytics, and machine learning in one seamless platform — without moving data between services or adding unnecessary latency. Why Databricks Workflows stands out: • Runs tasks, SQL, and notebooks directly where the data resides — no extra data hops. • Supports parameters, triggers, and complex dependencies so pipelines behave exactly how you need them to. • Built-in monitoring and alerting for production-grade reliability. • Eliminates many manual scheduling hassles — move from ad-hoc runs to automated, repeatable pipelines. • Integrates ML and analytics pipelines alongside ETL, making end-to-end automation practical and maintainable. In this guide, you’ll get a step-by-step walkthrough to build your first Databricks Workflow — from defining tasks and dependencies to triggering runs and monitoring outcomes. By the end, you’ll know how to shift from manual execution to automated, production-ready pipelines that scale with your business. Read more: https://lnkd.in/gJYJebMn #Databricks #Workflows #DataEngineering #ETL #Orchestration #Lakehouse #MLOps #Analytics #Automation #DataOps #CloudDataPlatform William Rathinasamy Sekhar Reddy Anuj Kumar Sen Lawrance Amburose Satya Srinivas Veerabadrachari R Brindha Sendhil Praveen Kumar C Rashika S Harini R

15

Modernizing legacy data warehouses is critical but notoriously complex — hidden dependencies, undocumented logic, and manual validation can stall even the most strategic projects. Meet Lakebridge — Databricks’ free migration tool that automates up to 80% of the work, accelerating implementation speed by up to 2x. With profiling, SQL conversion, validation, and reconciliation, Lakebridge tackles complexity so your team can spend more time unlocking insights, and less time untangling legacy code. Migrate smarter, faster, and with confidence: https://lnkd.in/g9EgGtwn

475

10 commentaires

🚨 Stop juggling 5 different data tools. There's a better way. Most data engineers piece together their stack like this: → One tool for ingestion → Another for transformation → A separate one for orchestration → Yet another for data quality checks → And something else entirely for governance It's a mess. Context-switching kills productivity. And stitching it all together? That's a full-time job in itself. Module 5 of the FREE Data Engineering Zoomcamp just changed the game. In this module, you'll learn how modern data platforms unify the entire data lifecycle under one roof — and build a real end-to-end pipeline on NYC taxi data using Bruin as a hands-on example. Here's what you'll build: ✅ Ingest data with Bruin CLI ✅ Write SQL & Python transformations ✅ Design a 3-layer architecture: ingestion → staging → reporting ✅ Configure pipelines with YAML ✅ Handle orchestration, scheduling & variables ✅ Add data quality checks ✅ Use Bruin MCP inside your dev workflow (yes, AI-assisted pipelines 🤖) This isn't theory. This is a production-style workflow — the kind you'd actually use on the job. Course link: https://lnkd.in/gWSKxnpB Are you currently learning data engineering? Drop a 🙋 below — let's connect! #DataEngineering #DataPlatforms #LearnInPublic #DataTalksClub #FreeCourse #Bruin #ETL #DataPipelines #TechCommunity #CareerGrowth

4

Sharding vs Partitioning in Databricks — Lessons from Netflix & Twitter When your datasets hit 100TB+, every decision about how you store and process data matters. Netflix and Twitter get it right — here’s how you can too. This article breaks down: What sharding is and when to use it How partitioning works in Databricks The billion-row problem and how to solve it Advanced Databricks optimizations (G-Ordering & Liquid Clustering) 🔗 Read here: https://lnkd.in/g4GbXVrR Have you used sharding or partitioning in Databricks? Which worked better for you? #BigData #DataEngineering #Databricks #ApacheSpark #DeltaLake #CloudOptimization #DataArchitectur

2

🚀 New resource for Data Engineers, Architects & Tech Leads! We’ve launched a real-world Palantir Data Engineering Track — designed for professionals who want to build at scale, not just learn concepts. This course includes: ✅ Scalable pipeline design & optimization ✅ Data operationalization inside the Palantir Foundry ecosystem ✅ Business-focused outcomes that drive real impact ✅ Led by a trainer with 22+ years of hands-on industry experience The goal? To help serious professionals engineer smarter systems — faster, and with purpose. 💡 Curious how Palantir-level engineering can fit into your org? Let’s connect and walk you through it. #DataEngineering #Palantir #EnterpriseTraining #FlorenceFennel #Foundry #DataTransformation 

2

💡What if instead of writing a query… you could just talk to your data? That’s what I’ve just built — a #Cortex Analyst inside #Snowflake Intelligence — and honestly, it feels like stepping into data, in God-mode. 🎥 Watch it here, and follow the step-by-step walkthrough to create one yourself: https://lnkd.in/g6a8dxhJ No dashboards. No #SQL. No waiting for someone to “pull a report.” Just you, a question, and instant, intelligent insight. “Which products are driving margin growth this quarter?” “What happens if we shift marketing spend by 10% in the West?” Ask it. Get answers. Visuals. Context. Explanations. On the fly. For execs, strategists, and data-driven teams, it’s a new muscle: 💭 Ad-hoc analysis at the speed of thought 🔍 Infinite perspective shifts 🚀 Decision-making without friction Once you try it, you realize you’re not querying data — you’re collaborating with it. That’s the future of intelligence inside #Snowflake. Not BI. Not dashboards. Dialogue. 👇 Dive into the demo — and tell me: Now that you can build an agent to talk to your data - what use-cases would you build for? #Snowflake #CortexAnalyst #DataIntelligence #AI #DecisionIntelligence #Leadership #AnalyticsReimagined

11

From raw campaign data to a fully consumable Semantic View in Tableau. This hands-on quickstart walks you through an end-to-end workflow that connects governed data modeling in Snowflake to BI consumption in Tableau. In this guide, you’ll: • Load and explore campaign data in Snowflake • Automatically generate a Semantic View using the Semantic View Wizard • Query it with standard SQL • Generate a Tableau Data Source (.tds) file • Connect Tableau directly to your Semantic View Get started: https://lnkd.in/gmCdk3VH

46

Gone are the days when you'd be asked: "𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝗿𝗲𝗮𝗱 𝗮𝗻 𝗘𝘅𝗰𝗲𝗹 𝗳𝗶𝗹𝗲 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀?" Today, the real challenge is: "𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗰𝗮𝗹𝗲 𝗘𝘅𝗰𝗲𝗹 𝗶𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆 𝗮𝗰𝗿𝗼𝘀𝘀 𝗦𝗽𝗮𝗿𝗸 𝗰𝗹𝘂𝘀𝘁𝗲𝗿𝘀—𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗿𝗲𝗹𝘆𝗶𝗻𝗴 𝗼𝗻 𝗽𝗮𝗻𝗱𝗮𝘀 𝗼𝗿 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗶𝗻𝘁𝗼 𝗱𝗿𝗶𝘃𝗲𝗿 𝗺𝗲𝗺𝗼𝗿𝘆 𝗶𝘀𝘀𝘂𝗲𝘀?" Thanks to 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸 𝟰.𝟬, now we can build 𝗰𝘂𝘀𝘁𝗼𝗺 𝗗𝗮𝘁𝗮𝗦𝗼𝘂𝗿𝗰𝗲 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿𝘀 in pure Python using the new PySpark-native DataSource API — no Java or Scala required. This solution is:  1. Fully 𝗱𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗿𝗲𝗮𝗱/𝘄𝗿𝗶𝘁𝗲 of Excel files  2. Native support for 𝗯𝗮𝘁𝗰𝗵 and 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴  3. No more driver bottlenecks from pandas.read_excel()  4. Seamless .format("excel") integration in Spark If you’re a data engineer or ML practitioner working in Python and need to integrate with non-standard formats, this is your opportunity to build robust, distributed, production-grade connectors — your way. Try it out on Spark 4.0 or Databricks Runtime 15.3+. There are a lot of updates from Apache Spark and Databricks 😍 let's learn one by one To know more how to achieve this: 🖇️ Medium article: https://lnkd.in/dZGEcGWy 🖇️For further reading: https://lnkd.in/dS2WsqRX #InterviewSeries #DataEngineering #Databricks #ApacheSpark #PySpark #Spark4 #CustomConnector #StreamingData #BigData #ETL #PythonTools #UnityCatalog #ExcelIngestion #DataSource

35

4 commentaires