Raul Landa

🚢 New in Compute Tracks 🚢 We promoted some solid improvements overnight. • Batch compute (⏱️ >15min jobs, so we can support longer running processes and model training) • Live track generation (watch the model “think”) • Agent Mode (alpha): edit across steps 🤖, we recommend to use with caution ⚠️ • Dozens of UI/UX upgrades to tighten up the experience 🛠️ Early adopters, thank you. www.computetracks.com #AI #LLM #DataPipelines

1

I might be a bit late to the party, but Spark Declarative Pipelines look seriously impressive. What really caught my attention is how similar this feels to the problem DBT is trying to solvev but pushed further into the Spark ecosystem and, ultimately, lakehouse architectures. The idea of defining data workflows at a higher level of abstraction while letting Spark handle the execution details feels like a big step forward for data engineering productivity. Definitely planning to run a couple of POCs to explore its potential in real-world scenarios. Curious to hear if anyone here has already tried it! what’s your experience so far? https://lnkd.in/efvmGyQq

11

Artem explains why distributed tracing is more than just structured logging. He illustrates how tracing follows requests across services to provide insights into latency and dependencies - capabilities not available with traditional logging. Through a real-world example of troubleshooting a slow checkout process, he demonstrates how tracing could identify bottlenecks that would be difficult to spot with database logs alone. While acknowledging tracing's effectiveness for debugging distributed systems, he notes that implementation requires significant expertise. Watch the full episode: https://ku.bz/9sGxhmm8s

4

1 Comment

Embedding JSON feels intuitive, but modern embedding models weren’t trained for it. Oleg Tereshin explores how aligning data structure with model training improves RAG retrieval.

8

We released Apache Camel 4.14.0 today. This is a new release LTS with 107 new features and improvements: https://lnkd.in/eggBFbsC #ApacheCamel

32

1 Comment

Most ML teams don’t struggle with models, they struggle with everything around them. Environment setup takes days, Iteration cycles take weeks, Evaluation pipelines break silently, The model is rarely the problem. This feels solvable

16

AI agents are only as powerful as their memory. Redis has been named the #1 data storage choice for AI agents in 2025. The reason? Redis delivers what AI truly needs: - Real-time vector storage - Semantic caching - Long-term memory for LLMs At Bassirah, we see this as validation of why Redis is central to the next generation of AI-native data platforms powering everything from agentic RAG systems to enterprise decision engines. 🔗 Read more: https://lnkd.in/db47k6zG #AI #Redis #RAG #LLM #DataEngineering #AIAgents #ArtificialIntelligence

2

The real engineering challenge isn’t squeezing more capability out of LLMs. It’s figuring out how to actually run them safely. When an agent can write and execute code, you’re not dealing with AI-powered software anymore. You’re dealing with autonomous software running inside your infrastructure. Most stacks were not built for that. https://lnkd.in/gEScgXdR

7

Excited to see inference moving beyond GPUs, with Yotta Labs and RadixArk bringing SGLang to AWS Trainium & Inferentia, pushing toward a truly multi-silicon serving future 🔥 Check out the blog here👇

49

📣 PSA for those that live under a rock: Apache Airflow 3 is now available! Vikram Koka and Oliver Dykstra from Astro customer Texas Rangers Baseball Club spoke with VentureBeat about everything new in Airflow 3, and why Airflow is the industry-standard for data orchestration and unified DataOps. Read the full article: https://bit.ly/42JtoQC #ApacheAirflow3 #Airflow3 #DataOrchestration #DataOps #DataEngineering

22

1 Comment

The real shift in AI infrastructure isn’t new models. It’s inference moving beyond GPUs. We just published a deep dive on running SGLang on Amazon Web Services (AWS) Trainium and Inferentia. The goal is simple: make it easier to run LLM inference across different types of hardware, not just GPUs. This is where things are heading. Multi-silicon, not single-vendor lock-in. Built with SGLang and the RadixArk team. Full breakdown: https://lnkd.in/enXu_4fE

8

Cloud compute costs are unpredictable — until now. Most teams running data experiments only see what they've spent when the monthly bill lands. By then, the damage is done: overprovisioned clusters, repeated runs, and budgets blown past what anyone expected. Yeedu's Cost-Aware UI changes that by putting real-time cost visibility — measured in YCU (Yeedu Compute Units) — directly inside the workflow, at every run and every cluster. No more waiting for invoices. No more guesswork on spend. Here's what that looks like in practice: → Live YCU tracking while a job is running → Run-level cost comparison across experiments → Cluster-level visibility to spot over-provisioned infrastructure Cost becomes part of how teams work — not a surprise they manage after the fact. Read how Yeedu makes compute spending transparent, predictable, and actionable. https://lnkd.in/gtPdZGyX  #FinOps #CloudCostOptimization #Yeedu #DataEngineering #CostVisibility 

16

Ever wondered why CDC UPDATE always outputs tworecords—BEFORE and AFTER? Turns out, it’s not a format choice but the foundation of correctness: idempotency, replay, PK changes, lakehouse merges, and distributed consistency all depend on it. A single AFTER can break your entire pipeline. Deep dive inside 👇 #CDC #MySQL #SeaTunnel #DataEngineering #Iceberg #Paimon #Hudi https://bit.ly/3MhcVP1

1

We’ve published a GitHub repo DWH & BI Fundamental with an end-to-end setup for: - Apache Superset (for BI dashboards) - Linux Centos - Apache Doris (as the data warehouse) - Linux Centos - Example datasets & schema design (star schema, SCD, Data Mart) - Ready-to-use ETL & configs to explore dashboards & KPIs It’s a practical starting point for anyone learning or building with open-source Data Warehousing + Business Intelligence 👉 Explore the repo here: https://lnkd.in/ggCedf6D #GitHub #DWH #BI #Analytics #DataEngineering

11

We just open-sourced FuzzBerg - a #fuzzing framework built for Apache Iceberg 🧊 While there are available fuzzers for individual file formats like Parquet or Avro, FuzzBerg is the first (to our knowledge) #opensource fuzzer which fuzzes all of Iceberg table format holistically properly handling account metadata and snapshots. We already used it to improve Firebolt's own #Iceberg support (and found some "fun" surprises along the way). Now we're sharing it with the community. 🧩 📖 More in the blog by Abhishek Sen: https://lnkd.in/dS59r-65

29

Everyone benchmarks latency. Almost nobody benchmarks accuracy drift across chipsets. Your model scores 93% on a Snapdragon 8 Gen 3. Ship it. But here's what your dashboard doesn't show: 70% of your users aren't on flagship devices That same model scores 79% on a Snapdragon 6 Gen 1 And 71% on a Snapdragon 4 Gen 2 You didn't ship a bad model. You shipped a model you only tested on one device. The problem isn't the weights. It's the assumption that accuracy is constant across hardware. It's not. Here's why: NPU precision varies by chipset — INT8 rounding behavior differs between Hexagon generations Operator fusion changes — the runtime optimizes differently per SoC, sometimes trading accuracy for speed Memory constraints force fallback — on lower-tier chips, some ops fall back to CPU, changing the execution graph entirely Cloud benchmarks test your model in isolation. Real deployment means your model + firmware + thermal state + runtime scheduler — all interacting at once. The fix isn't better models. It's better testing. Test on the devices your users actually have. Not just the ones in your lab. We're building EdgeGate — automated regression gates that test your model on real Snapdragon hardware before every release. No emulators. No simulators. Real devices, real results. #EdgeAI #MLOps #Snapdragon #QualcommAIHub #OnDeviceAI

1

P99s are the latency you feel but never want to explain. The OpenFGA team built a self-tuning planner that learns in production and cut P99 latency by 98%. No static rules. No guesswork. Just data doing the deciding 🤖 Read the blog by Yamil Asusta and Yissell Garma 👉 https://bit.ly/4t6U8ab

17