Apache Iggy (Incubating)

#iggy #rust #oss #asf #apache #streaming #messaging

Check out this blog post by Gordon Murray, who used Apache Iggy as the event spine for a 6-tier streaming lakehouse. Incredibly efficient, barely touching the CPU. https://lnkd.in/dScMAYPY #iggy #rust #apache #asf #streaming #messaging

Would you like to start contributing to OSS but don't know where or how to start? Check out this insightful, in-depth blog post by one of our community members! https://lnkd.in/dbKfbuhk #iggy #asf #oss #apache #rust #streaming #messaging

Innovation is thriving within the #Rust ecosystem at The Apache Software Foundation. We are thrilled to celebrate the incredible momentum of our most starred Rust projects. A massive congratulations to the communities behind these top 3 powerhouses: Apache DataFusion - A massive achievement reaching over 8.5k ⭐! You are leading the charge in query engine performance. Apache OpenDAL - Crossing the 5k ⭐ milestone. Unlocking seamless data access across varied storage layers. Apache Iggy (Incubating) - Rapidly climbing with over 4.1k ⭐. Building the future of high-performance message streaming. The future of open-source data infrastructure is written in Rust! https://iggy.apache.org #Rust #OpenSource #Apache #DataEngineering #Streaming #BigData #Cloud #ApacheIggy

I've been building a streaming pipeline as a learning project with no traditional database. Live crypto ticks from Coinbases Websocket service flow through Apache Iggy, get processed by Flink, and land in Paimon (warm tier) and Iceberg (cold tier), with Fluss for low-latency SQL and LanceDB for vector similarity search. No Flink 1.20 connector existed for Iggy, so I built a source sink with checkpointing. That ended up being the most educational part of the whole project. A few gotchas that cost me a few hours each: - Paimon's aggregation engine treats every INSERT as a delta. Insert your seed balance twice and you've got $200K instead of $100K in my case. Seed jobs must run exactly once. - Flink HA will resurrect finished one-shot jobs. Your seed job runs again after a restart, and now that $200K is $300K. Always verify dead jobs aren't lingering in ZooKeeper. - DuckDB can't read Paimon PK tables correctly. It globs all parquet files including pre-compaction snapshots, so you double-count everything. Fine for append-only tables, misleading for anything with a merge engine. Full write up and source in the comments below

Thanks to the never ending curiosity of Dipankar Mazumdar, I recently found out about Apache Iggy (Incubating), a Kafka alternative written in Rust. As I am wont to do, I wrote a "What the Heck is Apache Iggy?" article for HackerNoon for your weekend reading pleasure. https://lnkd.in/gvHEda2Z #apachekafka #dataengineering #streamingdata

The updated site and documentation are live! Check it out at iggy.apache.org #iggy #asf #apache #oss #streaming #rust #docs #messaging

I built an open-source Apache Flink source connector for Apache Iggy! Apache Iggy is a Rust-native message streaming platform and an interesting high-performance alternative to Kafka. An official Flink connector exists, but it targets Flink 2.1. I needed Flink 1.18 compatibility for a streaming platform I'm building, so I wrote my own. Key features: • Checkpoint-safe offset handling • Flink SQL support via DynamicTableFactory • Multi-partition consumption with round-robin fairness • Dynamic partition discovery • Configurable TLS, batch size, consumer ID, and poll backoff • Flink metrics (numRecordsIn, numBytesIn) • integration and unit tests! If you're working with Iggy or Flink, I'd love feedback https://lnkd.in/duSq7a83 #ApacheFlink #ApacheIggy

https://lnkd.in/dAGct-J2