A couple of weeks ago I had a fantastic chat with Hendrik Krack from CodeRabbit—we talked about what it's like to be an open-source maintainer, what it's like to work on AI, and where we think this is all going! Check it out.

Same model. Inference is the differentiator. Open-weight models are advancing at an incredible pace. Great work from the teams pushing the frontier forward. Real-world outcomes come down to inference. FriendliAI leads in performance, reliability, and cost with super-effective caching. 👇 GLM 5.1 cache hit rate on OpenRouter

KubeCon EU 2026 talks are live! If you're thinking about scaling inference beyond a single cluster, this session dives into federating inference servers—what it takes, what breaks, and how to design it right. A practical, forward-looking blueprint for where distributed inference is heading. #llmd #inference #AI #cloudnative #kubecon https://lnkd.in/g3eZVsr9

You scale the datastore and something else becomes the bottleneck. Connection limits, pooling, proxy layers. You think you’re scaling one thing, but something else caps you first. We’ve seen this show up pretty quickly once connection fanout increases. You can have plenty of headroom in the datastore, but the system in front of it starts to fall behind. There’s a talk at #UnlockedConf from Nextdoor on how they’re handling that with a RESP proxy. Worth a look if you’re dealing with similar constraints. Seattle · May 7

Google Gemma 4 is here. Open-weight models keep getting better—stronger performance, longer context, and multimodal capabilities becoming the default. But the real question isn’t just the model. It’s how well you can run it in production. At FriendliAI, we focus on high-performance, large-scale inference for open models. Same model. Different performance. 👉 Deploy Gemma 4: https://lnkd.in/gESSE4qP #Gemma #LLM #Inference #OpenModels #AIInfrastructure

🚨 Gemma 4 just dropped, marking the first time Google is shipping Gemma under a fully open commercial license with no usage restrictions. If you've been waiting to move your production workload off GPT-OSS, this might be it. > Gemma 4 31B Dense fits unquantized on a single 80GB GPU. Capable of folding two GPU serving jobs into one. > It currently sits at rank 27 on the Arena AI Text leaderboard with a score of 1452, statistically level with Claude Sonnet 4.5 at rank 25. > For the first time in the Gemma family's history, all four models ship under Apache 2.0. For teams building derivative fine-tunes on proprietary data, that license removes the legal review cycle entirely. Pipeshift (YC S24) will be dropping task-specific performance and deployment notes soon. Link to the arena in comments.

CNET published a category-defining piece declaring "claw" a standalone computing category. Multiple vendors are now shipping their own claw variants. Full story: https://lnkd.in/ek5YkaSv

We just open-sourced LLMesh. For the past year we have been building an AI orchestration platform for product teams. At that scale, you can't run every inference call through a cloud API. The cost is wrong, the privacy is wrong, and the latency is wrong. So we built our own infrastructure: a distributed inference broker that pools local hardware into a single endpoint. The problem it solves is simple. Your app points at localhost. It works. You push to staging. It breaks. You switch to a cloud API, start paying for tokens you didn't want to pay for, and send data you didn't intend to share. LLMesh sits between your application and your compute. Your app always hits one endpoint. Your hardware — laptops, GPU boxes, workstations — connects from wherever it is. OpenAI and Anthropic API compatible, so it's a drop-in replacement. No config changes across environments. No data leaving your infrastructure. Think of it as nginx for LLM inference. We open-sourced it because the defensive moat on infrastructure isn't deep enough to justify keeping it closed. The value is in the community, the feedback, and the credibility. MIT licensed. Zero vendor lock-in. If you're running Ollama, vLLM, or MLX locally and managing multiple machines, this is for you. https://llmesh.net https://lnkd.in/g3RQd-Mq

🚀𝗟𝗼𝘄 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 𝗮𝗿𝗲 𝗲𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹 𝘁𝗼 𝗯𝘂𝗶𝗹𝗱 𝗳𝗮𝘀𝘁, 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗶𝘃𝗲 𝘀𝘆𝘀𝘁𝗲𝗺𝘀.🚀 Many engineers over-engineer infrastructure and ignore these battle-tested low-latency techniques used by top tech companies. Here are the guiding principles to help you reduce latency across your stack: 1. For Frequently Accessed Data – Use In-Memory Caching 2. For Slow DB Queries – Use Database Indexing 3. For High Request Volume – Use Connection Pooling 4. For Large Payloads – Use Gzip/Brotli Compression 5. For Global Users – Use CDN for Static Assets 6. For Reducing Round-Trips – Use HTTP/2 Multiplexing 7. For Network Efficiency – Use Request Grouping/Batching 8. For Heavy Background Jobs – Use Async Message Queues 9. For Distributing Load – Use Traffic Load Balancing 10. For Reducing API Delays – Avoid External Dependencies 11. For Near-User Processing – Use Edge Computing 12. For Fast Serialization – Use Protobuf or Avro 13. For Local Performance – Use Vertical Scaling 14. For Faster UX – Use Lazy Loading 15. For Load Sharing – Use Client-Side Rendering 16. For Instant Readiness – Use Prefetching Critical Resources These principles are not theory, they are the foundation of low-latency, high-performance systems at scale (Netflix, Google, Amazon).

Envoy plugins are easy to write and easier to turn into a bottleneck. The plugin model opens up powerful possibilities for storage proxies, but threading models, buffer management, and filter chain design under real throughput are where things go wrong. At Snap's scale, getting those architectural foundations wrong isn't a performance issue, it's an incident. I'm covering the concrete patterns at Unlocked in Seattle on May 7. unlockedconf.io #Valkey #OpenSource #PerformanceEngineering #UnlockedConf