LLM Fails in Production: Understanding Knowledge Representation

In contrast to the Cognitive Dead Forest theory, Generative AI's inability to answer concrete software architecture questions (e.g. the output reveals that the question cannot be answered from their training data, or is mean-reverting in other ways) can be used as a litmus test for competitive advantage, i.e. how ahead of the curve my organisation is on technology. It also shows how limited the technology is; even the reasoning models cannot extrapolate.

Most agentic AI systems don't fail because the LLM is bad. They fail because of how the system around it was built. We Partnered with Paul Iusztin (Decoding AI) that covers exactly this. He breaks down 6 engineering mistakes that kill agents in production, often quietly. The 6: 1. Context window mismanagement: treating it as a dump, not working memory 2. Overengineered architecture before the problem actually requires it 3. Using agents where a deterministic workflow does the job 4. Brittle output parsing that breaks under real data 5. No planning logic in the tool loop, just reaction 6. No eval framework from day one, so degradation stays invisible These don't fail you individually. They compound. Full guide: https://lnkd.in/gcDWWZXs

There is a structural flaw quietly draining enterprise AI budgets. Most engineering teams are still building Retrieval-Augmented Generation using basic similarity search. This forces the AI to grab paragraphs that merely sound like a user's question without applying any actual reasoning. The result is context collapse. Your models hallucinate on dense documents because they are searching blind. The industry is mitigating this through Agentic frameworks. Rather than a blind database search, an intelligent reasoning agent is deployed to plan the retrieval strategy before execution. Stop chunking blindly. Upgrade your architecture. #AIArchitecture #MachineLearning #SoftwareEngineering #TechLeadership #GenerativeAI #Automation #AI #FutureOfWork #RAG #AIAgents #GenerativeAI #LLMs #ArtificialIntelligence #B2B

The AI edge in 2026 won't come from bigger models. It'll come from better loops. One pattern keeps showing up in the latest research — and once you see it, you can't unsee it. Hypothesis → Verify → Replan. Instead of an LLM generating a single answer and hoping for the best, the agent proposes, checks against a real source, and iterates from the strongest branch. It's not magic. It's structured reasoning. The new arXiv paper on Agentic AI (2603.20639) takes this further, arguing that the next intelligence explosion won't be a single bigger model. It'll be a network of cooperative agents that learn and coordinate together. Three things stood out to me: 1. Ecosystem over model. Competitive advantage is shifting. The teams that win won't own the biggest LLM — they'll own the most effective agent ecosystem around it. LangGraph and MetaGPT are already making this buildable today. 2. Governance has to be in the loop. Multi-agent systems introduce real audit complexity. The paper proposes constitutional AI rules and role-based access as the answer — declarative constraints, not manual oversight at every step. This is the right framing. 3. The LLM is a component, not the product. Modular, graph-oriented architecture — deployable on Kubernetes or edge — means the underlying model can be swapped. What you're really building is the orchestration layer around it. The honest trade-offs? Memory persistence adds storage overhead. Multi-agent coordination adds latency. More moving parts means more to audit. None of these are blockers — they're engineering problems with known mitigations. What I keep coming back to: the orgs treating AI as "call an API, get an answer" are building on sand. The orgs building verification layers, governance models, and agent orchestration, they're building infrastructure. The gap between those two groups is only going to widen. Have you started building multi-agent workflows in production? What's been the hardest part to get right? #AI #AgenticSystems #LLM #SystemDesign #AIEngineering #TechLeadership #CloudNative

The pattern that separates production AI pipelines from expensive demos: code orchestrates, LLM reasons where machines can't. Three lessons from building rule-heavy classification systems: 1. Scoped modules beat monolithic prompts. Even as frontier models improve on long contexts, splitting your rulebook into focused modules is better architecture — lower cost, faster, testable in isolation. 2. LLMs interpret ambiguity. Deterministic code enforces logic. The systems that work best in production use both — LLM extracts and reasons, code validates and decides. Giving the LLM control flow in a known decision space adds unnecessary non-determinism. 3. Your system should need the LLM less over time. Cache validated decisions. Learn from human corrections — every human override is training data for the next deterministic rule. Replace stable patterns with deterministic classifiers. Reserve the LLM for edge cases where reasoning matters. #ai #llm #contextengineering #utisha

There’s a growing focus on building AI agents. More tools. More frameworks. More orchestration layers. The assumption is that better systems come from better architecture. But every system starts somewhere earlier. With a prompt. Not just a question, but a definition of: • intent • constraints • success criteria • context That starting point is not a detail. It’s the foundation. If the prompt is vague or underspecified, the system fills in the gaps. Those assumptions compound across steps. And by the time you see the output, the issue is no longer visible. This becomes more important as systems become more complex. In multi-step workflows and agent-based systems, the initial prompt isn’t just input. It becomes the operating logic. A low-fidelity prompt doesn’t just create a weak answer. It creates a weak system. Personal perspective. Not speaking on behalf of any employer or organization. If you want the deeper breakdown, I wrote more here:

Been seeing a lot of posts lately about the gap between AI systems and governance. A lot of it comes down to language. People are using the same words to describe very different models. I’ve also had a few people ask how I define things inside my own framework. So I put it together. This is the set of definitions we actually use in TACA / Minerva. They’re constraint-based and tied directly to runtime behavior, not policy descriptions. No authority → no execution path Not “validated,” not “refused” Just not there DOI: https://lnkd.in/efesWdCH #AIGovernance #AIArchitecture #AgentSystems

There’s a clear signal emerging in AI right now: 👉 The race to build external memory for LLMs is on. In just the past weeks, we’ve seen a wave of approaches — from structured knowledge bases (like Karpathy’s proposal) to persistent memory systems (like MemPalace and others). Different designs, different abstractions, but all solving the same core problem: LLMs don’t have native, updatable memory. --- LLMs are incredible at compressing knowledge into weights. But the moment you want to add or change knowledge you’re back to: - retrieval pipelines - summaries In other words: «We’re spoon-feeding memory through a text interface.» --- And that’s the fundamental limitation. All current approaches — no matter how sophisticated — still rely on: 👉 serializing memory into tokens This creates a hard bottleneck: - limited bandwidth - forced linearization of structure (graphs → text) - pre-compression before reasoning Even perfect retrieval can’t escape this constraint. --- Where this likely goes The principled solution will probably require: - memory that is not converted to text - direct interaction with structured or latent representations - models that can query and update memory as part of their forward pass In other words: «Native memory, not serialized memory.» --- We’re not there yet — mainly because: - addressing memory reliably is hard - training across external state is non-trivial But the direction feels inevitable. --- My take The most interesting part isn’t the memory format itself. It’s how efficiently we can project large state into the model’s reasoning process. Because today: «Memory isn’t the bottleneck. Bandwidth is.» --- Curious to see how this evolves — feels like we’re at the very beginning of a much deeper architectural shift. #LLM #AIArchitecture #AgenticAI #RAG

AK’s new approach to knowledge management is very interesting. With the right use of AI agents, it may offer a surprisingly simple system for personal knowledge work—without relying on RAG or vector databases. That said, I still see a challenge in using this method to build shared knowledge effectively across an engineering team. https://lnkd.in/ej6ZR2W2