Harvinder Atwal

Reflections on Big Data Ldn - my first time at Big Data and it's a funny old show. Guys, did you know there's these things called AI Agents? They're here to solve all our challenges... It's good to hear from vendors and to know what products are about, but I'd much rather hear from people in the real world about the issues they are facing and how better data will solve them. I get the vendor focus to keep the event free but it felt imbalanced. Best thing I heard was a consistent message that data governance is a key to AI success. I have always thought that AI ready data does not mean your data has to be 100% perfect before you can use AI. Instead you should UNDERSTAND your data and use that understanding to guide how you take and use the outputs from AI. Basics of data governance and management constantly get ignored, is there an owner, do we understand the lineage, are there data models/dictionaries/glossaries? Let's get this done so we can have confidence in our AI based decision making. This was the only photo I took as well because, I'm rubbish at doing photos, but was amused by what "unprecedented networking" might be... And, I also had a lovely time at a Cap Gemini hosted lunch meeting some fantastic leaders from across government!

33

6 Comments

If your dashboards take minutes to load, you’ve already lost. Slow dashboards cost credibility. Snowflake in Australia gives BI speed, Databricks gives ML power. Which one is your team optimised for?

1

🎁 𝗪𝗵𝘆 𝗴𝗶𝗳𝘁𝗶𝗻𝗴 𝗵𝗮𝘀 𝗯𝗲𝗰𝗼𝗺𝗲 𝗲𝗺𝗼𝘁𝗶𝗼𝗻𝗮𝗹𝗹𝘆 𝗵𝗮𝗿𝗱𝗲𝗿 𝗶𝗻 𝘁𝗵𝗲 𝗮𝗴𝗲 𝗼𝗳 𝗮𝗯𝘂𝗻𝗱𝗮𝗻𝗰𝗲 I am a data psychologist. While analysing retail customer behaviour I noticed something odd... People freeze most when the gift matters most. Not because of price. Not because of options. But because of judgement. Gifting activates fear of getting it wrong - of being misunderstood... Of signalling that 𝘆𝗼𝘂 𝗱𝗼 𝗻𝗼𝘁 𝗸𝗻𝗼𝘄 𝘀𝗼𝗺𝗲𝗼𝗻𝗲 𝗮𝘀 𝘄𝗲𝗹𝗹 𝗮𝘀 𝘆𝗼𝘂 𝘀𝗵𝗼𝘂𝗹𝗱. In behavioural terms, it looks less like shopping and more like high-stakes social risk. More choice does not solve this. More reviews do not solve this. More 'inspo' does not solve this. Because the problem is not LOGICAL. It is PSYCHOLOGICAL. People are not asking, 'What should I buy?' They are asking, 'What does this say about me?' That insight led me and 🎀 Vaida S. to build Pebble Gifts. Not as another recommendation engine. But as a way to reduce gift anxiety - to replace overthinking with reassurance, and performance with intention. Early data is already telling: when anxiety drops, people do not spend more. They decide faster. With more confidence. And far less regret. Which makes me think: The next wave of retail will not be about better algorithms. It will be about 𝗲𝗺𝗼𝘁𝗶𝗼𝗻𝗮𝗹 𝘀𝗮𝗳𝗲𝘁𝘆. If this resonates, Pebble is live at 👉 www.pebble-gifts.com. Built for those who care. 💛

72

17 Comments

Patterns I keep noticing (stories about data leadership)… The first two patterns in this series connected quickly: • Pattern #1 explored why people disengage when data feels technical • Pattern #2 explored what breaks when we don’t capture decision knowledge There’s a third pattern that explains why neither of those ever really gets fixed. Pattern #3: Data sits in the organisational void Most organisations have long-established functions with clear purpose. Technology has existed as a formal organisational unit for decades. It owns systems of record, platforms, security and delivery. Its KPIs are uptime, cost, risk and execution. Business functions own outcomes. Revenue, margin, growth, accountability. No one endlessly debates where HR, Finance or IT should “sit”. Their role is understood. Data is different. Data teams are relatively new, and they emerged for good reasons. Transactional systems don’t optimise for decisions. Commercial teams needed abstraction, meaning and speed. So data inserted itself between Technology and “The Business”. Not as a replacement. As an interface. And that’s where the discomfort begins. Data doesn’t fully belong to Technology, even though it relies on technical tools. It doesn’t fully belong to the business, even though it exists to support decisions. So it gets treated like Technology: • funded through technology-style business cases • staffed with technical profiles • measured on delivery and output While being expected to behave like the business: • influencing decisions • changing behaviour • improving outcomes That tension never really resolves. Which is why one question never goes away: “Where should the CDO sit?” Closely followed by another: “How do you demonstrate value?” Very few data teams own a P&L. So they don’t create value directly. They enable it. They help Sales sell more. Marketing target better. Operations run leaner. Which means data’s value is always embedded in someone else’s success. When things go well, the function with the P&L gets the credit. When things don’t, data suddenly becomes very visible again. So data leaders end up trying to prove their worth by borrowing someone else’s value. An uncomfortable, political and fragile position. So data ends up judged on outcomes it can’t directly control and constrained by structures designed for things it isn’t. That’s the void. It’s why data leadership feels permanently transitional. Why remit debates never settle. Why value conversations never quite close. This isn’t a failure of data teams. It’s a signal that data behaves more like a capability than a function, but we keep trying to govern it like one. Next, I want to explore how organisations try to fill this void — and why those attempts so often fall short. In the meantime, where does data really “sit” where you work, and how do you evidence “value”?

62

25 Comments

Reliable data is the foundation of sound economic policy. In the UK, the "Office for National Statistics" (#ONS) which is an independent department reporting directly to Parliament, has delayed its latest UK retail sales release by two weeks, citing the need for additional quality checks. Retail sales are a key barometer of consumer confidence and spending, shaping decisions at the Bank of England on interest rates and informing government policy on growth and living standards. While the pause raises questions about consistency, it also underlines the critical importance of accuracy. A single misstep in reporting inflation, GDP, or jobs data can ripple through markets, impact borrowing costs, and alter the economic narrative for millions. Delays may frustrate, but I believe it is better to have a short wait for numbers which are reliable, than to publish uncertain numbers with the risk that decisions are based on flawed numbers. In statistics, as in policy, robustness, consistency through time and meaningful data matters more than outright speed. In the private sector, companies such as #Kantar #GfK #NIQ and #YouGov regularly face similar decisions and dilemmas - where some of the key input data looks noisy or is missing. That is why having strong processes based on rock solid scientific methodology, and sufficient expertise and bench strength in #statistics at all levels of the organisation is essential for any organisation publishing such data. Source - https://lnkd.in/eN8Ukc6R #datascience #confidence #decisionscience

1

Four weeks into leading a FTSE 100 data and analytics department – a few early observations: 1️⃣ The T-shaped skillset is real. Expect your range to be tested—one moment you're aligning with the exec team on the future of data, the next you're debating Delta Live Tables vs Materialized Views with engineers. You can't achieve the former without the latter. 2️⃣ People remain everything. A great driver in a Ford Mondeo will outperform an average one in a Ferrari. You need both great technology and great people for any transformational data strategy. 3️⃣ Data expertise scales to all industries. Whether it’s retail, property, or infrastructure—the opportunities to improve data access, quality, and actionability are universal. And so are the benefits of doing it well. 4️⃣ Regression is your best friend. Despite the real and growing demand for AI, there are few business problems where regression shouldn't be the first thing you try! #Landsec #FTSE100 #DataStrategy #Analytics #DataEngineering

30

2 Comments

SaaS sales and data subscription sales look similar until you hire wrong. Across the commodity and energy intelligence markets, we still see one assumption surface regularly: Strong SaaS sellers will naturally succeed in data subscription roles. Sometimes they do. Often they don’t. The gap is not sales capability. It is commercial context. In SaaS, buyers are usually purchasing efficiency or workflow improvement. In data subscription models, buyers are purchasing confidence in decision making under uncertainty. Your customers are traders, analysts, and commercial leaders whose P&L is shaped by the quality and credibility of information. They are not asking whether the platform looks good. They are asking: • Will I trust this data when the market moves • Will it influence trading, hedging, or risk decisions • Will it stand up under pressure This is where our grounding in shipbroking and commodity trading environments adds real value. Those markets reward credibility over process, domain understanding over generic playbooks, and trust built over time. For hiring managers, the takeaway is simple. You are not just hiring a seller. You are hiring someone who can operate credibly around trading desks and risk takers. Get it right and it compounds. Get it wrong and it is expensive. #commodityinsights #energy #techsales

10

4 Comments

The UK data talent market right now feels “selective, but still hungry”, especially for people who can ship production-grade data platforms and support AI/ML use cases. On the demand side, the work hasn’t slowed. What has changed is how picky teams are getting about scope and evidence. UK tech leaders are calling out AI skills scarcity as a major constraint, and that pressure is flowing straight into data teams (engineering + analytics + platform) because data quality, governance, and reliability are what make AI usable.  On the supply side, the market is tighter at the entry level and more robust for experienced hires. That’s consistent with recent coverage showing UK grad/junior job postings have been under pressure, while employers prioritise fewer, higher-signal hires.  Compensation is also holding relatively firm. IT Jobs Watch data has UK median Data Engineer pay around £69k over the six months to 1 Feb 2026, with vacancy volumes below the 2024 peak but stronger than the same period in 2025.  If you’re hiring: Tighter role outcomes (first 90–180 days), clear ownership boundaries, and a real view on hybrid expectations are what’s helping teams close roles faster right now. If you’re a candidate: The quickest way to stand out is showing what you’ve owned in production, what improved (reliability, cost, quality), and how you partnered with stakeholders. #DataEngineering #DataTalent #UKJobs #TechHiring #AI #AnalyticsEngineering #DataPlatform #MLOps

3

Our internal Data & Analytics Tech Demos are back — and it’s got me genuinely excited about how we’re delivering innovation. Today, we heard from one squads about how they transformed the way they process a large graph database - reducing processing time from hours to minutes. You may think this was purely a cost optimisation play. Actually, it was about delivering a better user experience: enabling real-time what-if analysis. We’re creating space for our teams to ask “What if we tried this?”

59

4 Comments

✨ Reflections from Snowflake World Tour London ✨ I had the chance to attend the Snowflake World Tour in London, and it was an inspiring look at where the future of data and AI is heading. The message was clear: we’re entering a new era where data platforms are evolving beyond storage and analytics into engines of intelligence. Snowflake is positioning itself at the centre of that shift — not just as a data warehouse, but as a foundation for AI, governance, and innovation. Some key themes that really resonated with me: 🚀 From potential to readiness: AI opportunities are enormous, but many organisations still face challenges with siloed, low-quality data and legacy architectures. Bridging this gap is becoming a strategic priority. 🔗 Trusted data as the foundation: Solutions like Semarchy’s native MDM integration with Snowflake show how unifying and governing core data is essential for building reliable analytics and enabling AI at scale. 🧠 AI, but contextual: Capabilities like Cortex AI and Semantic Views highlight a future where AI isn’t just powerful, but also deeply aware of the business context it operates in — unlocking more relevant insights and intelligent decision-making. ⚙️ Acceleration and agility: Innovations like Gen 2 Warehouses and Snowpark Connect for Apache Spark show how quickly the ecosystem is evolving to meet the demands of modern data products and real-time intelligence. More than anything, the event reinforced how critical it is for organisations to rethink their data strategies holistically — focusing not only on technology, but also on governance, trust, and usability to truly unlock AI’s potential. I’m walking away with fresh ideas on how we can apply these principles to deliver more impactful, intelligent, and trusted data products in our own work. #SnowflakeWorldTour #DataStrategy #AI #DataLeadership #Innovation #MDM #CortexAI #ProductManagement

8

Sure, working in data is about uncovering insights but it is also about destroying assumptions and pushing on the bounds of what’s possible. There has been absolutely no shortage of that in the WineFi 🍷 quantitative journey so far. Much more to come I am sure. 📹 Video credits to Marcus Grip

18

2 Comments

Most data teams spent the last decade optimizing speed: Time-to-Dashboard, query latency, time-to-insight. However, users don’t actually need faster SQL, but a correct answer they can confidently act on quickly. Time-to-Trusted-Answer (TTTA) is a KPI that correlates time and correctness by measuring the full journey from question to correct, explainable and consistent answer, and turns “trust” from a vague goal into an operational standard. I wrote a deep dive here (with examples, tooling, and an implementation blueprint): https://lnkd.in/gmwzJB94 Curious: what’s the biggest trust blocker in your org today—definitions, freshness, lineage, or evaluation?

17

Too many segmentation models never make it out of PowerPoint. You might have a snazzy set of personas, or archetypes to make the c-suite swoon, but are they actually helping drive results? Which customers are in which segment? How are you targeting your most valuable segments? Do you change how you speak to them? And how is this driving value? (What do you mean you don't know how to measure value?...) You catch my drift. Laura's blog explores why so many segmentations fail, and how to make sure they don't.

10

Want to grow from analyst to data leader? Stop focusing only on tools. Start mastering: • Translating business pain into data questions • Saying “no” to bad requests • Defining metrics clearly and defensibly Senior data roles are less about SQL… and more about judgment.

15

The Open Book Exam: Why RAG Isn't Enough to Stop AI Hallucinations If you are working with Generative AI, you have likely heard the acronym RAG (Retrieval-Augmented Generation) positioned as the silver bullet for accuracy. But a fascinating new research paper suggests that even when we give AI the "right answers," it can still fail to use them correctly. Let’s unpack the terminology, the problem, and the new solution proposed in arXiv:2601.03052. The Basics: What are we talking about? Before diving into the research, let’s clear up the jargon. RAG (Retrieval-Augmented Generation): Think of a standard AI model as a student taking a test from memory. If they don’t know the answer, they might guess. RAG changes the rules. It is an "Open Book Exam." It allows the AI to look up information in your private documents (the "textbook") before answering. Grounding: This is when the AI’s answer is firmly anchored to facts found in those documents. A grounded answer is trustworthy because you can point to the source. Hallucination: When the AI confidently generates an answer that is factually wrong or nonsensical. The Problem: Faithfulness Hallucinations Here is the catch: Just because the student has the textbook doesn't mean they read it. The research highlights a critical issue called Faithfulness Hallucination. This happens when the RAG system retrieves the correct document (e.g., an invoice saying "Total: $500"), but the AI ignores it and generates a wrong answer based on its own internal biases (e.g., "Total: $1,000"). Standard verification checks often miss this because the answer "looks" plausible. The Solution: The Internal Reasoning Graph The new paper, "Detecting Hallucinations in Retrieval-Augmented Generation via Semantic-level Internal Reasoning Graph," proposes a powerful way to fix this. Instead of just checking the final answer, the researchers developed a method to check the AI's internal work. They use a Semantic Reasoning Graph to trace the AI's thought process. Imagine drawing a line from every claim in the AI's answer back to the specific sentence in the source text that supports it. Verified: The graph shows a strong, logical path from the Source Document to the AI's Attention and finally to the Answer. Hallucination: The path is broken. The AI gave an answer, but the graph shows it wasn't actually "looking" at the relevant source text when it did so. Why This Matters This marks a shift from "Output Validation" (checking if the answer looks right) to "Process Validation" (checking if the logic is sound). For industries like Finance, Legal, and Healthcare, where "plausible" isn't good enough, this kind of internal verification is essential for building trust in GenAI systems. Source: Detecting Hallucinations in Retrieval-Augmented Generation via Semantic-level Internal Reasoning Graph Link: https://lnkd.in/gNRCeWmW #GenAI #RAG #MachineLearning #AIResearch #TechEducation #ResponsibleAI #DataScience #LLM

8

1 Comment

How much of the £9 billion spent on #insight is TOTALLY wasted? Estimates of the size of the UK research, insight and analytics industry is currently around £9 billion (source: Market Research Society). But a significant proportion is spent on insight that never gets used - never gets implemented - never creates any impact. Sad times 😢 This topic is actually a little more complex and nuanced than you might expect from the headline. Why? I think a multiple set of reasons behind this: 😱 THE RESEARCH MISSES ITS MARK - FINDINGS ARE IRRELEVANT This certainly can and does happen. Typically when the client & supplier are out of sync, perhaps thru miscommunication or below par briefing. Or the lack of experience and commercial nous from the researchers mean that findings are just a bit too meh, or a bit too out of touch. 🤔 CLIENTS DON'T KNOW HOW TO IMPLEMENT INSIGHT What I've always referred to as the 'insight-adoption gap'. This is where the task of implementing insight is beyond the scope of the researcher and beyond the experience of the client (who's skills and experience may well lie somewhere very different). A kind of stalemate position is reached - the opportunities are wasted. 🚧 THE RESEARCH WAS NEVER INTENDED FOR CHANGE This might feel wrong instinctively but often research is commissioned to provide evidence for something the organisation already 'knew' but didn't have 💯 certainty over. It might never have been about driving change. Or, worse - I've witnessed research being used to prove a point, to win an argument between people, teams, depts. So what can we do about this? Some top line thoughts: ✅ Ensure you're only buying insight when you REALLY need it ✅ Brief your suppliers well (use our research version of #BetterBriefs) ✅ Assign a team who will be responsible for implementing insight ✅ Bring your research partner in to help you bridge the gap Our new diagnostic tool (see Customer iQ website) is designed to find out how good you are at both 1) understanding your customers and CRUCIALLY 2) how well you action that insight. What are your thoughts about this? 💬 _______________ My name is Gideon, founder of Customer iQ I live in the world of customer insight Like this post? Follow me for more 🖖

18

7 Comments

Really enjoyed talking with Catherine Dowden-King as part of the Driven by Data podcast last week. Reminded me of the importance in spending time to talk openly as a community of digital, data and tech leaders about common leadership challenges in organisations and what to do about them. We have much more in common than we might think!

79

4 Comments

Great article on leveraging LLM workflows to identify and automate data quality resolution. Data quality gaps can be varied and unpredictable and the LLM workflow provides an approach to generate data expectations , detect violations and remediate. However , a human in the loop approach remains important to ensure transparency and right remediation approach. #datagovernance #ai #dataquality #aifordataquality

17