Building Event Software with Real-World Experience

After years building event-driven systems. Here are the top 4 mistakes I have seen: 1. Duplication Events often get re-delivered due to retries or system failures. Without proper handling, duplicate events can: • Charge a customer twice for the same transaction. • Cause duplicate inventory updates, messing up stock levels. • Create inconsistent or broken system states. Solution: • Assign unique IDs to every event so consumers can track and ignore duplicates. • Design event processing to be idempotent, ensuring repeated actions don’t cause harm. 2. Not Guaranteeing Order Events can arrive out of order when distributed across partitions or queues. This can lead to: • Processing a refund before the payment. • Breaking logic that relies on correct sequence. Solution: • Use brokers that support ordering guarantees (e.g., Kafka). • Add sequence numbers or timestamps to events so consumers can detect and reorder them if needed. 3. The Dual Write Problem When writing to a database and publishing an event, one might succeed while the other fails. This can: • Lose events, leaving downstream systems uninformed. • Cause mismatched states between the database and event consumers. Solution: • Use the Transactional Outbox Pattern: Store events in the database as part of the same transaction, then publish them separately. • Adopt Change Data Capture (CDC) tools to track and publish database changes as events automatically. 4. Non-Backward-Compatible Changes Changing event schemas without considering existing consumers can break systems. For example: • Removing a field might cause missing data for consumers. • Renaming or changing field types can trigger runtime errors. Solution: • Maintain versioned schemas to allow smooth migration for consumers. • Use formats like Avro or Protobuf that support schema evolution. • Add adapters to translate new schema versions into older ones for compatibility. "Every schema change is a test of your system’s resilience—don’t fail it." What other mistakes have you seen out there?

A Senior Data Engineer candidate was asked to design a real-time analytics pipeline during his interview at Netflix. Another candidate in a different loop at Uber got the same prompt. Real-time dashboards look simple until you add one layer of reality: – Add late arrivals? Now you need watermarks, session windows, and late-firing logic. – Add out-of-order events? Now event-time vs processing-time becomes your entire correctness model. – Add exactly-once semantics? Now idempotent sinks and transactional commits are non-negotiable. – Add backpressure? Now Kafka is lagging or your sink is choking and alerts are firing. – Add historical corrections? Now you're reconciling streaming state with batch recomputes. Here's my checklist of 15 things you must get right when building real-time analytics: 1. Start with your latency and correctness contract → Define what "real-time" actually means: sub-second? 5 minutes? End-to-end or just processing? And define correctness: approximate is fine, or must be exact? 2. Choose your processing model: Lambda vs Kappa → Lambda = separate batch + stream paths, eventually consistent. Kappa = stream-only, simpler but harder to backfill. Most companies say Kappa but run Lambda in disguise. 3. Pick your event-time strategy early → Use event timestamps, not processing timestamps. If events don't have timestamps, you're already behind. Decide: use producer time, log append time, or application time? 4. Design your windowing logic to match business semantics → Tumbling windows for fixed intervals. Hopping for overlapping aggregations. Session windows for user activity. Getting this wrong means your metrics lie. 5. Implement watermarking to handle late data → Watermark = "no events before this timestamp will arrive." But late data still arrives. Set your watermark delay based on observed lateness, not wishful thinking. 6. Build a late-firing strategy that doesn't break downstream → When late data arrives after the window closes, decide: update the past metric (retractions), append a correction, or drop it. Each has trade-offs for downstream consumers. 7. Handle out-of-order events with buffering and sorting → Events rarely arrive in order. Buffer and sort within your watermark delay. If you don't, your aggregations are wrong and nobody will notice until the CEO asks why revenue dropped. 8. Design for exactly-once semantics from source to sink → Kafka supports exactly-once within Kafka. Flink supports exactly-once with transactional sinks. But your sink (Postgres, Elasticsearch) must be idempotent or transactional too. 9. Make every sink operation idempotent → Assume every write happens twice. Use upsert patterns: INSERT ON CONFLICT, MERGE, or idempotency keys. Never use blind INSERT or INCREMENT operations. (Continued in comments)

🚀 The Era of One-Size-Fits-All Events Is Over. Stop Doing It. Personalization isn't a single action, it's a series of intentional, strategic choices that come together to make every attendee feel genuinely valued. We’re not just organizing events anymore — we’re crafting journeys. 🧭 In today’s marketplace, attendees expect more than just a badge and a schedule. They want curated content, meaningful connections, and real-time relevance that makes them feel seen. That’s where hyper-personalization comes in. And no, it’s not just using someone’s name in an email. It’s about using data and technology to design experiences that feel custom-built for each person. 🧠📊 As an event marketer, I’m all in on data-driven strategy. This is where we move beyond logistics and design every touchpoint to be personal, memorable, and valuable. Here's some ways that can look like across the attendee journey: Before the Event: 🎯 Targeted Invitations & Content: Use behavioral data to send invites that speak directly to someone's interests. A marketer might get a blog post on campaign strategy, while a developer receives a product case study. 📝 Dynamic Registration: Ask tailored questions based on the attendee’s role or industry to build rich attendee profiles from the start. During the Event: 🤖 AI-Powered Agendas & Recommendations: Event apps can recommend sessions, speakers, and exhibitors based on real-time behavior, interests, and profiles — reducing decision fatigue and maximizing impact. 🤝 Smart Networking: Go beyond job titles. Use AI to match attendees with shared goals, values, or expertise for deeper, more meaningful conversations. 🎉 Personalized On-Site Experiences: Greet attendees by name on welcome screens, print session tracks on badges, or use RFID to tailor in-person interactions. 📽️ Customized Content Delivery: Make booth visits unforgettable. When someone scans their badge, show a video personalized to their company, role, or industry — turning a quick interaction into a memorable moment. 🧢 Personalized Swag: Skip the generic t-shirt. Offer attendees the ability to choose colors, styles, or even print their name on a water bottle or notebook. After the Event 📬 Tailored Follow-Up: Instead of a generic “thanks for coming,” send curated content based on sessions they attended, people they connected with, and their unique interests. 📚 Personalized Content Hubs: Create a portal where attendees can revisit the event — with homepages tailored to their track, interests, or role. 📊 Custom Surveys: Don’t ask vague questions. Personalize post-event feedback forms to reflect their specific journey. 🤔 What's one thing you're doing to add a touch of personalization to your events? Or, as an attendee, what's a personalization strategy that has truly impressed you? Let's share some ideas in the comments! #EventProfs #EventMarketing #HyperPersonalization #EventTech #ExperienceDesign #EventStrategy #PersonalizedExperiences

Over the past 12 months, our average deal size has grown by 113% while win rates have increased from 54% to 58%. Here’s what we’ve learned about moving up market in the Event Tech Industry. 1. Large companies need an easy way for organizers to spin up new events. Everything must be templatized. 2. A pretty UI will never win over an inflexible data model. Scalability and flexibility are the difference between a product vs. an enterprise grade platform. 3. Up market doesn’t just mean bigger events. More often it’s about ability to handle a breath of events 4. Being all-in-one isn’t about the customer not needing other vendors, it’s about them having the option to bolt on best of breed solutions on their timeline. 5. Expandability is key. Open APIs, deep integrations, and development support set the tone during onboarding. 6. With enterprise deployments, its bottom up vs top down. E.g. instead of working backwards from a given event, you work from the program up creating templates for each type of event. 7. Partners matter more. Whether its integrations, professional services, or hardware, it needs to operate as one team. Simply put. Remove all friction. So why did deal size more than double AND win rates increase? Expandability & Flexibility. We’ve been at it for over 10 years. And haven’t been afraid to say “no” to a bad fit. Which allowed us to focus on building a data model that’s adaptable. While maintaining a user interface that anyone in the company can use. Today, we’re able to say “yes” to a lot more. Over the past 12 months, 62% of new customers over $50k ARR required custom implementations. Something we would have said “no” 3 years ago. The future of enterprise tech is Software & Service. That’s what we are building for.

Collecting first-party user event data from websites and apps seems straightforward. Set up a Kafka or Kinesis cluster, add AWS Lambda gateway (or an equivalent), and start firing events from your web or mobile apps. It works great—until it doesn’t. Here are five real-world issues we’ve seen 1️⃣ Cookie tracking issues: The way you’ve been managing cookies to track users and sessions doesn’t work consistently across all browsers. The Apple ecosystem, for example, is known for aggressively expiring cookies to protect user privacy. And tracking users across devices (e.g. mobile, desktop) can feel like rocket science. 2️⃣ Bot traffic or DDoS attacks: Sudden bot traffic or a DDoS attack overwhelms your data pipelines, causing severe delays and data-loss impacting your good traffic. And worse, all the machine-generated junk data corrupts all your downstream dashboards. 3️⃣ Data quality Issues: A developer accidentally changes an event name or property name in your mobile app. This slips through QA, breaking your downstream dashboards. The fix? It has to wait until the next app release the following month. The problem only becomes worse as your event taxonomy grows, and maintaining consistency becomes even more challenging. 4️⃣ GDPR violations: You had a cookie-consent banner, but your pipeline didn’t fully respect user consent. That data eventually made its way into a marketing tool and was used in a campaign—now you’re facing a GDPR lawsuit. 5️⃣Latency and real-time processing: Your team needs real-time recommendations with strict service level agreementsSLAs, but current system capabilities only support best-effort processing speeds. These aren’t hypothetical scenarios—they’re real issues we’ve heard directly from customers and prospects. DIY solutions are great, until they’re not. When configuring your customer data infrastructure, it’s important to consider the potential pitfalls of building your own event pipelines to decide if another approach might be the right choice for your business.  

I’ve been building Event Tech for 13 years. 11 years at Attendify, 1 year at Hopin, and 1 year since starting Highbar.ai I still feel like an outsider. Too many things in #eventtech don’t make sense: 💰 Spending big on services just to make the tech work. 🤷 Consultants to explain what “all in one” and other industry jargon actually means. 🤮 Multi-year SaaS contracts and threats if you need to terminate early. At Attendify our approach wasn’t particularly strategic. We built a DIY event app with transparent pricing because it felt like the only fair way to win. It worked, up to a point. Last year I attended IMEX America as I started building Highbar.ai and snapped a photo during Julius Solaris' session which touched on the industry’s waves of innovation. Online Reg → Event Apps → Engagement Tools → Virtual Events → AI. Each wave added new surfaces, but the same tradeoffs persisted: complexity over usability, customization over democratization. When I started Highbar.ai my biggest question was whether AI will be just another wave, or a true break in the cycle? I truly believe this time is different, because the technology allows us to: ✨ Build personal experiences without the crushing cost of customization. ✨ Deliver discovery that feels natural, not buried in menus and lists. ✨ Capture data and deliver real insights, instead of dashboards no one uses. ✨ Explore business models that make event tech a true win-win. To me, that was always the promise of event apps... A “tracking pixel for the real world.” But the industry never got close, and innovation stalled. We don’t have to repeat the same mistakes. That’s why I started Highbar.ai What are my blind spots? If you think the status quo will prevail, why?

Most event tech startups don't need more advisors. They need someone who will actually tell them the truth. I do that. Every week. Whether they like it or not. I speak to 3 to 6 event tech startups a week. Ya, I know, you think I'm kidding. I love talking with startups. I want to see how they present themselves, what made them do what they are doing. And because I'm nosy and I genuinely want them to land. 😄 This week I met a team building something genuinely cool. An AI-native agency. Their core product was around turning event data into a revenue and execution engine. Super smart team. Smart product. Real results. Strong first client story. They'd also just spent $30K on an industry event and membership that got them expensive dinners and a badge. Nobody told them not to. And when I asked them what three outcomes they actually deliver to a client? It didn't flow right. Then they spoke about features. Then they compared themselves to the other systems in the market. [never do that, by the way] That inconsistent flow is not a product problem. It's a reality problem. They were too close to it to see where the friction actually lived. If you're building in tech for events, pressure-test these constantly: Buyer clarity.  Can one person explain what you do in one sentence? If not, your market won't either. "Event organizer" is not an ICP anymore. Go niche. Pick one buyer. Talk to one buyer. Real pain vs. nice-to-have.  Event organizers aren't stuck because they lack tools. They're stuck because changing how they work is terrifying. If your pitch doesn't speak to that fear first, you've already lost them. Industry fit or are you a fit in their ecosystem? Not in a demo. Not in a deck. In real, messy, under-resourced event teams where the intern left with the data and the spreadsheet lives in someone's personal drive. Does your product actually work there? Where your money is actually going. This industry will charge you $20K to $30K for access to a room full of your competitors. Expensive memberships are not a perk anymore. Know the difference before you write the check, and be prepared to negotiate the package you are being offered. By the end of our call that team wasn't pitching a platform anymore. They were talking about the real problems they solve and who's actually drowning in it. That shift? That's everything. Those that know me know I don't sugarcoat anything. We don't have time for that. The event tech companies that win in the next few years won't be the ones with the most features. They'll be the ones that understand this industry better than anyone else in the room. 🔮 And those that super niche in the problems they are solving. And those they are serving. Dig deep on the ICPs not only by titles. That's why I do this. I'm nosy. I care. And I will absolutely tell you when your messaging is way off. Oh, and so much more. But I am loving all the new products coming up! Carry on. 🧿

Printing a receipt is easy. Reprinting it? Not so much. Let me explain 👇 Our app handles store returns. Customer comes in → Returns item → Gets refund → Gets receipt. Easy, right? Now imagine this: Customer comes back the next day and asks: “Can I get a copy of my receipt?” That’s where things get tricky. The original transaction is done. The upstream service that gave us the data is silent now. We can’t reprocess the transaction or we risk double refunding. So what do we do? Answer: Event-driven architecture. Here’s how we solved it: ➞ The upstream service emits an event when the return is completed. ➞ That event is published to Kafka. ➞ Our service consumes it and stores what’s needed for a reprint. ➞ Now, whenever we need a receipt later, we’re covered without re-triggering the business logic. You might ask: “Why not just store all this in the upstream DB and call it when needed?” Because then: ➞ Every consumer needs an API key. ➞ Every new consumer adds load. ➞ The upstream owns the DB but now they’re also managing access and logic for everyone. With events, none of that’s needed. It’s decoupled. It’s scalable. It works. Anyone else dealt with tricky “edge cases” that changed the way you thought about system design? #SoftwareEngineering #EventDrivenArchitecture #Kafka #SystemDesign #CareerInTech #BackendEngineering #Microservices

Here's what nobody tells you about event-driven apps with notifications, signals, or streaming updates: Most teams are re-inventing the same reliability problems that databases already solved decades ago. I've seen (and built) the classic stacks: you bolt on a message queue, send notifications out, maybe checkpoint some state. But if your queue and your data store aren't tightly coupled, you get the same pain over and over: - Lost events when a crash happens between DB and queue - Out-of-order signals because the data and the stream aren’t synced - Painful retries and dedup logic, all glued together by hand What surprised me is how much of this goes away when you use Postgres as the core state *and* event backbone. When events are modeled as transactional writes, you get atomicity for free: either both the event and the state change happen, or neither does. No more "half-sent" notifications or orphaned signals. This isn't theoretical: real systems checkpoint every notification as part of a durable workflow. Postgres' ACID guarantees handle the messy parts: crash recovery, ordering, and ensuring every event is tied to real state. So before reaching for a new eventing tool, ask: does my database already have my back ? :) #postgres

*One of the Most In-Depth Event Systems I’ve Built in the last 4 Years* At IBTM Barcelona I got to join Julius Solaris and Stephenie Lintl-McLean on stage to talk about AI and automation in events. On my return I officially handed over one of the most comprehensive systems I’ve created. It starts with a simple Google Form... ...and ends with an event fully automated down to every detail. With tasks routed to the appropriate departments and a clear list of any remaining manual actions. This system supports a global team handling more than 70 events per year, where efficiency is essential. ✅ What it does do - Automates heavy lifting and repetitive admin work - Streamlines communication across multiple departments & stakeholders - Assigns tasks automatically based on roles and responsibilities - Captures essential data in a consistent, structured way - Reduces errors, delays, and unnecessary back-and-forth - Creates clarity on timelines and expectations from the start - Leaves only unavoidable manual tasks for the team to complete ❌ What it doesn’t do - This system does not replace experienced event managers. - It does not replicate the judgment required to ask the right questions when booking or confirming a venue, - the ability to foresee contingencies, or the intuition gained from years of working with diverse venues, stakeholders, and situations. - It also doesn’t remove the human nuance needed in real-world event execution. 🚀 What it should do - Create a clear, consistent, and scalable process - Ensure all expectations and requirements are gathered and communicated properly - Eliminate unnecessary manual work - Support teams rather than replace them Provide a strong operational foundation that improves with each event. You'd be amazed how short the tech stack was that we've used to create this, if you want more details comment 🤓 This is a sneak peek of the process ⬇️ Kent Martin - Claire Hunter - Jorge Eduardo Q. - Gus Hotchin - Katie Rose Hester