Paul Brebner

An ongoing activity to benchmark Vector Search on Open Source technologies to understand which is best for RAG use cases.

After attending (and speaking!) at Current London 2025, I was inspired by several talks I attended to take a closer look at some new Kafka 4.0 features. There will be more blogs coming soon.

A small project (but maybe the start of something bigger, an open Kafka performance model?!) to understand, model and predict what resources a Kafka cluster with local storage (SSDs or EBS) needs compared with a cluster with tiered storage enabled. Starting out with an excel model and SankeyDiagrams, I ended up with a JavaScript calculator to compute the minimum resources for IO, Network, AWS instance types/sizes/number for Kafka brokers, and local/remote storage sizes and costs. The first… Show more

A small project (but maybe the start of something bigger, an open Kafka performance model?!) to understand, model and predict what resources a Kafka cluster with local storage (SSDs or EBS) needs compared with a cluster with tiered storage enabled. Starting out with an excel model and SankeyDiagrams, I ended up with a JavaScript calculator to compute the minimum resources for IO, Network, AWS instance types/sizes/number for Kafka brokers, and local/remote storage sizes and costs. The first blog is now out and the calculator prototype is also available. Show less

Apache Kafka Tiered Storage is a major architectural change in the Kafka storage architecture, and has potential benefits for streaming more data for less $. In a new blog series I've started to explore the new architecture, do some initial performance testing, report on the results, look at Kafka space/time and tiered storage use cases, and recently, start to build a Kafka performance model to size Kafka workloads and clusters. I'm also preparing a talk for FOSSASIA 2025 in Bangkok in March… Show more

Apache Kafka Tiered Storage is a major architectural change in the Kafka storage architecture, and has potential benefits for streaming more data for less $. In a new blog series I've started to explore the new architecture, do some initial performance testing, report on the results, look at Kafka space/time and tiered storage use cases, and recently, start to build a Kafka performance model to size Kafka workloads and clusters. I'm also preparing a talk for FOSSASIA 2025 in Bangkok in March - hope to see you there! Show less

Some people think that AI and ML are "new" - however, the field has been around since the 1960's, and I had my first introduction a bit late in the 1980's. I did some AI/ML research and used some of the breakthroughs during my career so I decided to have a go at listing them.

Autonomous incremental machine learning in a robot world to solve blog stacking: "Paradigm-directed computer learning", MSc Thesis, 300 pages, 1984, Waikato University, NZ.

Commonwealth postgraduate research… Show more

Some people think that AI and ML are "new" - however, the field has been around since the 1960's, and I had my first introduction a bit late in the 1980's. I did some AI/ML research and used some of the breakthroughs during my career so I decided to have a go at listing them.

Autonomous incremental machine learning in a robot world to solve blog stacking: "Paradigm-directed computer learning", MSc Thesis, 300 pages, 1984, Waikato University, NZ.

Commonwealth postgraduate research scholarship, UNSW, 1985-89. Developed first-order inductive logic ML algorithms (in temporal domains over 1st order logic e.g. Prolog, also 1st-order clustering algorithms, very fast rules-based systems, etc - invited attendee at 1987 international workshop on ML UCI, multiple UNSW technical reports published.

In the 1990's, I applied some of the ideas from this work to software engineering (startup), in the form of automated (combination of heuristics and prolog style backtracking) test case generation and execution, automatic protocol generation (for Optus Mailbox system integration contract, again utilising specifications and backtracking to generate the protocol implementation at run-time), and a temporal logic enabled file system for the ABC's "D-Cart" digital radio system (using temporal logic to store and retrieve different audio formats over time based queries - e.g. for audio and transcripts etc).

At CSIRO I developed a distributed Prolog model for water catchments, and an expert system for soil hydraulic property modelling.

During the last decades, I specialised in distributed systems and performance engineering - there's lots of overlap between AI/ML and MDAs (model-driven architectures) & data engineering (e.g. forecasting and prediction, optimisation techniques such as box packing). I also explored using GPUs for Markov models - super fast!

In the last 8 years, I've done Spark ML, anomaly detection, and machine learning over streaming Kafka data with TensorFlow! Show less

The Drone delivery simulation project generates massive amounts of spatiotemporal data, so it's an ideal platform to try out some real-time streaming Machine Learning with. This project resulted in a 6 part blog series and numerous talks.

A short project to evaluate Karapace, an open source Kafka schema registry service. Resulted in a 6 part blog series (to be published in 2023) covering Apache Avro, how Karapace works with Kafka producers, consumers and clusters, schema compatibility and schema evolution. Fun!

I was running the first performance engineering track at ApacheCon NA 2022 last year so needed something interesting to talk about. The new Kafka KRaft mode was just out so the timing was ideal to do some evaluations of the performance and scalability of Zookeeper vs. KRaft for both data and meta-data workloads and operations. The conclusions were presented at ApacheCon and in a 3 part blog series (parts 1 and 2 published so far)… Show more

I was running the first performance engineering track at ApacheCon NA 2022 last year so needed something interesting to talk about. The new Kafka KRaft mode was just out so the timing was ideal to do some evaluations of the performance and scalability of Zookeeper vs. KRaft for both data and meta-data workloads and operations. The conclusions were presented at ApacheCon and in a 3 part blog series (parts 1 and 2 published so far).

https://www.linkedin.com/pulse/1st-performance-engineering-track-apachecon-na-new-orleans-brebner/?trk=pulse-article_more-articles_related-content-card

Slides: https://www.apachecon.com/acna2022/slides/04_McCandless_Learning_from_11+.pdf
Blog 1: https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
Blog 2: https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-2-partitions-and-meta-data-performance/ Show less

Currently experimenting and blogging about Cadence, an open source scalable fault-tolerant workflow engine focussed on developers with workflows as code. I've come up with a Drone Delivery Application Demo using Cadence and Apache Kafka (to demonstrate multiple Cadence+Kafka integration patterns). The 6-part blog series is now complete and published.

This project built a demonstration streaming pipeline for ingesting, indexing, and visualizing some publicly available tidal data using multiple open source technologies including Apache Kafka, Apache Kafka Connect, Apache Camel Kafka Connector, Open Distro for Elasticsearch and Kibana, Prometheus and Grafana. Outputs included configuration examples, blogs, and talks.

Quick! Grab your top hat, passport, carpet bag stuffed with (mainly) cash, and your valet (if you have one), and join with me on a wild journey around the world in approximately 8 data centers—a new blog series to explore the world of globally distributed storage, streaming, and search with Instaclustr Managed Open Source Technologies.

In this new blog series, we’ll explore globally distributed applications (probably a Stock Broker) in the context of Cloud Location, including many of… Show more

Quick! Grab your top hat, passport, carpet bag stuffed with (mainly) cash, and your valet (if you have one), and join with me on a wild journey around the world in approximately 8 data centers—a new blog series to explore the world of globally distributed storage, streaming, and search with Instaclustr Managed Open Source Technologies.

In this new blog series, we’ll explore globally distributed applications (probably a Stock Broker) in the context of Cloud Location, including many of the drivers and concerns such as correctness and replication (making sure applications work correctly by having the data they need in the correct locations), latency (reducing lag for end users), redundancy and replication (making sure applications are available even if an entire data center fails), cost (understanding how much it costs to run an application in multiple locations, and if the cost can be reduced), and data sovereignty (ensuring customer data is only stored in legal locations). Surprisingly, Jules Verne captured these elements in Around the World: correctness (proof that they traveled around the world), time (80 days, and selecting locations and routes to travel around the world in the allotted time), success and failure (the wager), money (half Fogg’s fortune was in the carpetbag), and sovereignty (as Detective Fix repeatedly tries to arrest them at locations in the British Empire). By the end of this first blog I wager we’ll have answered our first question of the series: “How many data centers do you need for a globally distributed application?” Show less

This blog provides an overview around the two fundamental concepts in Apache Kafka : Topics and Partitions. While developing and scaling our Anomalia Machina application we have discovered that distributed applications using Kafka and Cassandra clusters require careful tuning to achieve close to linear scalability, and critical variables included the number of Kafka topics and partitions. In this blog, we test that theory and answer questions like “What impact does increasing partitions have on… Show more

This blog provides an overview around the two fundamental concepts in Apache Kafka : Topics and Partitions. While developing and scaling our Anomalia Machina application we have discovered that distributed applications using Kafka and Cassandra clusters require careful tuning to achieve close to linear scalability, and critical variables included the number of Kafka topics and partitions. In this blog, we test that theory and answer questions like “What impact does increasing partitions have on throughput?” and “Is there an optimal number of partitions for a cluster to maximize write throughput?” And more! Show less

This project explored how we added location data to a scalable real-time anomaly detection application, built around Apache Kafka, and Cassandra.

Kafka and Cassandra are designed for time-series data, however, it’s not so obvious how they can process geospatial data. In order to find location-specific anomalies, we need a way to represent locations, index locations, and query locations.

We explore alternative geospatial representations including: Latitude/Longitude points… Show more

This project explored how we added location data to a scalable real-time anomaly detection application, built around Apache Kafka, and Cassandra.

Kafka and Cassandra are designed for time-series data, however, it’s not so obvious how they can process geospatial data. In order to find location-specific anomalies, we need a way to represent locations, index locations, and query locations.

We explore alternative geospatial representations including: Latitude/Longitude points, Bounding Boxes, Geohashes, and go vertical with 3D representations, including 3D Geohashes.

For each representation we also explore possible Cassandra implementations including: Clustering columns, Secondary indexes, Denormalized tables, and the Cassandra Lucene Index Plugin.

To conclude we measure and compare the query throughput of some of the solutions, and summarise the results in terms of accuracy vs. performance to answer the question “Which geospatial data representation and Cassandra implementation is best?” Show less

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we… Show more

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day. Show less

Join with me in a journey of exploration upriver with "Kongo", a scalable streaming IoT logistics demonstration application using Apache Kafka, the popular open source distributed streaming platform. Along the way you'll discover: an example logistics IoT problem domain (involving the rapid movement of thousands of goods by trucks between warehouses, with real-time checking of complex business and safety rules from sensor data); an overview of the Apache Kafka architecture and components;… Show more

Join with me in a journey of exploration upriver with "Kongo", a scalable streaming IoT logistics demonstration application using Apache Kafka, the popular open source distributed streaming platform. Along the way you'll discover: an example logistics IoT problem domain (involving the rapid movement of thousands of goods by trucks between warehouses, with real-time checking of complex business and safety rules from sensor data); an overview of the Apache Kafka architecture and components; lessons learned from making critical Kaka application design decisions; an example of Kafka Streams for checking truck load limits; and finish the journey by overcoming final performance challenges and shooting the rapids to scale Kongo on a production Kafka cluster. Show less

At the start of 2018 I spent a month doing an internal evaluation of Apache Ignite. It's a "multi-model" distributed database, persistence, caching and processing ("grid") system. I did some benchmarking of a few use cases (including Java object caching, SQL and self-joins, and Apache Cassandra or Ignite persistence), and wrote an internal report.

"2001: A Space Odyssey" themed Introduction to Apache Cassandra and Spark and Machine Learning! Included creating and connecting to a Cassandra cluster, adding Spark, exploring some real Cassandra performance monitoring data (from around 600 nodes), and trying to predict when a node is likely to have long response times.

Cassandra Cluster Creation in under 10 minutes (1st contact with the monolith)

Consulting Cassandra: Second Contact with the Monolith

Hello Cassandra! A… Show more

"2001: A Space Odyssey" themed Introduction to Apache Cassandra and Spark and Machine Learning! Included creating and connecting to a Cassandra cluster, adding Spark, exploring some real Cassandra performance monitoring data (from around 600 nodes), and trying to predict when a node is likely to have long response times.

Cassandra Cluster Creation in under 10 minutes (1st contact with the monolith)

Consulting Cassandra: Second Contact with the Monolith

Hello Cassandra! A Java Client Example

Third contact with a Monolith – Long Range Sensor Scan - Exploring some real Cassandra monitoring data with Buckets and Materialised Views

Third Contact With a Monolith – Beam Me Down Scotty - using regression analysis to predict Garbage Collections from Heap Use metrics

Third Contact with a Monolith - In the Pod - Initial experiments using Apache SPARK and MLLib to predict Garbage Collections

Fourth Contact with a Monolith - Using DataFrames, ML Pipelines and Scala to predict which Cassandra nodes had increased response times.

Behind the Scenes - how we pre-processed the raw metric data to get it into a format to learn from, using pivot

A Luxury Voyage of (Data) Exploration by Apache Zeppelin - what does the data look like using a data notebook?

Spark Structured Streaming with DataFrames Show less