Skip to content

apache/pulsar-connectors

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

513 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Apache Pulsar Connectors

This repository contains the IO connectors for Apache Pulsar.

Connectors are packaged as NAR files and can be deployed into any Pulsar installation by placing them in the connectors directory or mounting them into the apachepulsar/pulsar Docker image.

Available Connectors

Sources

Connector Description
Canal MySQL binlog via Alibaba Canal
Debezium (MySQL, PostgreSQL, MongoDB, MSSQL, Oracle) CDC via Debezium
DynamoDB Amazon DynamoDB Streams
File Local filesystem
Kafka Apache Kafka
Kinesis Amazon Kinesis Data Streams
MongoDB MongoDB change streams
NSQ NSQ messaging
RabbitMQ RabbitMQ / AMQP

Sinks

Connector Description
Aerospike Aerospike database
Alluxio Alluxio distributed storage
Azure Data Explorer Azure Data Explorer (Kusto)
Cassandra Apache Cassandra
Elasticsearch / OpenSearch Elasticsearch and OpenSearch
HBase Apache HBase
HDFS3 Hadoop HDFS
HTTP HTTP endpoint
InfluxDB InfluxDB time-series database
JDBC (PostgreSQL, MariaDB, ClickHouse, SQLite, OpenMLDB) JDBC databases
Kafka Apache Kafka
Kinesis Amazon Kinesis Data Streams
MongoDB MongoDB
Redis Redis
Solr Apache Solr

Adaptor

Connector Description
Kafka Connect Adaptor Run Kafka Connect connectors on Pulsar

Prerequisites

Note: This project includes a Gradle Wrapper so no separate Gradle installation is needed. Use ./gradlew on Linux/macOS and gradlew.bat on Windows.

Building

Compile and assemble all modules:

./gradlew assemble

NAR files are produced under each connector's build/libs/ directory.

Build a specific connector:

./gradlew :elastic-search:assemble

Build the distribution package containing all connector NARs:

./gradlew :distribution:pulsar-io-distribution:assemble

Check source code license headers:

./gradlew rat spotlessCheck

Auto-fix license headers:

./gradlew spotlessApply

Running Tests

# All unit tests
./gradlew test

# Specific connector
./gradlew :elastic-search:test

# Specific test class
./gradlew :elastic-search:test --tests "ElasticSearchSinkTests"

Using Connectors

With Docker

Mount connector NARs into the Pulsar container:

docker run -v /path/to/connectors:/pulsar/connectors apachepulsar/pulsar

Manual Installation

Copy NAR files to the connectors/ directory of your Pulsar installation:

cp elastic-search/build/libs/pulsar-io-elastic-search-*.nar $PULSAR_HOME/connectors/

Versioning

This repository follows its own release cadence, independent from Apache Pulsar releases. All connectors are released together as a single release. The initial release version matches the Pulsar version at the time of the split.

Each release specifies which Pulsar versions it is compatible with. The pulsar-io-core API has been stable for years, so connectors are generally compatible across Pulsar 4.x releases.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md in the main Pulsar repository for guidelines.

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

About

Apache Pulsar Connectors - source and sink connectors for Apache Pulsar

Topics

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages