Skip to content

Configuration of Cluster and Brokers in Apache Kafka, Kafka Connect, Kafka Schema Registry, ksqlDB Server, PostgreSQL and pgAdmin in Docker containers.

Notifications You must be signed in to change notification settings

ali-ramirez/kafka-stack

Repository files navigation

kafka Logo kafka logo kafka logo kafka logo Conduktor logo Postgresql logo

Apache kafka Stack Tools

Basic Cluster and Brokers Environment Configuration for Apache kafka, Kafka Connect, Kafka Schema Registry, ksqlDB-server, Kafka REST Proxy, Conduktor, PostgreSQL, and pgAdmin in Docker containers.

Description

Configuration of Cluster and Brokers in Apache Kafka with connection to kafka Connect, kafka-schema-registry, ksqlDB, kafka REST Proxy, Conduktor, PostgreSql and PgAdmin 4 in Docker containers.

Requirements

Make sure you have the latest versions of Docker and Docker Compose installed on your machine.

Clone this repository or copy the files from this repository into a new folder. In the docker-compose.yml files, you can change the ports (if you are running multiple containers on your system).

With this project, you can quickly execute the following:

Contenido:

Configuration

Edit the .env file to change the default environment variables.

Installation

Open a terminal and run cd to the folder where you can see the files docker-compose-kafka-single-stack.yml, which contains the necessary configuration to deploy a Kafka broker, and the file docker-compose-kafka-multiple-stack.yml, which contains the configuration for multiple Apache Kafka brokers.

Execute as needed:

docker-compose -f docker-compose-kafka-single-stack.yml up -d  

This will start the containers and create volumes according to the configuration in your docker-compose.yml files. Make sure to grant write permissions to the directory so it can create the necessary volumes for the containers. Similarly, remember to clean the volumes folder when running different files to avoid data inconsistency.

  • .env – File containing the environment variable data.
  • volumenes – Folder containing container data.

The containers are already built and running. You should be able to access the installation of each container.

Use

Services

You can visit http:/localhost:8083 to check the status of kafka-connect, or go to http:/localhost:8088 to view the status of ksqldb-server, as well as the other services such as kafka-rest-proxy and kafka-schema-registry.

kafka connect

ksqldb

Conduktor

ou can visit http:/localhost:8080 to access Conduktor. The default username is admin@admin.io, and the password is admin. You can modify these credentials in the environment variables located in the .env file."

conduktor

The Conduktor tool will allow you to view the status of the cluster as well as visualize the brokers you have deployed in Apache Kafka.

conduktor

conduktor

In the Conduktor interface, you will be able to see the status of the connection with your configured and connected services, such as kafka-connect and kafka-schema-registry to Apache Kafka.

conduktor

conduktor

Conduktor will be useful for conducting the necessary tests and visual management of Apache Kafka. You can create new topics and send/receive messages as well.

conduktor

From the Conduktor interface, you can create connectors and schemas, manage them, and test them visually.

conduktor

License

CC0

About

Configuration of Cluster and Brokers in Apache Kafka, Kafka Connect, Kafka Schema Registry, ksqlDB Server, PostgreSQL and pgAdmin in Docker containers.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published