Apache Kafka

-

Apache Kafka was originated at LinkedIn and later became an open sourced Apache project in 2011 

Then First-class Apache project in 2012 

Kafka is written in Scala and Java

Apache Kafka is publish-subscribe based fault tolerant messaging system. 

Apache Kafka is fast, scalable and distributed by design.


What is a Messaging System?

A Messaging System is responsible for transferring data from one application to another

Distributed messaging is based on the concept of reliable message queuing. 

Messages are queued asynchronously between client applications and messaging system. 

Two types of messaging patterns are available − 

1) point to point 

Messages are persisted in a queue. 

One or more consumers can consume the messages in the queue, but a particular message can be consumed by a maximum of one consumer only. 

Once a consumer reads a message in the queue, it disappears from that queue.

2) publish-subscribe (pub-sub) 

message producers are called publishers and message consumers are called subscribers.

messages are persisted in a topic.

consumers can subscribe to one or more topic and consume all the messages in that topic.


Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another.

Kafka is suitable for both offline and online message consumption. 

Kafka messages are persisted on the disk and replicated within the cluster to prevent data loss. 

Kafka is built on top of the ZooKeeper synchronization service. 

Integrates very well with Apache Storm and Spark for real-time streaming data analysis.


Benefits:

    Reliability − Kafka is distributed, partitioned, replicated and fault tolerance.

    Scalability − Kafka messaging system scales easily without down time..

    Durability − Kafka uses Distributed commit log which means messages persists on disk as fast as possible, hence it is durable..

    Performance − Kafka has high throughput for both publishing and subscribing messages. It maintains stable performance even many TB of messages are stored.

Kafka is very fast and guarantees zero downtime and zero data loss.


Use Cases

    1. Metrics
    2. Log Aggregation Solution
    3. Stream Processing

Kafka is very fast, performs 2 million writes/sec. 

Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket.

Kafka is a distributed streaming platform for real-time data with low latency and high throughput.

Install in windows:
https://www.geeksforgeeks.org/installation-guide/how-to-install-and-run-apache-kafka-on-windows/



Comments

Post a Comment

Popular posts from this blog

PL/SQL

-
PL/SQL PL/SQL stands for Procedural Language extensions to the Structured Query Language (SQL). PL/SQL can execute a number of queries in one block using single command. One can create a PL/SQL unit such as procedures, functions, packages, triggers, and types, which are stored in the database for reuse by applications. The PL/SQL engine resides in the Oracle engine. The Oracle engine can process not only single SQL statement but also block of many statements. Structure of PL/SQL Block: DECLARE     declaration statements; BEGIN     executable statements EXCEPTIONS     exception handling statements END; / Example : SET SERVEROUTPUT ON; SQL> DECLARE        var varchar2(40) := 'I love GeeksForGeeks' ;      BEGIN        dbms_output.put_line(var);      END;      / PL/SQL Functions PL/SQL functions are reusable blocks of code that can be used to perform specific tasks. They are...
Read more

JAVA8 Features

-
Java8 Features 1) Lambda Expressions : Concise functional code using ->. Lambda Expression basically expresses an instance of the functional interface, Lambda Expression provides a clear and concise way to represent a method of the functional interface using an expression. 2) Functional Interfaces : Single-method interfaces. An interface that contains only one abstract method is known as a functional interface. There is no restriction, that can have n number of default and static methods inside a functional interface. A functional interface is an interface that contains only one abstract method.  @FunctionalInterface annotation is used to ensure that the functional interface can’t have more than one abstract method. Runnable, ActionListener, and Comparable are some of the examples of functional interfaces.   Runnable –> This interface only contains the run() method.   Comparable –> This interface only contains the compareTo() method.   ActionListener –> ...
Read more

Build Automation

-
ANT: Ant was the first build tool released in 2000. Ant was developed on procedural programming idea. Ant was improved with an ability to accept plug-ins and dependency management over the network, with the help on Apache-IVY. Drawbacks: 1) XML is used as a format to write the build scripts. 2) Being hierarchical is not good for procedural programming, and 3) XML is relatively unmanageable.   MAVEN: Maven was introduced in 2004. Maven relied on the conventions and was able to download the dependencies over the network. Benefits : Life cycle of Maven, while following the same life cycle for multiple projects continuously. Problems: 1) Maven does not handle the conflicts between versions of the same library. 2) Complex customised build scripts are difficult to write in Maven, as compared to writing the build scripts in ANT. GRADLE : Gradle came into picture in 2012 Features  Declarative builds and build-by-convention 1) Gradle is available with separate Domain Specific Language ...
Read more