HARDCODE

 There are two keywords that you must understand when publishing a message: latency and throughput.  Throughput is a measure of how much data can be processed in a given amount of time. It's usually measured in bits per second (bit/s), or data packets per second. High throughput means the system can process a large amount of data quickly, which is often desirable in high-load scenarios. For example, if a Kafka producer can send 1000 messages per second to a broker, the throughput is 1000 messages per second. Latency, on the other hand, is a measure of time delay experienced in a system, the time it takes for a bit of data to travel from one point to another in a network. It is usually measured in milliseconds. Low latency means that data can be transferred quickly from source to destination. For example, if a message takes 10 milliseconds from the time it's sent by a Kafka producer until it's received by a broker, the latency is 10 milliseconds. In all systems, there's ...

 People say that Kafka is a dumb broker. It just holds data under some defined topics and forward messages from producers to consumers. It doesn't do much processing on the messages, doesn't route them based on content, and doesn't transform them. But that actually isn't entirely true. Kafka does much more than just storing messages. It keeps count of which consumer group consumed which message by tracking its offset. The consumer group is the group Id you give your consumer once you initialize it. Having more than one consumer helps you avoid consumer failures and scale better. If a message is consumed by one consumer in a group, no other consumers in the same group will receive it. However, all other consumers with other group Ids will receive the message. It's important to note that the offset of each message is basically the id of the message related to the group. So, if "message A" was published on a brand-new group its offset will be 0. But if anoth...

When we talk about Kafka it's important to mention Avro serialization. Like I mentioned in my Kafka introductory post : as a message broker, Kafka is quite known for being complex as opposed to some other brokers such as SQS (which is literally called Simple Queue Service) or RabbitMQ which calls its producers Basic Produce and consumers Basic Consume . And although it requires a little more effort than the other brokers, it offers much more for services which need to trim down every inch of latency they can, due to the criticality of the business of the sheer volume of data it requires to stream. And Confluent Kafka can help with that by transporting Avro serialized messages. But in order to do that, Kafka relies on Confluent   Schema Registry . Because after it gets serialized to bytes, it needs to look for the schema that can help these bytes go back to the model that it initially was. Confluent Schema Registry is basically a store that keeps track of the schemas of the mess...

 Message brokers come in all colors. Each broker has its own edge. There are some brokers which are aimed to be simple and direct such as Amazon SQS (which is literally called Simple Queue Service). Some of them can be used in a simple fashion, but can also be used to implement complex patterns like RabbitMQ which is a topic I talked about in my previous posts . And then there is a broker that was designed to: handle heavy-duty streaming, maximize efficiency, allow you to scale up, serialize messages to bytes, partition messages and much more. That broker is indeed Apache Kafka . Due to its wide array of features, Kafka can be overwhelming. And sometimes it feels too exhausting to start comprehending its principles and what the broker can provide. So, why not strip down the broker from all of its extra shining features and just start from its core features like publishing and consuming simple JSON serialized string messages. What we're going to explore isn't just Apache Kafk...

    Logstash did a very impressive job transforming our logs into documents that allowed us to understand and visualize how our applications are behaving. But, because of that Logstash could require quite the memory and CPU to run. So, it's not the most efficient move to make Logstash collect the data from various sources. Elastic offered a solution to this concern and introduced Beats . Beats  is a lightweight shipper for forwarding and centralizing log data. It's installed as an agent on your servers to capture all sorts of operational data like logs or network packet data. Beats is great for gathering data and works efficiently with a large number of files. It can also handle back pressure (when Logstash is busy) and ensures that no data is lost during such periods. And to be clear Logstash can do most of what Beats . So, why use Beats instead? 1. Lightweight Data Shipping : Beats is designed to be lightweight and requires fewer resources than Logstash . This makes it ...

 What's the first thing you imagine when you hear the word logs . I bet you pictured a huge amount of text that's hard to read and it just gives you a headache every time you try to tell the lines apart. And if you're looking for an error or something, it feels like you're looking for a needle in a haystack. So, how can you transform these logs from clunky text which is hard to read into documents in your Elasticsearch index? The answer lies in the letter L in the ELK stack. We've talked about the E , we've talked about the K , now it's time to talk about L : Logstash . Logstash is a powerful open-source data processing pipeline tool that collects data, transforms it into a common format, and sends it to a destination for storage or further analysis. And as I previously pointed out in the previous posts: the great advantage of using this stack together is the seamless integration that allows you to save the effort of integration. You can look at Logstash a...