HARDCODE

       Imagine going out to buy bread. And you know that there are two types of bakeries. One that gives you the bread right away because they know exactly when it's going to be done. The other kind takes the order from you and tells you to go home and it will deliver the bread to you once it's ready because it's unclear for them when it's going to be done. Then it's clear for you what to expect when dealing with both sorts of bakeries. Now, imagine if there were a third type of bakeries. One that tells you that it's going to give it to you right away while it's actually unclear for them when the bread is going to done. So, you expect the bakery to behave in a certain way but it will actually behave in another. Well, imagine working in such a bakery. Eventual 📨 VS Transactional⚡ In my humble opinion, the problem that I'm discussing in this article is probably the most critical one when designing a system. It's a problem about the identity of your so...

      When designing a system, you should not limit yourself with using only one type of databases. Some business needs might require having a more flexible sort of databases like NoSQL / Document databases.  One of these databases is Elasticsearch ,   which is a topic that I have talked about in a previous post that you should definitely check out. However, if you use more than one database for your system, making sure that all your databases are in sync is crucial . How It Started 🚩 As I said in my previous post . The other sort of problems that we have faced while working on the microservices project is data inconsistency. To explain this problem I have to tell you why the system needed an Elasticsearch index. In microservices projects you usually have more than one database, one for each service/domain. The client-side needed an aggregated document of the data from all the databases. Which in microservices is called The Aggregate Root ,   and that is ...

     People always want the newest things. The more trendy something is, the more it appeals to most people. Same applies to technology. Adding the word microservices to the CV can help it shine a little bit. These are some of the thoughts I was having while I was 2 hours deep into debugging an issue in a project that involved .Net, Kafka, SignalR, React js, Elasticsearch, ECS, MediatR and more. Beware of Microservices  ⚠️ What I'm about to share is a cautionary tale for people who are on the verge of shifting their solution into a microservices architecture. Before you start implementing such an architecture, please read carefully the decisions that others have made along with their consequences so that you can see for yourself the outcome of each design decision instead of falling for the same mistakes. Before anything : The case study that I’m about to discuss might have had its problems. However, the people who made the design decisions are very good software e...

  Confluent has built some useful tools that rely on Kafka . One of these tools is Kafka Connect . And as I have shown you before  one of the ways to use Kafka Connect as a consumer of Kafka Topics, I will show you how to use Kafka Connect as a producer to Kafka topics Kafka Connect offers a very powerful feature which is connecting to databases using certain connectors as JDBC (Java Database Connectivity).  The Kafka Connect JDBC source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka topic. This connector supports a wide variety of databases. Data is loaded by periodically executing a SQL query and creating an output record for each row in the result set. It enables you to pull data (source) from a database into Kafka , and also push data (sink) from a Kafka topic to a database. In this tutorial we'll be focusing on pulling data from a SQL database. I will be reusing the same setup as my p...

 Suppose you need to store all the messages published on a certain topic. Normally you would create a consumer that listens to that topic, receives the message then stores it in the dataset that you want. What if I told you that Confluent has created a shortcut. Instead of doing this, you can just create a Kafka connector using Confluent Kafka Connect and this connector will automatically store any published message in your database. Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka . Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. It can also deliver data from Kafka topics into secondary indexes like Elasticsearch or batch systems such as Hadoop for offline analysis. In this tutorial we...

When we talk about Kafka it's important to mention Avro serialization. Like I mentioned in my Kafka introductory post : as a message broker, Kafka is quite known for being complex as opposed to some other brokers such as SQS (which is literally called Simple Queue Service) or RabbitMQ which calls its producers Basic Produce and consumers Basic Consume . And although it requires a little more effort than the other brokers, it offers much more for services which need to trim down every inch of latency they can, due to the criticality of the business of the sheer volume of data it requires to stream. And Confluent Kafka can help with that by transporting Avro serialized messages. But in order to do that, Kafka relies on Confluent   Schema Registry . Because after it gets serialized to bytes, it needs to look for the schema that can help these bytes go back to the model that it initially was. Confluent Schema Registry is basically a store that keeps track of the schemas of the mess...

   In my previous post  Introduction to Elasticsearch: Create, Update, Delete and Search Documents 🔍  I showed you how to set up an Elasticsearch index and manage its data. But as you could see in that post the data looked very ugly on the cmd window. That's because this is not Elasticsearch 's job. Data representation and visualization is key to understanding your data and even predicting its trend. And as I previously said, the power in Elasticsearch lies in its integration with other powerful tools such as our guest of honor: Kibana . Kibana , developed by Elastic , is a powerful open-source data visualization and exploration platform. It seamlessly integrates with Elasticsearch, making it an essential component of the Elastic Stack. Whether you’re a data analyst, developer, or business user, Kibana empowers you to unlock valuable insights from your data. With its intuitive interface, you can create interactive dashboards, explore logs, analyze metrics, and visu...