HARDCODE

      When designing a system, you should not limit yourself with using only one type of databases. Some business needs might require having a more flexible sort of databases like NoSQL / Document databases.  One of these databases is Elasticsearch ,   which is a topic that I have talked about in a previous post that you should definitely check out. However, if you use more than one database for your system, making sure that all your databases are in sync is crucial . How It Started 🚩 As I said in my previous post . The other sort of problems that we have faced while working on the microservices project is data inconsistency. To explain this problem I have to tell you why the system needed an Elasticsearch index. In microservices projects you usually have more than one database, one for each service/domain. The client-side needed an aggregated document of the data from all the databases. Which in microservices is called The Aggregate Root ,   and that is basically an entity that repr

     People always want the newest things. The more trendy something is, the more it appeals to most people. Same applies to technology. Adding the word microservices to the CV can help it shine a little bit. These are some of the thoughts I was having while I was 2 hours deep into debugging an issue in a project that involved .Net, Kafka, SignalR, React js, Elasticsearch, ECS, MediatR and more. Beware of Microservices  ⚠️ What I'm about to share is a cautionary tale for people who are on the verge of shifting their solution into a microservices architecture. Before you start implementing such an architecture, please read carefully the decisions that others have made along with their consequences so that you can see for yourself the outcome of each design decision instead of falling for the same mistakes. Before anything : The case study that I’m about to discuss might have had its problems. However, the people who made the design decisions are very good software engineers, and I

   In the software engineering industry, the term CQRS is used quite often. It describes a pattern which allows you to handle services in your system in a vertical manner. It gives you the ability to free yourself from any other service in the same domain and allows you to focus on the service that you wish to implement. I know that there are hundreds of other resources that can explain the CQRS pattern better in terms of definition and philosophy. However, in this article I'm going to explain it in another way, a rather practical way. Imagine you have an entity named Customer . Normally if we follow clean architecture , we will have a single horizontally scaled service called CustomerService . In this CustomerService there are all types of services that perform different operations on this entity. However not all services need to operate on the whole entity model. This is why Data Transfer Objects exist. Data Transfer Objects (DTOs) are a simplified versions of entity models wh

  Confluent has built some useful tools that rely on Kafka . One of these tools is Kafka Connect . And as I have shown you before  one of the ways to use Kafka Connect as a consumer of Kafka Topics, I will show you how to use Kafka Connect as a producer to Kafka topics Kafka Connect offers a very powerful feature which is connecting to databases using certain connectors as JDBC (Java Database Connectivity).  The Kafka Connect JDBC source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka topic. This connector supports a wide variety of databases. Data is loaded by periodically executing a SQL query and creating an output record for each row in the result set. It enables you to pull data (source) from a database into Kafka , and also push data (sink) from a Kafka topic to a database. In this tutorial we'll be focusing on pulling data from a SQL database. I will be reusing the same setup as my previous Kafka Connect post so

 Suppose you need to store all the messages published on a certain topic. Normally you would create a consumer that listens to that topic, receives the message then stores it in the dataset that you want. What if I told you that Confluent has created a shortcut. Instead of doing this, you can just create a Kafka connector using Confluent Kafka Connect and this connector will automatically store any published message in your database. Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka . Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. It can also deliver data from Kafka topics into secondary indexes like Elasticsearch or batch systems such as Hadoop for offline analysis. In this tutorial we wil

 There are two keywords that you must understand when publishing a message: latency and throughput.  Throughput is a measure of how much data can be processed in a given amount of time. It's usually measured in bits per second (bit/s), or data packets per second. High throughput means the system can process a large amount of data quickly, which is often desirable in high-load scenarios. For example, if a Kafka producer can send 1000 messages per second to a broker, the throughput is 1000 messages per second. Latency, on the other hand, is a measure of time delay experienced in a system, the time it takes for a bit of data to travel from one point to another in a network. It is usually measured in milliseconds. Low latency means that data can be transferred quickly from source to destination. For example, if a message takes 10 milliseconds from the time it's sent by a Kafka producer until it's received by a broker, the latency is 10 milliseconds. In all systems, there's