What Is a Message Queue? Asynchronous Architecture with Kafka and RabbitMQ

You place an order on an e-commerce site. Within seconds, a confirmation email arrives, the shipping company is notified, inventory is updated, and the analytics system records it.

Does all of this happen synchronously inside the same HTTP request? No. If it were designed that way, the order endpoint would take 10 seconds — and if any service crashed, the order would fail too.

The solution: a message queue.

Synchronous vs Asynchronous Communication

In synchronous communication, service A calls service B and waits for a response. Simple but fragile: if B is slow, A is slow; if B crashes, A crashes.

Synchronous:
OrderService → EmailService (waiting...)     2 sec
             → SMSService (waiting...)       1 sec
             → InventoryService (waiting...) 0.5 sec
Total: 3.5 seconds, any failure = entire operation fails

In asynchronous communication, service A leaves a message and moves on. Service B picks up the message when it's ready.

Asynchronous:
OrderService → Queue: "order.placed" (0.01 sec)
← Instant response to user

(In the background, independently)
EmailService     ← received message, sent email
SMSService       ← received message, sent SMS
InventoryService ← received message, updated stock

The order service completed in 10ms. Other services run independently and in parallel. If one crashes, the message waits in the queue.

Core Concepts

Producer: Creates and sends messages to the queue. The order service produces an "order.placed" event.

Consumer: Receives and processes messages from the queue. Email, SMS, inventory services are all consumers.

Queue/Topic: The structure where messages wait. Called a "queue" in RabbitMQ, a "topic" in Kafka.

{
  "eventType": "order.placed",
  "orderId": "ord-789",
  "userId": "usr-123",
  "items": [{ "productId": "prod-456", "quantity": 2, "price": 49.99 }],
  "totalAmount": 99.98
}

RabbitMQ: The Traditional Message Broker

RabbitMQ is a mature message broker using AMQP. Extremely flexible in message routing.

Exchange: Producer sends to an exchange, not directly to a queue. The exchange routes messages based on rules.

import amqp from "amqplib";

async function publishOrderEvent(order) {
    const connection = await amqp.connect("amqp://localhost");
    const channel = await connection.createChannel();
    await channel.assertExchange("order_events", "topic", { durable: true });
    channel.publish(
        "order_events",
        "order.placed",
        Buffer.from(JSON.stringify(order)),
        { persistent: true }
    );
}

async function startEmailConsumer() {
    const connection = await amqp.connect("amqp://localhost");
    const channel = await connection.createChannel();
    await channel.assertExchange("order_events", "topic", { durable: true });
    await channel.assertQueue("email_service_queue", { durable: true });
    await channel.bindQueue("email_service_queue", "order_events", "order.*");
    channel.prefetch(1);

    channel.consume("email_service_queue", async (msg) => {
        if (!msg) return;
        const order = JSON.parse(msg.content.toString());
        try {
            await emailService.sendOrderConfirmation(order);
            channel.ack(msg);
        } catch (error) {
            channel.nack(msg, false, true);
        }
    });
}

Acknowledgment (ACK): After processing, consumer tells broker "received, delete it." If processing fails, nack returns the message to the queue. No data loss.

Kafka: High-Volume Event Streaming

Apache Kafka is not a message queue — it is a distributed event log.

The most critical difference: in Kafka, messages are not deleted after a consumer reads them. Messages are kept on disk for a configurable period (default 7 days). A newly added consumer can also read past messages.

RabbitMQ: Message received → Deleted
Kafka:    Message received → Still on disk (for the retention period)

import { Kafka } from "kafkajs";

const kafka = new Kafka({ clientId: "order-service", brokers: ["kafka:9092"] });
const producer = kafka.producer();

async function publishEvent(topic, event) {
    await producer.connect();
    await producer.send({
        topic,
        messages: [{ key: event.orderId, value: JSON.stringify(event) }]
    });
}

const consumer = kafka.consumer({ groupId: "email-service" });

async function startConsumer() {
    await consumer.connect();
    await consumer.subscribe({ topic: "order-events", fromBeginning: false });
    await consumer.run({
        eachMessage: async ({ message }) => {
            const event = JSON.parse(message.value.toString());
            if (event.type === "order.placed") {
                await emailService.sendConfirmation(event);
            }
        }
    });
}

When to Use Which?

Choose RabbitMQ: Complex routing rules. Lower volume, easier setup. Task queue use case.

Choose Kafka: Very high-volume event streaming. Message history needed (replay, audit). Multiple independent consumers processing the same data for different purposes.

Dead Letter Queue

After a certain number of failed attempts, the message is moved to a Dead Letter Queue (DLQ) where it can be inspected and manually reprocessed.

await channel.assertQueue("order_processing", {
    durable: true,
    arguments: {
        "x-dead-letter-exchange": "dlx",
        "x-message-ttl": 60000,
        "x-max-retries": 3
    }
});

Message queues are one of the most powerful building blocks of distributed systems. They decouple services, provide load balancing, and add fault tolerance. Used correctly, the system becomes both more resilient and more scalable.