Kafka acts as a broker for messages, which is why it’s classified as a distributed event streaming platform (often loosely called a message broker, though it’s more powerful than traditional ones).

Idea

The core idea behind Kafka was simple:

Prevent servers from collapsing under massive load by decoupling data producers and consumers.

Instead of services directly talking to each other (tight coupling), Kafka introduces a middle layer that stores and distributes events efficiently.

It was engineered by LinkedIn in 2010 to handle high-throughput, real-time data ingestion and event processing—because existing systems couldn’t handle their scale.

It was developed by Jay Kreps, Jun Rao, and Neha Narkhede, open-sourced in 2011, and became a top-level Apache Software Foundation project in 2012 as Apache Kafka.

Kafka eventually became the de facto standard for high-throughput, low-latency event streaming—powering logs, analytics pipelines, microservices communication, and real-time systems globally.

But Why Kafka Was Needed?

Before Kafka, systems looked like this:

• Service A → directly calls Service B

• Service B → directly calls Service C

Problems:

• Tight coupling

• System crashes under load

• No buffering mechanism

• Hard to scale

Kafka solves this by acting as a central pipeline:

Producers → Kafka → Consumers

Now:

• Producers just send data

• Consumers read when ready

• System becomes asynchronous and resilient

Core Principles of Kafka

When Kafka was designed, it relied on a few foundational assumptions:

1. High Throughput Over Low Latency

Kafka is optimised to handle millions of messages per second efficiently.

2. Sequential Disk Writes

Instead of random writes, Kafka appends messages to logs:

• Faster disk I/O

• Better performance

3. Immutable Logs

Once written, messages:

• Are not modified

• Are only appended and read

4. Pull-Based Consumption

Consumers:

• Pull data at their own pace

• Avoid overload

5. Partition-Based Scaling

Topics are divided into partitions:

• Enables parallel processing

• Horizontal scalability

Kafka Core Components

1. Producer

• Sends messages to Kafka

• Example: a backend service sending user activity

2. Broker

• Kafka server that stores data

• Handles read/write requests

3. Topic

• Logical category of messages

• Example: user-signups, payments

4. Partition

• Subdivision of a topic

• Enables scaling and parallelism

5. Consumer

• Reads messages from Kafka

6. Consumer Group

• Multiple consumers working together

• Each partition is consumed by only one consumer in a group

How Kafka Works (Simple Flow)

1. Producer sends message to a topic

2. Kafka stores it in a partition (append-only log)

3. Consumer reads from that partition using an offset

4. Message stays in Kafka for a configured retention period

Why Kafka Became So Popular

• Handles real-time data streams

• Fault-tolerant and distributed

• Scales horizontally

• Decouples services

• Works as a backbone for event-driven architecture

Real-World Use Cases

• Logging systems

• Real-time analytics

• Event-driven microservices

• Fraud detection systems

• Streaming pipelines (ETL)

The Problem: Kafka at Extreme Scale

Kafka works incredibly well—but 15 years later, its original assumptions are being pushed to the limit at LinkedIn-scale systems.

We’re talking about:

• Trillions of daily messages

• Multi-region deployments

• Terabytes of metadata

• Highly dynamic workloads needing auto-rebalancing

At this scale, new challenges emerge:

1. Metadata Bottlenecks

Kafka relies heavily on cluster metadata:

• Becomes large and complex

• Hard to manage efficiently

2. Rebalancing Issues

When consumers join/leave:

• Kafka needs to rebalance partitions

• Causes latency spikes

3. Operational Complexity

Running Kafka clusters at massive scale:

• Requires heavy tuning

• Complex infrastructure management

4. Multi-Region Limitations

Cross-region replication:

• Adds latency

• Hard to maintain consistency

LinkedIn’s Shift: NorthGuard

To address these challenges, LinkedIn is rethinking event streaming systems entirely with a new system called NorthGuard.

What is NorthGuard?

NorthGuard is LinkedIn’s next-generation event streaming architecture designed to:

• Handle extreme scale more efficiently

• Reduce operational overhead

• Improve elasticity and rebalancing

• Better support multi-region systems

What’s Changing?

1. Dynamic Scaling

Kafka assumes relatively stable workloads.

NorthGuard:

• Adapts dynamically

• Handles fluctuating traffic seamlessly

2. Improved Metadata Management

Instead of massive centralized metadata:

• More efficient distribution

• Better scalability

3. Faster Rebalancing

Kafka rebalancing is expensive.

NorthGuard:

• Aims for near-zero disruption

• Faster partition movement

4. Cloud-Native Thinking

Kafka was designed pre-cloud era.

NorthGuard:

• Built with modern distributed systems + cloud infra in mind

Key Takeaway

Kafka is still incredibly powerful and widely used—but:

At extreme scale, even great systems need evolution.

Kafka solved:

• Decoupling

• High-throughput streaming

• Fault tolerance

NorthGuard is solving:

• Elastic scaling

• Massive metadata

• Global distribution challenges

JavaScript was originally designed to run inside browsers, handling user interactions like clicks and form submissions. When Node.js brought JavaScript to the server, it introduced a challenge:

How can a single-threaded environment handle thousands of requests efficiently?

The answer lies in one of Node.js’s most important concepts — the Event Loop.

The Single-Thread Limitation

Unlike traditional backend technologies (like Java or PHP), Node.js runs on a single thread.

This means:

• Only one piece of code executes at a time

• No parallel execution like multi-threaded systems

At first glance, this sounds like a bottleneck.

If one task blocks the thread (e.g., reading a file or calling an API), everything else would have to wait.

So how does Node.js still handle thousands of users?

👉 This is exactly why the event loop exists.

What is the Event Loop?

The event loop is essentially a task manager.

It continuously checks:

1. Is the main thread free?

2. Are there any tasks waiting to be executed?

If both conditions are satisfied, it takes a task from a queue and executes it.

Simple Analogy

Think of it like a restaurant waiter:

• The waiter (event loop) takes orders (tasks)

• Sends them to the kitchen (background workers)

• Comes back to serve completed dishes (callbacks)

The waiter doesn’t cook — just manages flow efficiently.

Why Node.js Needs the Event Loop

Because Node.js is single-threaded:

• It cannot afford to block execution

• It must delegate slow operations

So instead of waiting:

• Node.js offloads tasks like file reading, database queries, or API calls

• Continues handling other requests

• Comes back when the task is done

Without the event loop:

• Every request would block others

• Performance would collapse under load

Call Stack vs Task Queue (Conceptual)

To understand the event loop, you need two core components:

1. Call Stack

• Where code is executed

• Works in LIFO (Last In, First Out) manner

• Only one function runs at a time

Example:

main() → fetchData() → processData()

2. Task Queue (Callback Queue)

• Stores completed async tasks

• Works in FIFO (First In, First Out)

How the Event Loop Connects Them

The event loop continuously:

1. Checks if the call stack is empty

2. If yes → takes the first task from the queue

3. Pushes it into the call stack

How Async Operations are Handled

Let’s say you write:

setTimeout(() => {
  console.log("Hello after 2 seconds");
}, 2000);

What happens internally?

1. Timer is registered

2. Node.js does not wait

3. Timer runs in background

4. After 2 seconds → callback goes to task queue

5. Event loop picks it when stack is free

👉 This is non-blocking behavior

Timers vs I/O Callbacks (High-Level)

Node.js handles different types of async operations differently.

Timers

• Functions like setTimeout, setInterval

• Scheduled after a certain delay

Example:

setTimeout(() => console.log("Timer done"), 1000);

I/O Callbacks

• File system operations

• Network requests

• Database queries

Example:

fs.readFile("file.txt", () => {
  console.log("File read complete");
});

Key Difference

Both eventually: ➡️ Push callbacks to the queue ➡️ Event loop executes them

Event Loop and Scalability

This is where Node.js shines.

Traditional Model (Blocking)

• One thread per request

• High memory usage

• Limited scalability

Node.js Model (Event Loop)

• Single thread

• Non-blocking I/O

• Handles thousands of concurrent requests

Why It Works

Because Node.js:

• Doesn’t wait for slow operations

• Keeps the thread free

• Uses the event loop to manage execution

👉 Result: High throughput with minimal resources

Real-World Example

Imagine a server handling 3 requests:

1. Fetch user data (DB call)

2. Read file

3. Send response

Traditional Server

• Request 1 blocks everything

• Others wait

Node.js with Event Loop

• Delegates DB + file tasks

• Immediately handles next request

• Executes callbacks when ready

👉 All requests are handled efficiently without blocking

Key Takeaways

• Node.js is single-threaded, but not slow

• The event loop enables concurrency

• It works by:

  • Monitoring the call stack

  • Managing the task queue

• Async operations are:

  • Delegated

  • Returned via callbacks

• This model makes Node.js highly scalable

Let’s break it down in a simple, practical way.

🚧 What is Blocking Code?

Blocking code means: 👉 The program waits for a task to finish before moving to the next line.

Example (Synchronous / Blocking)

const fs = require("fs");

const data = fs.readFileSync("file.txt", "utf-8");
console.log(data);

console.log("Next line");

What happens here:

1. Node starts reading the file

2. It pauses everything else

3. Only after reading completes → it prints "Next line"

❗ Problem

If the file is large or slow to access:

• The entire server is stuck

• No other user request can be processed

⚡ What is Non-Blocking Code?

Non-blocking code means: 👉 The program does not wait — it moves forward and handles the result later.

Example (Asynchronous / Non-Blocking)

const fs = require("fs");

fs.readFile("file.txt", "utf-8", (err, data) => {
  console.log(data);
});

console.log("Next line");

What happens here:

1. File read starts in background

2. Node immediately moves on

3. "Next line" prints first

4. File result prints later

🧠 Simple Analogy

Think of it like ordering food:

Blocking

• You go to a restaurant

• Stand at the counter until your food is ready

• Do nothing else

Non-Blocking

• You order food

• Sit with friends / do other work

• Get notified when food is ready

👉 Node.js works like the second scenario.

🐢 Why Blocking Code Slows Down Servers

Node.js uses a single-threaded event loop.

That means:

• Only one main thread handles all requests

• If one request blocks → everything else waits

Real Impact:

If 100 users hit your server:

• Blocking → requests handled one-by-one ❌

• Non-blocking → requests handled concurrently ✅

🔄 Async Operations in Node.js

Node.js is designed around asynchronous (async) operations, such as:

• File system operations (fs)

• Database queries

• API calls

• Network requests

These tasks are:

• Delegated to background workers

• Returned via callbacks, promises, or async/await

📂 Real-World Example: File Handling

Blocking Version

app.get("/", (req, res) => {
  const data = fs.readFileSync("largeFile.txt", "utf-8");
  res.send(data);
});

🔴 Problem:

• Each request waits for file read

• Server becomes slow under load

Non-Blocking Version

app.get("/", (req, res) => {
  fs.readFile("largeFile.txt", "utf-8", (err, data) => {
    res.send(data);
  });
});

🟢 Advantage:

• Server continues handling other requests

• Much better performance

🗄️ Real-World Example: Database Calls

Blocking (Bad Practice)

const user = db.getUserSync(id);

Non-Blocking (Correct)

db.getUser(id).then(user => {
  console.log(user);
});

or

const user = await db.getUser(id);

👉 Database operations are slow — always keep them async.

⚖️ Blocking vs Non-Blocking Summary

🎯 When (If Ever) to Use Blocking?

Blocking code is acceptable when:

• Running startup scripts

• CLI tools

• One-time operations (not handling users)

👉 Never use blocking code in production servers.

🚀 Final Takeaway

• Node.js shines because of non-blocking architecture

• Blocking code defeats its purpose

• Always prefer:

  • Callbacks

  • Promises

  • async/await

👉 If your server feels slow, chances are you're accidentally blocking the event loop.

This guide breaks it down in a simple, practical way.

🚀 What is a REST API?

A REST API (Representational State Transfer) is a way for different systems (usually a client and a server) to communicate over HTTP.

Think of it like this:

• The client (browser, mobile app) sends a request

• The server processes it

• The server sends back a response

👉 Example:

• You open Instagram → app requests data → server sends posts

So essentially:

APIs are the communication bridge between client and server.

📦 What are Resources in REST?

In REST architecture, everything is treated as a resource.

A resource is simply:

Any data you want to expose via your API

Examples:

• Users → /users

• Products → /products

• Orders → /orders

Each resource is accessed via a URL endpoint.

🔧 HTTP Methods (Core of REST)

REST APIs use HTTP methods to define actions on resources.

1. GET → Fetch Data

Used to retrieve data from the server.

GET /users

👉 Fetch all users

2. POST → Create Data

Used to create a new resource.

POST /users

👉 Create a new user

3. PUT → Update Data

Used to update an existing resource (usually full update).

PUT /users/1

👉 Update user with ID = 1

4. DELETE → Remove Data

Used to delete a resource.

DELETE /users/1

👉 Delete user with ID = 1

📊 Status Codes Basics

Status codes tell the client what happened with the request.

Common ones you should know:

• 200 OK → Request successful

• 201 Created → Resource created successfully

• 400 Bad Request → Client error

• 404 Not Found → Resource doesn’t exist

• 500 Internal Server Error → Server failed

👉 Example:

res.status(200).json(users);

🛣️ Designing RESTful Routes

A good REST API follows clean and predictable naming conventions.

Rules:

• Use nouns, not verbs

• Keep URLs plural

• Use IDs for specific resources

👨‍💻 Example: Users Resource

Let’s design a simple users API using Express.js.

Setup

const express = require("express");
const app = express();

app.use(express.json());

Routes

Get all users

app.get("/users", (req, res) => {
  res.status(200).json({ message: "Get all users" });
});

Get single user

app.get("/users/:id", (req, res) => {
  res.status(200).json({ message: `Get user ${req.params.id}` });
});

Create user

app.post("/users", (req, res) => {
  res.status(201).json({ message: "User created" });
});

Update user

app.put("/users/:id", (req, res) => {
  res.status(200).json({ message: `User ${req.params.id} updated` });
});

Delete user

app.delete("/users/:id", (req, res) => {
  res.status(200).json({ message: `User ${req.params.id} deleted` });
});

🧠 Key Design Principles (Keep This in Mind)

• Keep routes consistent and predictable

• Use correct HTTP methods

• Return meaningful status codes

• Separate logic (routes vs controllers in real apps)

• Think in terms of resources, not actions

🔚 Final Thoughts

REST API design is not about complexity — it's about clarity and consistency.

If your API:

• Uses proper HTTP methods

• Has clean route naming

• Returns correct status codes

👉 You're already ahead of most beginners.

Once you're comfortable with this, the next step is:

• Adding authentication (JWT)

• Connecting databases (MongoDB)

• Structuring scalable backend systems

In this article, we’ll break down why Node.js is considered fast, not through benchmarks, but by understanding how it behaves under the hood.

🚀 What Makes Node.js Fast?

Node.js is fast because of its architecture, not magic.

At its core:

• It uses the V8 JavaScript engine (same as Chrome)

• It follows a non-blocking, asynchronous execution model

• It is built on an event-driven architecture

• It runs on a single-threaded event loop

Instead of handling one request at a time (like traditional servers), Node.js can handle thousands of concurrent requests efficiently.

🔄 Understanding Non-Blocking I/O

Blocking (Traditional Approach)

Imagine a server handling requests like this:

1. User requests data

2. Server queries database

3. Server waits for response

4. Only then handles the next request

👉 Problem: While waiting, the server is idle but blocked

Non-Blocking (Node.js Approach)

Node.js does something smarter:

1. User requests data

2. Server sends database request

3. Moves on to handle other requests immediately

4. When data arrives → callback executes

👉 Result: No waiting. The server stays busy and efficient

🍽️ Restaurant Analogy (Async Handling)

Think of a restaurant:

❌ Blocking Model

• One waiter

• Takes one order

• Goes to kitchen

• Waits until food is ready

• Serves it

• Then takes next order

Slow. Inefficient.

✅ Node.js Model

• One waiter

• Takes orders from multiple tables

• Hands them to kitchen

• Keeps taking more orders

• Serves food when ready

Efficient. Scalable.

👉 Node.js is like the second waiter.

⚡ Event-Driven Architecture

Node.js operates on events.

Instead of continuously checking for updates, it reacts when something happens.

Examples of events:

• HTTP request received

• File read completed

• Database response returned

How it works:

• Events are placed in a queue

• The event loop processes them one by one

• Associated callback functions execute

👉 This avoids unnecessary CPU usage and improves responsiveness.

🧵 Single-Threaded Model (But Still Concurrent)

At first glance, this sounds like a limitation.

“Single-threaded? Then how is it fast?”

Here’s the key:

• Node.js uses one main thread

• But delegates heavy work (I/O, file system, network) to background systems

• The main thread only handles coordination

👉 So instead of multiple threads competing, Node.js:

• Avoids thread overhead

• Avoids context switching

• Remains lightweight

🔁 Concurrency vs Parallelism (Simple Explanation)

• Concurrency = Handling many tasks at once (not necessarily simultaneously)

• Parallelism = Running multiple tasks at the exact same time

Node.js:

• Focuses on concurrency

• Uses async operations to handle multiple users efficiently

Example:

• 100 users request data

• Node.js doesn’t process one-by-one

• It manages all requests together without blocking

📍 Where Node.js Performs Best

Node.js shines in scenarios that are:

1. I/O Heavy Applications

• APIs

• Database-driven apps

• File uploads/downloads

2. Real-Time Applications

• Chat apps

• Live notifications

• Gaming servers

3. Streaming Applications

• Video/audio streaming

• Data streaming pipelines

4. Microservices Architecture

• Lightweight services

• Fast communication between services

❗ Where Node.js is NOT Ideal

For balance:

Node.js is not best for CPU-heavy tasks like:

• Image processing

• Machine learning computations

• Complex data transformations

👉 Because heavy computation blocks the single thread.

🏢 Real-World Companies Using Node.js

Many large-scale companies rely on Node.js:

• Netflix → Streaming backend

• LinkedIn → Backend services

• Uber → Real-time ride system

• PayPal → Payment processing APIs

• Walmart → High-traffic e-commerce systems

👉 These companies chose Node.js for scalability and performance under load

🔍 Blocking vs Non-Blocking (Quick Comparison)

🧠 Final Thoughts

Node.js is fast not because it executes code faster, but because it handles work smarter.

Its non-blocking, event-driven architecture allows it to:

• Handle massive concurrent traffic

• Stay responsive under load

• Use system resources efficiently

If your application involves real-time communication, APIs, or high concurrency, Node.js is one of the best choices available.

When building backend applications using Express.js, one of the most important concepts you’ll encounter is middleware. Middleware is what gives Express its flexibility and power.

At a high level, middleware acts like a checkpoint system between a client request and the server response.

What is Middleware in Express?

Middleware is simply a function that has access to:

• req (request object)

• res (response object)

• next (function to pass control)

Basic Syntax

function middleware(req, res, next) {
  console.log("Middleware executed");
  next();
}

👉 Middleware can:

• Modify request/response

• End the request-response cycle

• Call the next middleware

Where Middleware Sits in the Request Lifecycle

Think of Express as a pipeline:

Client Request → Middleware → Middleware → Route Handler → Response

Each middleware processes the request step by step before reaching the final route handler.

Example Flow

app.use((req, res, next) => {
  console.log("Step 1");
  next();
});

app.use((req, res, next) => {
  console.log("Step 2");
  next();
});

app.get("/", (req, res) => {
  res.send("Final Response");
});

Execution Order

Step 1 → Step 2 → Route Handler → Response

Types of Middleware in Express

1. Application-Level Middleware

These are bound to the entire app using app.use().

app.use((req, res, next) => {
  console.log("App-level middleware");
  next();
});

✔ Runs on every request (unless restricted by path)

2. Router-Level Middleware

Used with Express routers (express.Router()).

const router = express.Router();

router.use((req, res, next) => {
  console.log("Router middleware");
  next();
});

app.use("/api", router);

✔ Only runs for specific route groups (/api here)

3. Built-in Middleware

Express provides some ready-made middleware:

• express.json() → Parses JSON body

• express.urlencoded() → Parses form data

• express.static() → Serves static files

app.use(express.json());

✔ Saves time—no need to write common logic manually

Execution Order of Middleware

Middleware executes in the order it is defined.

app.use((req, res, next) => {
  console.log("First");
  next();
});

app.use((req, res, next) => {
  console.log("Second");
  next();
});

Output:

First
Second

⚠️ Important: If next() is not called, the request will hang.

Role of next() Function

next() is what moves the request forward in the pipeline.

Without next()

app.use((req, res, next) => {
  console.log("Stops here");
});

❌ Request never reaches route handler → client hangs

With next()

app.use((req, res, next) => {
  console.log("Continue");
  next();
});

✔ Execution continues

Ending Response in Middleware

app.use((req, res, next) => {
  res.send("Response from middleware");
});

✔ No need to call next() because response is already sent

Real-World Examples of Middleware

1. Logging Middleware

app.use((req, res, next) => {
  console.log(`\({req.method} \){req.url}`);
  next();
});

✔ Helps track incoming requests

2. Authentication Middleware

function auth(req, res, next) {
  const isLoggedIn = true;

  if (!isLoggedIn) {
    return res.status(401).send("Unauthorized");
  }

  next();
}

app.get("/dashboard", auth, (req, res) => {
  res.send("Welcome to dashboard");
});

✔ Protects routes

3. Request Validation Middleware

function validate(req, res, next) {
  if (!req.body.name) {
    return res.status(400).send("Name is required");
  }
  next();
}

app.post("/user", validate, (req, res) => {
  res.send("User created");
});

✔ Ensures clean and valid input

Key Takeaways

• Middleware = functions between request and response

• They run sequentially

• next() controls flow

• You can stack multiple middleware

• Used for logging, auth, validation, parsing, etc.

Final Analogy

Think of middleware like security checkpoints at an airport:

• Passport check → Security scan → Boarding gate

• Each step verifies something before allowing you forward

Similarly:

Request → Middleware checks → Final Response

Conclusion

Middleware is the backbone of Express applications. Once you understand how middleware flows and how next() works, you gain fine-grained control over request handling.

Mastering middleware is essential for building scalable, maintainable backend systems.

When building real-world backend systems, handling file uploads is unavoidable—profile pictures, documents, PDFs, etc. But unlike JSON data, file uploads require a different handling mechanism. This is where middleware like Multer becomes essential.

Why File Uploads Need Middleware

In a typical Express.js application running on Node.js, request bodies are usually parsed using middleware like express.json() or express.urlencoded().

However, file uploads are sent using a special encoding type:

👉 multipart/form-data

This format:

• Breaks the request body into multiple parts

• Each part can contain text fields or files

• Cannot be parsed by default Express body parsers

So without middleware, Express simply cannot read file data.

What is Multer?

Multer is a middleware specifically designed to handle multipart/form-data.

It:

• Parses incoming file data

• Stores files (locally or memory)

• Attaches file info to the request object

Think of Multer as a bridge between raw file data and usable JavaScript objects.

Understanding the Upload Lifecycle

Here’s what happens when a file is uploaded:

1. Client sends a form with multipart/form-data

2. Request reaches Express

3. Multer intercepts the request

4. File is processed and stored

5. File metadata is attached to:

   • req.file (single file)

   • req.files (multiple files)

6. Your route handler uses this data

Installing Multer

npm install multer

Basic Setup

const express = require("express");
const multer = require("multer");

const app = express();

Storage Configuration (Basic)

By default, Multer stores files in memory. But usually, you want local storage.

const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    cb(null, "uploads/");
  },
  filename: function (req, file, cb) {
    cb(null, Date.now() + "-" + file.originalname);
  }
});

const upload = multer({ storage });

Key Concepts:

• destination → where file is stored

• filename → how file is named

Handling Single File Upload

app.post("/upload", upload.single("profile"), (req, res) => {
  console.log(req.file);
  res.send("File uploaded successfully");
});

Important:

• "profile" must match the form input name

• File info is available in req.file

Handling Multiple File Uploads

Option 1: Multiple files with same field

app.post("/upload-multiple", upload.array("photos", 5), (req, res) => {
  console.log(req.files);
  res.send("Multiple files uploaded");
});

• "photos" → field name

• 5 → max number of files

Option 2: Different fields

app.post("/upload-fields", upload.fields([
  { name: "avatar", maxCount: 1 },
  { name: "documents", maxCount: 3 }
]), (req, res) => {
  console.log(req.files);
  res.send("Files uploaded");
});

Serving Uploaded Files

Once files are stored locally, you need to serve them.

app.use("/uploads", express.static("uploads"));

Now files are accessible via:

http://localhost:3000/uploads/filename.jpg

Example HTML Form

<form action="/upload" method="POST" enctype="multipart/form-data">
  <input type="file" name="profile" />
  <button type="submit">Upload</button>
</form>

Key Point:

• enctype="multipart/form-data" is mandatory

Common Mistakes to Avoid

• ❌ Forgetting enctype="multipart/form-data"

• ❌ Mismatched field name between form and Multer

• ❌ Not creating the uploads/ directory

• ❌ Trying to use express.json() for file uploads

Minimal Mental Model

• Express alone → handles JSON/text

• Multer → handles files

• multipart/form-data → required format

When to Use Multer

Use Multer when:

• Uploading profile images

• Uploading documents (PDF, resumes)

• Handling media files

Avoid overcomplicating early:

• Don’t jump to cloud storage yet (S3, Cloudinary)

• First master local uploads + serving

Before Node.js existed, JavaScript had one job — run inside browsers.

When you opened a website:

• HTML → structure

• CSS → styling

• JavaScript → interactivity

But all of this happened only in the browser. JavaScript had no access to the server, filesystem, or backend logic.

👉 If you wanted backend logic, you had to use languages like:

• PHP

• Java

• Python

So developers were forced to use:

JavaScript for frontend + another language for backend

That split created complexity.

🧠 What is Node.js?

Node.js is a JavaScript runtime that allows you to run JavaScript outside the browser — on the server.

In simple terms:

Node.js lets you build backend systems using JavaScript.

🧩 JavaScript: Language vs Runtime

This is where many beginners get confused.

• JavaScript = Programming Language

• Node.js = Runtime Environment

Analogy:

Think of JavaScript as:

🧠 A brain that knows how to think

And Node.js as:

🏠 A place where that brain can work outside the browser

🌐 Browser JS vs Server JS

👉 Same language, different environment.

⚙️ How Node.js Made This Possible

Node.js uses the V8 Engine (the same engine used by Chrome).

🔍 V8 Engine (High-Level Overview)

• Converts JavaScript into machine code

• Executes it very fast

• Written in C++

👉 Node.js took this engine and said:

“Let’s run JavaScript outside the browser.”

And added:

• File system access

• Network handling

• OS-level capabilities

🔄 Event-Driven Architecture (Core Idea)

Node.js works differently from traditional servers.

Traditional Servers (PHP, Java)

• Create a new thread per request

• Can become heavy under high traffic

Node.js Approach

• Single thread

• Uses event loop + non-blocking I/O

Analogy:

Imagine a waiter in a restaurant:

• Traditional system → One waiter per table

• Node.js → One smart waiter handling all tables efficiently

👉 Instead of waiting for one task to finish, Node.js:

• Starts a task

• Moves to the next

• Comes back when the first task is done

This is called:

Non-blocking, event-driven execution

⚔️ Node.js vs Traditional Backend Runtimes

👉 Biggest advantage:

You can use JavaScript everywhere (frontend + backend)

🌍 Real-World Use Cases of Node.js

Node.js is widely used in production systems:

🔹 APIs & Backend Services

• REST APIs

• GraphQL servers

🔹 Real-Time Applications

• Chat apps (like WhatsApp Web)

• Live collaboration tools

🔹 Streaming Applications

• Video streaming platforms

• Music apps

🔹 Microservices

• Scalable backend systems

🔹 Developer Tools

• Build tools (Webpack, Vite)

• CLI tools

🤔 Why Developers Adopted Node.js

Node.js became popular because:

✅ Single Language Stack

No need to switch between JS and another backend language.

⚡ High Performance for I/O

Handles thousands of requests efficiently.

🔄 Asynchronous Nature

Perfect for modern web apps.

🌱 Massive Ecosystem

• npm (Node Package Manager)

• Millions of libraries

🧠 Final Takeaway

Node.js didn’t create a new language.

It simply changed where JavaScript can run.

From being “just a browser language” 👉 to becoming a full-stack powerhouse

Before diving into JWT, let’s understand the core problem.

When a user interacts with your application (login, profile, payments), your server needs to answer:

👉 “Who is this user?”

Without authentication:

• Anyone can access protected data

• No user-specific experience

• Security risks increase drastically

Authentication ensures:

• Only valid users can access resources

• Actions are tied to identities

• Sensitive data remains protected

What is Authentication?

Authentication is the process of verifying who a user is.

Example:

• You log in using email & password

• Server checks credentials

• If valid → user is authenticated

What is JWT?

JWT (JSON Web Token) is a compact, secure way to transmit user identity between client and server.

👉 Instead of storing session data on the server, JWT allows stateless authentication.

Stateless Authentication (Simple Explanation)

• Server does not store user session

• All required user info is stored inside the token

• Every request carries that token

This makes systems:

• Scalable

• Faster (no DB lookup for session)

• Ideal for APIs

Structure of a JWT

A JWT consists of 3 parts, separated by dots:

xxxxx.yyyyy.zzzzz

1. Header

The header contains metadata about the token.

Example:

{
  "alg": "HS256",
  "typ": "JWT"
}

• alg → algorithm used for signing

• typ → token type

2. Payload

This contains the actual data (called claims).

Example:

{
  "userId": "12345",
  "email": "user@example.com"
}

Types of claims:

• Registered → exp, iat

• Public → custom data

• Private → app-specific

⚠️ Important: Payload is not encrypted, only encoded.

3. Signature

The signature ensures the token is not tampered with.

Created using:

Header + Payload + Secret Key

If someone modifies the token → signature becomes invalid.

Login Flow Using JWT

Here’s how JWT-based login works:

1. User sends login request (email + password)

2. Server verifies credentials

3. Server generates JWT

4. Token is sent back to client

5. Client stores token (localStorage/cookie)

Sending Token with Requests

After login, every request must include the token.

Common method:

Authorization: Bearer <token>

Example in fetch:

fetch('/profile', {
  headers: {
    Authorization: `Bearer ${token}`
  }
})

Protecting Routes Using JWT

On protected routes:

1. Server reads token from request header

2. Verifies token using secret key

3. Extracts user data

4. Allows or denies access

Example (Express middleware):

const jwt = require('jsonwebtoken');

function authMiddleware(req, res, next) {
  const token = req.headers.authorization?.split(' ')[1];

  if (!token) {
    return res.status(401).send('Access Denied');
  }

  try {
    const decoded = jwt.verify(token, 'SECRET_KEY');
    req.user = decoded;
    next();
  } catch (err) {
    res.status(400).send('Invalid Token');
  }
}

Key Takeaways

• Authentication = verifying user identity

• JWT enables stateless authentication

• Token has 3 parts: Header, Payload, Signature

• No server-side session storage required

• Token must be sent with every request

• Middleware protects routes

Final Insight

JWT is powerful, but use it correctly:

• Always set expiration (exp)

• Never store sensitive data in payload

• Keep your secret key secure

That’s where Express.js comes in.

📌 What is Express.js?

Express.js is a minimal and flexible web framework for Node.js that provides a structured way to build web applications and APIs.

Instead of writing everything from scratch using Node’s core modules, Express gives you:

• Clean routing

• Middleware support

• Simplified request/response handling

🤔 Why Express Simplifies Node.js Development

Using the built-in http module in Node.js is powerful—but verbose.

🔴 Raw Node.js Example

const http = require("http");

const server = http.createServer((req, res) => {
  if (req.url === "/" && req.method === "GET") {
    res.end("Hello World");
  }
});

server.listen(3000);

Problems:

• Manual routing logic

• Hard to scale

• Poor readability

🟢 Express.js Equivalent

const express = require("express");
const app = express();

app.get("/", (req, res) => {
  res.send("Hello World");
});

app.listen(3000);

Much cleaner, right?

👉 Express abstracts away boilerplate so you can focus on logic instead of plumbing.

⚙️ Creating Your First Express Server

Step 1: Install Express

npm init -y
npm install express

Step 2: Create Server

const express = require("express");
const app = express();

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

🌐 Handling GET Requests

GET requests are used to fetch data.

app.get("/", (req, res) => {
  res.send("Welcome to my server");
});

You can also use route paths:

app.get("/about", (req, res) => {
  res.send("About Page");
});

📦 Handling POST Requests

POST requests are used to send data to the server.

First, enable JSON parsing middleware:

app.use(express.json());

Now handle POST:

app.post("/data", (req, res) => {
  const userData = req.body;
  res.send(`Received: ${JSON.stringify(userData)}`);
});

📤 Sending Responses

Express provides multiple response methods:

res.send("Text response");
res.json({ message: "JSON response" });
res.status(200).send("Status set");

These make response handling far more intuitive compared to raw Node.js.

🧭 Understanding Routing (Core Concept)

Routing means defining how your server responds to different endpoints.

app.get("/user", (req, res) => {
  res.send("User route");
});

app.post("/user", (req, res) => {
  res.send("Create user");
});

👉 Same route, different methods → different behavior.

🧠 Key Takeaways

• Express.js is a lightweight abstraction over Node.js

• It simplifies:

  • Routing

  • Request handling

  • Response formatting

• Makes your backend:

  • Cleaner

  • Scalable

  • Easier to maintain

🔚 Final Thoughts

If you're serious about backend development, learning Express.js is non-negotiable. It’s the foundation for building APIs, full-stack apps, and scalable systems in Node.js.

Once you're comfortable with this, the next step is:

• Middleware

• Routing architecture

• REST API design

1. Installing Node.js

Node.js is a JavaScript runtime that allows you to execute JavaScript outside the browser.

Steps:

1. Go to the official Node.js website: https://nodejs.org

2. Download the LTS (Long-Term Support) version

3. Install it using the default settings

This installation also includes npm (Node Package Manager), which you’ll use later for managing dependencies.

2. Verifying Installation

Once installed, you need to confirm that Node.js is correctly set up.

Open your terminal (or command prompt) and run:

node -v

You should see a version number like:

v24.x.x

Now check npm:

npm -v

If both commands return versions, your setup is successful.

💡 Tip: If you ever forget common commands or want quick one-liner solutions for JavaScript/Node tasks, platforms like Practica JS can be very handy for fast reference and practice.

3. Understanding Node REPL

REPL stands for:

Read – Evaluate – Print – Loop

It’s an interactive environment where you can execute JavaScript code line-by-line.

Start REPL:

node

You’ll see something like:

>

Try this:

console.log("Hello from REPL");

Output:

Hello from REPL

Exit REPL:

.exit

or press:

Ctrl + C (twice)

Why REPL matters:

• Quick testing of JavaScript code

• Debugging logic

• Learning Node interactively

4. Creating Your First JavaScript File

Now let’s move from interactive mode to writing actual scripts.

Step 1: Create a file

Create a file named:

app.js

Step 2: Add code

console.log("Hello, Node.js!");

5. Running Your Script

To execute the file, navigate to its directory in terminal and run:

node app.js

Output:

Hello, Node.js!

This is your first Node.js execution 🎉

6. Writing Your First “Hello World” Server

Now let’s create a basic HTTP server using Node’s built-in module (no frameworks).

Step 1: Update app.js

const http = require("http");

const server = http.createServer((req, res) => {
  res.write("Hello World from Node.js Server");
  res.end();
});

server.listen(3000, () => {
  console.log("Server running at http://localhost:3000/");
});

Step 2: Run the server

node app.js

Step 3: Open browser

Go to:

http://localhost:3000

Output:

Hello World from Node.js Server

What’s Happening Behind the Scenes?

• require("http") → imports Node’s built-in HTTP module

• createServer() → creates a server instance

• req → incoming request object

• res → response object sent back to client

• listen(3000) → server starts listening on port 3000

Key Takeaways

• Node.js allows you to run JavaScript outside the browser

• REPL is useful for quick experimentation

• Scripts are executed using the node command

• You can create a basic server without any frameworks

• Node’s core modules (like http) are powerful enough for fundamentals

What’s Next?

Once you’re comfortable with this setup, you can move on to:

• File system operations (fs module)

• Handling routes manually

• Understanding asynchronous behavior in Node.js

If you want, I can also:

• Add SEO title + meta description

• Suggest Hashnode tags

• Create a cover image prompt

• Or help you make this stand out from others in your cohort

Just tell me 👍

They may look similar at first glance, but they serve very different purposes in API design.

🔹 What Are URL Parameters?

URL parameters (also called route parameters) are dynamic values embedded directly in the URL path.

They are typically used to identify a specific resource.

Example:

/users/123

Here:

• 123 is a URL parameter

• It represents a specific user ID

Express Example:

app.get('/users/:id', (req, res) => {
  const userId = req.params.id;
  res.send(`User ID is ${userId}`);
});

Key Characteristics:

• Part of the URL path

• Used as identifiers

• Required to match the route

• Clean and RESTful

🔹 What Are Query Parameters?

Query parameters are key-value pairs appended to the end of a URL after a ?.

They are used to modify or filter data, not identify it.

Example:

/users?age=25&city=Kolkata

Here:

• age=25 and city=Kolkata are query parameters

• They act as filters

Express Example:

app.get('/users', (req, res) => {
  const age = req.query.age;
  const city = req.query.city;

  res.send(`Filtering users by age \({age} and city \){city}`);
});

Key Characteristics:

• Appear after ? in the URL

• Used as filters, sorting, or modifiers

• Optional by nature

• Can have multiple values

Mind map -

🔹 Core Difference: Params vs Query

Mind map -

🔹 Practical Examples

1. User Profile (Identifier → Params)

GET /users/123

app.get('/users/:id', (req, res) => {
  const id = req.params.id;
  res.send(`Fetching user with ID ${id}`);
});

✔ Use params because you're targeting one specific user

2. Search or Filtering (Modifier → Query)

GET /users?age=25&city=Kolkata

app.get('/users', (req, res) => {
  const { age, city } = req.query;
  res.send(`Filtering users with age \({age} in \){city}`);
});

✔ Use query because you're refining results

3. Combining Both

You can use both together:

GET /users/123/posts?sort=latest

app.get('/users/:id/posts', (req, res) => {
  const userId = req.params.id;
  const sort = req.query.sort;

  res.send(`Posts of user \({userId}, sorted by \){sort}`);
});

🔹 When to Use Params vs Query

Use URL Parameters when:

• You are accessing a specific resource

• The value is mandatory

• It represents an identity

✔ Examples:

• /products/456

• /orders/789

Use Query Parameters when:

• You are filtering, sorting, or paginating

• The values are optional

• You want flexible API behavior

✔ Examples:

• /products?category=electronics

• /orders?status=delivered&page=2

🔹 Best Practices

• Treat params as identifiers, not filters

• Keep URLs clean and meaningful

• Avoid mixing responsibilities (don’t use query for IDs)

• Use query params for:

  • Pagination (page=1)

  • Sorting (sort=price)

  • Filtering (category=books)

🔹 Mental Model (Important)

• Params = “Which exact thing?”

• Query = “How should I view or filter it?”

🔹 Final Takeaway

If you're designing APIs:

• Think of URL params as primary keys

• Think of query strings as SQL WHERE clauses

This distinction leads to cleaner, more scalable, and RESTful API design.

If you want, I can next help you turn this into a perfect Hashnode blog with visuals + diagrams + SEO optimization, or even align it with your cohort grading rubric.

In this guide, we’ll break down how file storage works in Node.js with Express, along with best practices used in production systems.

📦 Where Uploaded Files Are Stored

When a user uploads a file, the backend must decide where to store it. Broadly, there are two approaches:

1. Local File System (Server Storage)

Files are stored directly on the server's disk.

Example structure:

project-root/
│
├── uploads/
│   ├── images/
│   │   ├── img1.jpg
│   │   ├── img2.png
│   │
│   ├── documents/
│       ├── file1.pdf
│
├── server.js

How it works:

• A middleware like multer saves files into a folder (uploads/)

• File path is stored in your database

• Later, files are served via Express

2. External Storage (Cloud / Object Storage)

Instead of storing files locally, you use services like:

• AWS S3

• Cloudinary

• Firebase Storage

How it works:

• File is uploaded directly (or via backend) to cloud storage

• You store the file URL in your database

• Files are served directly from CDN

⚖️ Local Storage vs External Storage

👉 Rule of thumb:

• Use local storage for small projects / learning

• Use external storage for production apps

📁 Serving Static Files in Express

Express provides a built-in middleware to serve static files:

const express = require("express");
const path = require("path");

const app = express();

// Serve files from "uploads" folder
app.use("/uploads", express.static(path.join(__dirname, "uploads")));

What this does:

• Maps /uploads URL path to the actual uploads/ folder

• Makes files publicly accessible

🌐 Accessing Uploaded Files via URL

If you have a file stored like:

uploads/images/profile.jpg

You can access it via:

http://localhost:3000/uploads/images/profile.jpg

👉 This works because of:

app.use("/uploads", express.static("uploads"));

🛠️ Upload Example Using Multer

const multer = require("multer");

const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    cb(null, "uploads/images");
  },
  filename: function (req, file, cb) {
    cb(null, Date.now() + "-" + file.originalname);
  }
});

const upload = multer({ storage: storage });

app.post("/upload", upload.single("file"), (req, res) => {
  res.send({
    message: "File uploaded successfully",
    filePath: req.file.path
  });
});

🔐 Security Considerations for File Uploads

This is where most beginners make mistakes. File uploads are a major attack surface.

1. Validate File Type

Never trust client input.

fileFilter: (req, file, cb) => {
  if (file.mimetype.startsWith("image/")) {
    cb(null, true);
  } else {
    cb(new Error("Only images allowed"), false);
  }
}

2. Limit File Size

limits: { fileSize: 5 * 1024 * 1024 } // 5MB

3. Avoid Executable Files

Block .js, .exe, .sh, etc.

4. Use Unique File Names

Prevent overwriting:

Date.now() + "-" + file.originalname

5. Store Outside Public Root (Advanced)

Instead of directly exposing /uploads, use a controller:

app.get("/file/:name", (req, res) => {
  res.sendFile(path.join(__dirname, "uploads", req.params.name));
});

6. Scan Files (Production)

Use antivirus or scanning tools for:

• PDFs

• Documents

• User uploads

Here's a prod grade structure ( it has sharding, careful if following this )

📚 Static File Serving Concept (Important)

Static files are files that:

• Don’t change dynamically

• Are served directly to the client

Examples:

• Images

• CSS files

• PDFs

Express acts like a file server when using:

express.static()

👉 Instead of writing routes manually, Express:

• Locates the file

• Streams it to the client

• Sets correct headers

Here's a prod grade example -

🧠 Best Practices Summary

• Use multer for handling uploads

• Separate folders by file type (images, docs, etc.)

• Never trust user-uploaded files

• Prefer cloud storage in production

• Use CDN for faster delivery

• Store only file paths/URLs in DB (not actual files)

🚀 Final Thought

File handling may look simple, but at scale, it becomes a system design problem involving:

• Storage architecture

• Security layers

• CDN optimization

• Cost management

Three terms come up repeatedly in this context:

• Sessions

• Cookies

• JWT (JSON Web Tokens)

They’re often confused or used interchangeably — but they solve different parts of the problem.

Let’s break this down clearly and practically.

What Are Cookies?

A cookie is a small piece of data stored in the user’s browser.

• Sent from server → browser

• Automatically included in every request → server

Example

Set-Cookie: sessionId=abc123; HttpOnly

Key Points

• Stored on the client (browser)

• Automatically attached to requests

• Can store:

  • Session IDs

  • JWT tokens

  • Preferences

👉 ❗️Cookies are just a storage + transport mechanism, not an authentication strategy by themselves.

What Are Sessions?

A session is a server-side way to track a user.

How It Works

1. User logs in

2. Server creates a session:

   session = {
     userId: "123",
     createdAt: ...
   }
   

3. Server stores it (memory / DB / Redis)

4. Server sends a session ID via cookie

5. Browser sends cookie on every request

6. Server looks up session using that ID

Key Points

• Stateful (server stores user data)

• Session ID is usually stored in a cookie

• Requires server-side storage

What Are JWT Tokens?

A JWT (JSON Web Token) is a self-contained token that stores user data.

Structure

header.payload.signature

Example Payload

{
  "userId": "123",
  "role": "admin"
}

How It Works

1. User logs in

2. Server generates JWT

3. Sends it to client

4. Client stores it (cookie / localStorage)

5. Client sends token with each request

6. Server verifies signature, no DB lookup needed

Key Points

• Stateless

• No server-side storage required

• Everything is inside the token

Stateful vs Stateless Authentication

Mapping

• Sessions → Stateful

• JWT → Stateless

• Cookies → Transport layer (used in both)

Sessions vs JWT: Core Differences

Where Cookies Fit In

Cookies are used in both approaches:

Real-World Comparison

Session-Based Authentication

Best for:

• Traditional web apps

• Server-rendered apps

• Admin dashboards

Why:

• Easy to manage

• Easy to revoke access

• Strong control from server

Example Stack:

• Express + express-session + Redis

JWT-Based Authentication

Best for:

• APIs

• Mobile apps

• Microservices

• Distributed systems

Why:

• No shared session store needed

• Works across services

• Scales horizontally

Example Stack:

• Node.js + JWT + Authorization headers

When to Use What (Decision Guide)

Use Sessions if:

• You have a monolithic backend

• You want easy logout / revoke

• You prefer simplicity over scalability

Use JWT if:

• You’re building APIs for multiple clients

• You need stateless architecture

• You’re working with microservices

Use Cookies if:

• You want automatic request handling

• You're working with browser-based apps

Practical Insight (What Most Beginners Miss)

• Cookies ≠ Authentication

• JWT ≠ always better

• Sessions are still widely used in production

A common modern setup:

👉 JWT stored inside HttpOnly cookies

This combines:

• Security of cookies

• Scalability of JWT

Final Comparison Table

Mind Map ➡️

Conclusion

• Cookies = how data travels

• Sessions = server-managed identity

• JWT = self-contained identity

There’s no “best” option — only context-appropriate choices.

If you're building:

• A simple web app → Sessions

• A scalable API → JWT

• A browser-based system → Cookies (with either approach)

At first glance, it feels confusing — why not just run code line by line like normal programs?

Let’s break it down step by step.

Why Async Code Exists in Node.js

Node.js is built on a single-threaded, non-blocking event loop architecture.

This means:

• It can handle many operations at once

• But it does not create a new thread for each task

Now imagine this scenario:

const data = fs.readFileSync("file.txt");
console.log(data);
console.log("Done");

This is blocking (synchronous):

• Node waits for the file to be read

• Nothing else happens during that time

Problem

If file reading takes time:

• Your server is stuck

• Other users have to wait

Solution: Asynchronous Code

fs.readFile("file.txt", "utf-8", (err, data) => {
  console.log(data);
});

console.log("Done");

Now:

• Node does not wait

• It continues execution

• Handles the result later

Real Scenario: File Reading

Let’s understand the flow properly.

Code:

console.log("Start");

fs.readFile("file.txt", "utf-8", (err, data) => {
  if (err) {
    console.log("Error:", err);
    return;
  }
  console.log("File Data:", data);
});

console.log("End");

Step-by-Step Execution

1. "Start" is printed

2. fs.readFile is sent to the system (async task)

3. Node does NOT wait

4. "End" is printed

5. Once file reading completes → callback executes

6. "File Data" is printed

Output Order

Start
End
File Data: ...

This is the core idea of async in Node.js.

Callback-Based Async Execution

A callback is simply:

A function passed as an argument to another function, executed later.

Example:

function fetchData(callback) {
  setTimeout(() => {
    callback("Data received");
  }, 1000);
}

fetchData((data) => {
  console.log(data);
});

Key Idea

• You give control to another function

• It calls you back when work is done

Problem: Callback Hell

Now let’s chain multiple async operations:

fs.readFile("file1.txt", "utf-8", (err, data1) => {
  fs.readFile("file2.txt", "utf-8", (err, data2) => {
    fs.readFile("file3.txt", "utf-8", (err, data3) => {
      console.log(data1, data2, data3);
    });
  });
});

Issues

• Deep nesting (pyramid shape)

• Hard to read

• Hard to debug

• Error handling becomes messy

This is called:

Callback Hell (or Pyramid of Doom)

Promise-Based Async Handling

Promises were introduced to fix these problems.

What is a Promise?

A Promise represents:

A value that will be available in the future

States of a Promise

• Pending

• Resolved (fulfilled)

• Rejected

Basic Example

const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    resolve("Data received");
  }, 1000);
});

promise.then((data) => {
  console.log(data);
});

Rewriting File Example Using Promises

Using fs.promises:

const fs = require("fs").promises;

fs.readFile("file1.txt", "utf-8")
  .then((data1) => {
    return fs.readFile("file2.txt", "utf-8")
      .then((data2) => {
        return fs.readFile("file3.txt", "utf-8")
          .then((data3) => {
            console.log(data1, data2, data3);
          });
      });
  })
  .catch((err) => {
    console.log(err);
  });

Better Version (Flat Chain)

fs.readFile("file1.txt", "utf-8")
  .then((data1) => {
    console.log(data1);
    return fs.readFile("file2.txt", "utf-8");
  })
  .then((data2) => {
    console.log(data2);
    return fs.readFile("file3.txt", "utf-8");
  })
  .then((data3) => {
    console.log(data3);
  })
  .catch((err) => {
    console.log(err);
  });

Benefits of Promises

1. Better Readability

• No deep nesting

• Linear flow

2. Centralised Error Handling

.catch((err) => {
  console.log(err);
});

Instead of handling errors at every level.

3. Easier Composition

You can chain multiple async operations cleanly.

4. Foundation for Async/Await

Promises enable:

const data = await fs.readFile("file.txt", "utf-8");

Callback vs Promise Comparison

Final Insight

• Callbacks are low-level primitives

• Promises are structured abstractions

In modern Node.js:

You should prefer Promises (and async/await) over callbacks

Linux is not about commands. Linux is about structure.

Everything — networking, users, processes, devices — is exposed through the file system. So instead of practicing commands, I decided to investigate the system itself.

This blog is a collection of the most interesting things I discovered while exploring a real Linux environment. Each section focuses on what exists, why it exists, and what problem it solves.

1. /etc — The Brain of System Configuration

What it is

/etc contains configuration files that control almost every aspect of system behaviour.

Why it exists

Instead of hardcoding system behaviour into binaries, Linux separates configuration from execution. This makes the system flexible and editable.

What problem it solves

• Allows admins to change system behaviour without recompiling software

• Centralises configuration for easier debugging and auditing

What I found interesting

Files like:

• /etc/hosts → local DNS overrides

• /etc/resolv.conf → DNS server configuration

• /etc/passwd → user identity mapping

This made me realise: Linux doesn’t hide its logic — it exposes it in plain text.

2. DNS Resolution — /etc/resolv.conf

What it is

A file that defines which DNS servers your system uses.

Why it exists

When you type a domain (like google.com), your system needs a resolver to translate it into an IP address.

What problem it solves

• Enables domain name resolution

• Allows custom DNS configuration (e.g., Google DNS, Cloudflare)

Insight

I noticed that sometimes this file gets overwritten automatically (e.g., by NetworkManager or DHCP).

This taught me:

DNS in Linux is not static — it's dynamically managed by networking services.

3. /proc — The Live System Mirror

What it is

A virtual filesystem that exposes real-time system and process information.

Why it exists

Linux treats everything as a file — even running processes.

What problem it solves

• Provides introspection into system state without special tools .

• Allows programs to read kernel and process data easily .

What I found fascinating

• /proc/cpuinfo → CPU details

• /proc/meminfo → memory usage

• /proc/<pid>/ → everything about a running process

This changed my mental model completely:

Processes are not abstract — they are directories with data.

4. /dev — Devices as Files

What it is

A directory containing device files that represent hardware.

Why it exists

Linux uses a unified interface: everything is treated as a file, including hardware.

What problem it solves

• Standardises interaction with hardware .

• Enables simple read/write operations for devices .

Examples

• /dev/sda → disk

• /dev/null → data sink

• /dev/random → entropy source

Insight

Instead of using special APIs, programs interact with hardware using file operations.

That’s elegant system design.

5. /var/log — The System’s Memory

What it is

A directory storing logs generated by the system and services.

Why it exists

Systems need historical data to debug issues and monitor behaviour.

What problem it solves

• Debugging failures

• Security auditing

• Monitoring system activity

Interesting files

• /var/log/syslog or /var/log/messages

• /var/log/auth.log

Insight

Logs are not just for debugging — they are evidence of system behavior over time.

6. User Management — /etc/passwd & /etc/shadow

What they are

• /etc/passwd → user metadata

• /etc/shadow → encrypted passwords

Why they exist

Separates identity from authentication for security.

What problem they solve

• Secure storage of credentials

• System-wide user management

Insight

Passwords are not stored in /etc/passwd anymore — that’s intentional for security.

Also, users are just structured entries — not “accounts” in the abstract sense.

7. Permissions — The Real Security Layer

What it is

Linux uses a permission model based on:

• Owner

• Group

• Others

Why it exists

To enforce access control at the filesystem level.

What problem it solves

• Prevents unauthorised access

• Controls execution and modification rights

Insight

Permissions are not optional — they are core to Linux security.

Even root privileges operate within this model (though with override capability).

8. /boot — The System’s Entry Point

What it is

Contains files required to boot the system.

Why it exists

Before the OS runs, the system needs a minimal environment to start the kernel.

What problem it solves

• Initialises the OS

• Loads the kernel and bootloader

Insight

Files like:

• Kernel images

• Bootloader configs (GRUB)

This made me realise:

Booting is just a staged file-loading process.

9. System Services — /etc/systemd

What it is

Configuration for system services managed by systemd.

Why it exists

Modern Linux systems need a structured way to manage background services.

What problem it solves

• Service lifecycle management

• Dependency handling

• Startup sequencing

Insight

Services are defined declaratively using .service files.

Linux doesn’t “run things magically” — everything is explicitly defined.

10. Networking Internals — Interfaces & Routing

What I explored

• Network interface configs

• Routing tables

• Kernel networking data

Where it lives

• /proc/net/route → routing table

• /sys/class/net/ → interfaces

Why it exists

Networking must be dynamically configurable and observable.

What problem it solves

• Packet routing

• Interface management

• Network diagnostics

Insight

Routing is not abstract — it's literally a table the kernel follows.

Final Realisation

After exploring all these components, one idea became very clear:

Linux is not a black box. It is a transparent system built on files and structure.

• Processes → files

• Devices → files

• Configuration → files

• System state → files

Everything is inspectable.

Why This Matters (Especially for Developers)

Understanding the Linux filesystem at this level changes how you think about systems:

• Debugging becomes easier

• Backend systems make more sense

• DevOps concepts feel natural

• You stop relying on tools blindly

Instead of asking:

“Which command should I use?”

You start asking:

“Where in the system is this behavior defined?”

That’s a much more powerful mindset.

Closing Thought

This exploration made me realise that Linux is less about commands and more about philosophy:

Expose everything. Keep it simple. Make it inspectable.

And once you see that, you stop using Linux like a user — you start understanding it like a system engineer.

That’s where Map and Set come in.

They provide more control, better performance in some cases, and cleaner semantics.

🔹 What is a Map?

👉 A Map is a collection of key-value pairs, similar to objects—but with important differences.

📌 Example

const map = new Map();

map.set("name", "Sayantan");
map.set("age", 21);

console.log(map.get("name")); // Sayantan

🔹 Key Features of Map

• Keys can be any data type (not just strings)

• Maintains insertion order

• Has built-in methods like:

  • .set()

  • .get()

  • .has()

  • .delete()

🔹 Example with Non-String Keys

const map = new Map();

map.set(1, "Number key");
map.set(true, "Boolean key");

console.log(map.get(1)); // Number key

👉 This is NOT possible with normal objects.

🔹 What is a Set?

👉 A Set is a collection of unique values.

It automatically removes duplicates.

📌 Example

const set = new Set([1, 2, 2, 3, 4]);

console.log(set); // {1, 2, 3, 4}

🔹 Key Features of Set

• Stores only unique values

• No duplicate entries

• Maintains insertion order

• Useful methods:

  • .add()

  • .has()

  • .delete()

🔹 Example: Removing Duplicates

const arr = [1, 2, 2, 3];

const unique = [...new Set(arr)];

console.log(unique); // [1, 2, 3]

🔥 Map vs Object

📌 Problem with Objects

const obj = {};

obj[1] = "Number key";

console.log(obj["1"]); // "Number key"

👉 Keys are converted to strings → loss of flexibility

🔥 Set vs Array

📌 Problem with Arrays

const arr = [1, 2, 2, 3];

const unique = arr.filter((item, index) => arr.indexOf(item) === index);

console.log(unique); // [1, 2, 3]

❌ Complex ❌ Inefficient

✔ Set solves this easily

🔹 When to Use Map

Use Map when:

• You need key-value storage

• Keys are not strings

• Frequent additions/deletions

• Need guaranteed insertion order

📌 Real Example

const userRoles = new Map();

userRoles.set("Sayantan", "Admin");
userRoles.set("Rahul", "User");

console.log(userRoles.get("Sayantan"));

🔹 When to Use Set

Use Set when:

• You need unique values

• Removing duplicates

• Checking existence quickly

📌 Real Example

const visitedPages = new Set();

visitedPages.add("/home");
visitedPages.add("/about");
visitedPages.add("/home");

console.log(visitedPages); // no duplicates

🧠 Mental Model

• Map → advanced object (key-value storage)

• Set → advanced array (unique values)

🚀 Practical Use Cases

1. Remove Duplicates

const nums = [1, 2, 2, 3];

const uniqueNums = [...new Set(nums)];

2. Fast Lookup

const set = new Set([1, 2, 3]);

console.log(set.has(2)); // true

3. Dynamic Key Storage

const map = new Map();

const objKey = { id: 1 };

map.set(objKey, "Data");

console.log(map.get(objKey)); // Data

⚠️ Common Mistakes

❌ Using Object Instead of Map

const obj = {};

obj[{ id: 1 }] = "value";

console.log(obj); // "[object Object]"

👉 Key gets stringified → incorrect behavior

❌ Expecting Index in Set

const set = new Set([10, 20, 30]);

console.log(set[0]); // undefined ❌

👉 Sets don’t support indexing

Destructuring solves this problem by allowing you to unpack values from arrays or objects into variables in a clean and concise way.

🔹 What is Destructuring?

👉 Destructuring = breaking down complex data structures (arrays/objects) into smaller variables

Instead of accessing values one by one, you can extract them in a single line.

🔹 Before vs After Destructuring

❌ Without Destructuring

const user = {
  name: "Sayantan",
  age: 21
};

const name = user.name;
const age = user.age;

✅ With Destructuring

const user = {
  name: "Sayantan",
  age: 21
};

const { name, age } = user;

✔ Less code ✔ Cleaner ✔ More readable

🔹 Destructuring Arrays

Array destructuring is based on position (index).

📌 Example

const arr = [10, 20, 30];

const [a, b, c] = arr;

console.log(a); // 10
console.log(b); // 20
console.log(c); // 30

🔹 Skipping Values

const arr = [10, 20, 30];

const [first, , third] = arr;

console.log(first); // 10
console.log(third); // 30

🔹 Using Rest Operator

const arr = [1, 2, 3, 4];

const [first, ...rest] = arr;

console.log(first); // 1
console.log(rest);  // [2, 3, 4]

🔹 Destructuring Objects

Object destructuring is based on property names, not position.

📌 Example

const user = {
  name: "Sayantan",
  age: 21,
  city: "Kolkata"
};

const { name, age } = user;

console.log(name); // Sayantan
console.log(age);  // 21

🔹 Renaming Variables

const user = {
  name: "Sayantan"
};

const { name: userName } = user;

console.log(userName); // Sayantan

🔹 Nested Destructuring

const user = {
  name: "Sayantan",
  address: {
    city: "Kolkata"
  }
};

const {
  address: { city }
} = user;

console.log(city); // Kolkata

🔹 Default Values

If a value is missing, you can assign a default.

const user = {
  name: "Sayantan"
};

const { name, age = 18 } = user;

console.log(name); // Sayantan
console.log(age);  // 18

🔹 Destructuring in Function Parameters

function greet({ name, age }) {
  console.log(`Hello \({name}, age \){age}`);
}

greet({ name: "Sayantan", age: 21 });

✔ Very common in real-world code (especially React)

🔹 Benefits of Destructuring

✅ 1. Reduces Repetitive Code

// Without
const name = user.name;
const age = user.age;

// With
const { name, age } = user;

✅ 2. Improves Readability

Cleaner and easier to understand structure.

✅ 3. Makes Code More Expressive

You clearly see what data is being used.

✅ 4. Useful in Modern JavaScript

• React props

• API responses

• Function arguments

🔹 Real-World Example

const response = {
  data: {
    user: {
      name: "Sayantan",
      age: 21
    }
  }
};

const {
  data: {
    user: { name, age }
  }
} = response;

console.log(name, age);

⚠️ Common Mistakes

❌ Wrong Variable Names

const user = { name: "Sayantan" };

const { username } = user; // ❌ undefined

✔ Must match property name

❌ Accessing Nested Without Safety

const user = {};

const {
  address: { city }
} = user; // ❌ error

✔ Use optional chaining or defaults

🧠 Mental Model

• Array destructuring → position-based

• Object destructuring → key-based

This is where Promises come in.

They provide a cleaner and more structured way to handle asynchronous operations.

🔹 What Problem Do Promises Solve?

🚨 Callback Hell

getUser(function(user) {
  getOrders(user.id, function(orders) {
    getOrderDetails(orders[0], function(details) {
      console.log(details);
    });
  });
});

❌ Deep nesting ❌ Hard to read ❌ Difficult to debug

✅ With Promises

getUser()
  .then(user => getOrders(user.id))
  .then(orders => getOrderDetails(orders[0]))
  .then(details => console.log(details))
  .catch(err => console.error(err));

✔ Flat structure ✔ Better readability ✔ Easier error handling

🔹 What is a Promise?

👉 A Promise is a placeholder for a value that will be available in the future.

Think of it like:

"I don’t have the data right now, but I promise I’ll give it to you later."

🔹 Promise States

A promise has 3 states:

1. Pending

   • Initial state

   • Operation not completed yet

2. Fulfilled

   • Operation successful

   • Value is available

3. Rejected

   • Operation failed

   • Error is returned

📌 Example

const promise = new Promise((resolve, reject) => {
  const success = true;

  if (success) {
    resolve("Data fetched");
  } else {
    reject("Error occurred");
  }
});

🔹 Basic Promise Lifecycle

const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    resolve("Done!");
  }, 2000);
});

promise
  .then(result => console.log(result))  // success
  .catch(error => console.error(error)) // failure
  .finally(() => console.log("Finished"));

🔍 Flow

1. Promise starts → Pending

2. After 2 sec → Fulfilled

3. .then() runs

4. .finally() always runs

🔹 Handling Success and Failure

fetchData()
  .then(data => {
    console.log("Success:", data);
  })
  .catch(error => {
    console.log("Error:", error);
  });

✔ .then() → handles success ✔ .catch() → handles errors

🔹 Promise Chaining

Promises allow sequential execution without nesting.

getUser()
  .then(user => getOrders(user.id))
  .then(orders => getOrdersDetails(orders))
  .then(details => console.log(details))
  .catch(err => console.error(err));

👉 Each .then() returns a new promise

🔥 Why Chaining Works

• Each step waits for the previous one

• No pyramid structure

• Cleaner and more readable

🔹 Real-World Example

fetch("https://jsonplaceholder.typicode.com/posts/1")
  .then(res => res.json())
  .then(data => console.log(data))
  .catch(err => console.error(err));

🔹 Promises vs Callbacks

🔹 Mental Model

• Promise = future value

• .then() = "When it's ready, do this"

• .catch() = "If it fails, handle it"

🔹 Common Mistakes

❌ Not Returning Promise in Chain

getUser()
  .then(user => {
    getOrders(user.id); // ❌ missing return
  })
  .then(orders => console.log(orders)); // undefined

✔ Always return the promise

❌ Multiple .then without Chain

promise.then(a => console.log(a));
promise.then(b => console.log(b));

👉 Runs independently (not sequential)

🚀 Why Promises Matter

• Foundation of modern async JavaScript

• Used in:

  • APIs (fetch)

  • Databases

  • File systems

• Base for async/await

At the core, JavaScript can execute code in two ways:

• Synchronous (blocking)

• Asynchronous (non-blocking)

Let’s break this down step by step in a very intuitive way.

🔹 What is Synchronous Code?

Synchronous code executes line by line, one after another.

👉 Each line must finish before the next one starts.

📌 Example

console.log("Start");

console.log("Middle");

console.log("End");

🧠 Output

Start
Middle
End

🔍 Explanation

• JavaScript executes the first line

• Then moves to the next

• No skipping, no waiting

✔ Simple ✔ Predictable ❌ Can become slow if a task takes too long

🔹 What is Asynchronous Code?

Asynchronous code allows JavaScript to perform tasks without blocking the main thread.

👉 It can start a task and move on without waiting for it to finish.

📌 Example with Timer

console.log("Start");

setTimeout(() => {
  console.log("Inside Timeout");
}, 2000);

console.log("End");

🧠 Output

Start
End
Inside Timeout

🔍 Explanation

• setTimeout is asynchronous

• JavaScript does not wait for 2 seconds

• It moves to the next line immediately

✔ Faster ✔ Efficient ✔ Non-blocking

🔥 Synchronous vs Asynchronous (Core Difference)

🔹 Why JavaScript Needs Asynchronous Behavior

JavaScript is single-threaded.

👉 It can do only one thing at a time.

So imagine this situation:

❌ Without Async (Blocking)

console.log("Start");

// Simulating heavy work
for (let i = 0; i < 1e9; i++) {}

console.log("End");

⛔ The browser freezes ⛔ User cannot interact ⛔ Bad user experience

✅ With Async

console.log("Start");

setTimeout(() => {
  console.log("Heavy task done");
}, 0);

console.log("End");

✔ UI remains responsive ✔ Tasks handled efficiently

🔹 Real-World Example: API Call

❌ If It Were Synchronous

const data = fetch("https://api.com/data"); // imagine blocking

console.log(data);

👉 The app would freeze until data arrives

✅ Actual Async Behavior

fetch("https://api.com/data")
  .then(res => res.json())
  .then(data => console.log(data));

✔ App continues running ✔ Data comes later

🔹 Blocking vs Non-Blocking (Simple Analogy)

☕ Real Life Example

• Synchronous → Ordering coffee and waiting at the counter until it's ready

• Asynchronous → Ordering coffee, getting a token, and sitting down while it’s prepared

👉 Which is better? Obviously async.

🔹 How JavaScript Handles Async Internally

Even though JavaScript is single-threaded, it uses:

• Call Stack → Executes functions

• Web APIs → Handle async tasks (timers, fetch)

• Callback Queue → Stores completed async tasks

• Event Loop → Moves tasks back to execution

👉 This system makes async possible.

🔹 Problems with Blocking Code

1. Freezes UI

2. Bad user experience

3. Slow performance

4. Unresponsive applications

🔹 When to Use What?

Use Synchronous:

• Simple calculations

• Small tasks

Use Asynchronous:

• API calls

• File handling

• Timers

• Database operations

🧠 Mental Model

• Synchronous → "Wait and do"

• Asynchronous → "Start and move on"