articleAugust 15, 2026

How I Scrape Data from Websites

# Comprehensive Guide to Web Scraping Using Node.js (Case Study: Komiku API)

Web scraping is a powerful technique for extracting data from websites that do not provide an official API. In this deep dive, I will share the architectural decisions, tools, and code behind **Komiku API**—a robust, multi-source manga REST API built with Node.js.

Building a scraper that reliably fetches data from multiple sources (like Komiku, WestManga, and WeebCentral) presents unique challenges. You have to deal with inconsistent DOM structures, ISP-level blocking, and aggressive anti-bot protections like Cloudflare.

Here is exactly how I built a robust scraping architecture to overcome these hurdles.

---

## 1. The Architecture: Express & Modular Routing

When scraping multiple sites, it's crucial to keep your codebase organized. Dumping all your scraping logic into a single file becomes unmaintainable very quickly.

For the Komiku API, I structured the application using a classic modular Express architecture:

```javascript
// server.js
const express = require('express');
const dotenv = require('dotenv');
const routes = require('./routes');
const asiaRoutes = require('./asia/routes');
const internationalRoutes = require('./international/routes');

dotenv.config();
const app = express();

app.use(express.json());

// Modular Routing based on region/source
app.use('/api', routes); // Komiku.id (Original)
app.use('/api/asia', asiaRoutes); // WestManga
app.use('/api/international', internationalRoutes); // WeebCentral

app.listen(4123, () => console.log('Scraper API running on port 4123'));
```

By separating routes and controllers per source, adding a new manga provider later on requires zero modifications to the existing, stable codebase.

---

## 2. Overcoming ISP Blocks with Custom DNS

Many manga and anime sites are subject to ISP-level DNS blocking in certain regions. If your Node.js server relies on the default system DNS, your scraping requests will fail with `ENOTFOUND` or timeout errors.

To bypass this programmatically without needing a system-wide VPN, you can force Node.js to use Cloudflare's public DNS (`1.1.1.1`):

```javascript
const dns = require('dns');

// Force Node.js to use Cloudflare DNS to bypass ISP blocking
dns.setServers(['1.1.1.1', '1.0.0.1']);
```
Placing this at the very top of your `server.js` ensures that all subsequent HTTP requests made by Axios or Cloudscraper resolve domains freely and globally.

---

## 3. Method 1: The Standard Approach (Axios + Cheerio)

For websites that serve static HTML without aggressive anti-bot protection, the combination of **Axios** (for fetching) and **Cheerio** (for parsing) is unmatched in speed and low memory footprint.

Cheerio provides a jQuery-like syntax that makes DOM traversal incredibly intuitive.

```javascript
const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeMangaList() {
    try {
        const url = 'https://komiku.id/';
        
        // 1. Fetch HTML source
        const { data } = await axios.get(url, {
            headers: {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
            }
        });
        
        // 2. Load HTML into Cheerio
        const $ = cheerio.load(data);
        const mangas = [];

// 3. Traverse the DOM
        $('.bge').each((index, element) => {
            const title = $(element).find('.kan h3').text().trim();
            const latestChapter = $(element).find('.kan .judul2').text().trim();
            const coverImage = $(element).find('.bgei img').attr('src');
            const endpoint = $(element).find('a').attr('href');

mangas.push({ title, latestChapter, coverImage, endpoint });
        });

return mangas;
    } catch (error) {
        console.error("Standard scraping failed:", error.message);
    }
    }
}
```

### How to Find the Right CSS Selectors
You might be wondering: *how did I know to use `$('.bge')` or `$('.kan h3')` in the code above?*

The secret is **Chrome Developer Tools (Inspect Element)**. Here is the step-by-step process:

1. Open your target website in your browser (e.g., `komiku.id`).
2. Right-click directly on the piece of data you want to extract (like the manga title or cover image) and click **Inspect** (or press `Ctrl+Shift+I`).
3. The Elements panel will open, highlighting the exact HTML tag for that text or image.
4. Look at the `class` or `id` attributes of that tag, and look at the parent container wrapping it.
5. In my case, I noticed that every single manga card on the page was wrapped inside a `<div class="bge">`. Inside that card, the title was sitting inside a `<div class="kan">` wrapped in an `<h3>`. 
6. Translate this visual structure into standard CSS Selectors: `$('.bge')` loops through all manga cards, and `.find('.kan h3')` digs into each specific card to extract the title text.

**Tip:** Always include a legitimate `User-Agent` string in your headers. Many basic firewalls immediately block requests originating from the default `axios/1.x` user agent.

---

## 4. Method 2: Bypassing Basic Cloudflare (Cloudscraper)

As a site grows, administrators often place it behind Cloudflare. When this happens, Axios requests will suddenly start returning `403 Forbidden` errors, or you will scrape a CAPTCHA challenge page instead of the actual content.

If the Cloudflare protection is set to "Under Attack" mode but doesn't strictly require full browser fingerprinting, `cloudscraper` is the perfect drop-in replacement for Axios.

```javascript
const cloudscraper = require('cloudscraper');
const cheerio = require('cheerio');

async function scrapeProtectedManga() {
    try {
        const url = 'https://komiku.id/';
        
        // Cloudscraper automatically handles cookies and JS challenges
        const html = await cloudscraper.get(url);
        
        const $ = cheerio.load(html);
        const title = $('.kan h3').first().text().trim();
        
        console.log("Successfully bypassed Cloudflare! Manga Title:", title);
    } catch (error) {
        console.error("Cloudscraper failed:", error.message);
    }
}
```

---

## 5. Method 3: The Heavy Artillery (Puppeteer)

When a website uses React, Vue, Next.js, or employs aggressive CAPTCHA/Turnstile protections, static fetching tools become useless. The data you want might not even exist in the initial HTML payload; it might be generated by JavaScript milliseconds later.

This is where **Puppeteer** comes in. Puppeteer launches an actual headless instance of Google Chrome, allowing you to execute scripts, wait for network calls, and simulate real user interactions.

```javascript
const puppeteer = require('puppeteer');
const cheerio = require('cheerio');

async function scrapeDynamicSite() {
    // 1. Launch a headless browser
    const browser = await puppeteer.launch({ 
        headless: true,
        args: ['--no-sandbox', '--disable-setuid-sandbox'] // Required for Linux servers
    });
    const page = await browser.newPage();
    
    // Set a realistic viewport and user agent
    await page.setViewport({ width: 1920, height: 1080 });
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');
    
    // 2. Visit the site and wait for all network requests to finish
    await page.goto('https://komiku.id/', { waitUntil: 'networkidle2' });
    
    // 3. Extract the fully rendered HTML
    const html = await page.content();
    
    // 4. Parse with Cheerio for faster data extraction
    const $ = cheerio.load(html);
    const title = $('.kan h3').first().text().trim();
    
    console.log("Scraping with Puppeteer successful! Manga:", title);
    
    // Always close the browser to free up RAM!
    await browser.close();
}
```

**Warning:** Puppeteer is incredibly resource-intensive. While Axios can handle hundreds of concurrent requests per second, launching hundreds of Chrome tabs will instantly crash your server due to Out-Of-Memory (OOM) errors. Use Puppeteer sparingly and implement a request queue system if necessary.

---

## 6. Best Practices for Production Scrapers

Building the scraper is only half the battle. Keeping it alive in production requires foresight.

### Caching is Mandatory
Do not scrape the target website every time a user requests your API. If 1,000 users request the latest manga list, you should only scrape the target site once. Store the result in a memory cache (like `node-cache` or `Redis`) with a TTL (Time To Live) of 5 to 15 minutes. This prevents your IP from getting banned for sending too many requests.

### Error Handling & Fallbacks
DOM structures change. Website administrators redesign their sites. Your Cheerio selectors (`.kan h3`) will inevitably break at some point. Ensure your API gracefully handles `TypeError`s and returns standard JSON error responses (`500 Internal Server Error`) instead of crashing the Node process.

### Respect the Target
Web scraping sits in a legal and ethical grey area. Always respect the site's `robots.txt` when possible, do not hammer their servers with thousands of requests per minute, and cache heavily to minimize the bandwidth you consume from their hosting.

---

## Conclusion

Building the Komiku API demonstrated that there is no "one size fits all" tool for web scraping.

Start small with **Axios and Cheerio** for blazing-fast data extraction. When firewalls block you, escalate to **Cloudscraper**. Finally, when faced with dynamic JavaScript rendering or impenetrable CAPTCHAs, bring out **Puppeteer**. By combining these tools intelligently with a modular architecture and aggressive caching, you can build reliable data APIs out of almost any website on the internet.

How I Scrape Data from Websites

Table of Contents