Reverse Engineering Private APIs for Faster, Cleaner Scraping

The fastest scraper is often not a scraper at all. Most modern websites load their data from an internal API, then render it in the browser. If you call that API directly, you skip the HTML parsing, the browser overhead, and a lot of the anti-bot friction. This guide shows how to find and use those private APIs.

Why APIs beat HTML scraping

When a site fetches its data from a backend endpoint, that endpoint usually returns clean JSON. Calling it directly has big advantages over scraping rendered HTML:

• Structured data. JSON with named fields, no fragile CSS selectors.
• Speed. A single HTTP request instead of launching a browser and rendering a page.
• Stability. Internal APIs change less often than visual markup.
• Efficiency. No proxy bandwidth wasted on images, CSS, and fonts.

The catch is that these APIs are undocumented and not meant for public use, so you have to discover them and figure out how they work.

Finding the API in network traffic

Open the site in your browser, open DevTools, and go to the Network tab. Filter to Fetch/XHR. As you interact with the page, watch the requests that return the data you want.

What you are looking for:

• A request whose response is JSON containing the data shown on the page.
• The full URL, including query parameters.
• The request method and any headers, especially auth tokens.
• The payload, if it is a POST.

Once you find the right request, DevTools lets you right click and "Copy as cURL," which gives you the exact request with all headers. That is your starting point.

Replicating the request in Python

Translate the copied request into code. Start with everything the browser sent, then trim it down to what is actually required.

import requests

resp = requests.get(
    "https://example.com/api/v2/products",
    params={"category": "electronics", "page": 1},
    headers={
        "Accept": "application/json",
        "User-Agent": "Mozilla/5.0 ...",
        "Referer": "https://example.com/category/electronics",
    },
    timeout=20,
)
data = resp.json()
for product in data["results"]:
    print(product["name"], product["price"])

The pagination is usually a simple parameter, so iterating every page is a clean loop instead of clicking through a rendered interface.

Trimming headers to what matters

The browser sends many headers, but most are not required. Remove them one at a time and see what breaks. Usually only a few matter:

• Authorization or API key headers. These are mandatory if present.
• Referer or Origin. Some APIs check these to block off site calls.
• User-Agent. Some reject default library agents, so set a browser like value.
• A custom token header. Sites often add a header like x-api-token their frontend generates.

Knowing the minimal set keeps your requests clean and makes them less fragile.

Handling authentication

Private APIs use a few common auth patterns, and each has a way to handle it:

• Bearer token in a header. Capture it from a logged in session and include it. Tokens expire, so you may need to refresh them.
• Session cookies. Log in once with a browser or a login request, then reuse the cookie jar for API calls.
• A token generated by frontend JavaScript. The hardest case. Sometimes you can replicate the token logic, and sometimes you need a real browser to generate it, then hand it to your API calls.

session = requests.Session()
session.post("https://example.com/api/login", json={
    "email": "[email protected]",
    "password": "secret",
})
# The session now holds the auth cookie for subsequent calls
data = session.get("https://example.com/api/v2/orders").json()

When the token comes from JavaScript

Some sites sign each request with a token computed in obfuscated JavaScript. You have two options. Replicate the algorithm in your own code if it is simple enough to read, which is fragile but fast. Or run a real browser to load the page, extract the token it generates, and feed it into your direct API calls, which is more robust. The hybrid approach, a browser for the token and plain HTTP for the data, often gives the best of both.

Respecting limits and staying reliable

A private API is still subject to rate limiting and can still ban you. Apply the same discipline as any scraper: rotate proxies if needed, throttle your rate, and add retries with backoff. See my guides on rotating proxies and running scrapers in production.

Watch for the API changing. Internal APIs are more stable than markup but not permanent. A version bump in the path or a new required header will break your client, so monitor your success rate.

When this approach does not fit

Direct API access is not always possible. Some sites render everything server side with no JSON endpoint, in which case you are back to HTML scraping. Others sign requests so heavily that replicating the auth is more work than just driving a browser. Inspect first, and pick the cheaper path for that specific site.

Need an API integration or reverse engineering done?

I find and integrate private and undocumented APIs to build fast, clean data pipelines, and fall back to browser scraping when an API is not available. If you have a project, hire me on Upwork or reach out through the contact form. I respond within 24 hours.