> For the complete documentation index, see [llms.txt](https://docs.iproyal.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.iproyal.com/scraping-tools/web-scraper-api-beta/api/scraping.md).

# Scraping

Scrapes a web page, Google search results, or an image, and returns the content.

```
POST /v1/scrape
```

Each successful request costs one credit.

### Request Body

| Parameter           | Type    | Required                    | Description                                                                                                                                                                                                                                                  |
| ------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `source`            | string  | Yes                         | What to scrape: `universal`, `google`, or `image`                                                                                                                                                                                                            |
| `url`               | string  | For `universal` and `image` | The full target URL, starting with `http://` or `https://`. Up to 2,048 characters.                                                                                                                                                                          |
| `query`             | string  | For `google`                | The search keywords. Cannot be empty. Up to 500 characters.                                                                                                                                                                                                  |
| `geo_location`      | string  | No                          | Two-letter country code in the ISO 3166-1 alpha-2 format, such as `us` or `de`. Not case-sensitive. [See Supported Countries](https://docs.iproyal.com/scraping-tools/web-scraper-api-beta/supported-countries.md).                                          |
| `parse`             | boolean | No                          | For `universal` only. Set to `true` to receive structured JSON instead of HTML. Defaults to `false`.                                                                                                                                                         |
| markdown            | boolean | No                          | `universal` only. `true` returns the page as Markdown instead of HTML. Can't be combined with `parse`. Defaults to `false`.                                                                                                                                  |
| main\_content\_only | boolean | No                          | Only with `markdown: true`. `true` keeps only the main content, without navigation, site header, footer and sidebars. Defaults to `false` (the whole page).                                                                                                  |
| full\_urls          | boolean | No                          | `google` only. `true` waits until Google provides every result's real URL, which can add up to about 40 seconds. Without it, a result whose link isn't ready yet gets the site's home page or a Google redirect link. Same credit cost. Defaults to `false`. |

Send only the fields your source uses. Fields not listed here are ignored.

{% tabs %}
{% tab title="cURL" %}

```bash
curl -X POST https://scrape.iproyal.com/v1/scrape \
  -H "Authorization: Bearer $WEB_SCRAPER_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"source": "universal", "url": "https://example.com"}' \
  --max-time 90
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php

$ch = curl_init('https://scrape.iproyal.com/v1/scrape');

curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 90,
    CURLOPT_HTTPHEADER => [
        'Authorization: Bearer ' . getenv('WEB_SCRAPER_API_TOKEN'),
        'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS => json_encode([
        'source' => 'universal',
        'url' => 'https://example.com',
    ]),
]);

$body = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if ($status === 200) {
    $result = $body['results'][0];
    echo $result['status_code'] . ' ' . $result['job_id'] . PHP_EOL;
    $html = $result['content'];
} else {
    echo $status . ' ' . $body['error']['code'] . ': ' . $body['error']['message'] . PHP_EOL;
}
```

{% endtab %}

{% tab title="Python" %}

```python
import os

import requests

response = requests.post(
    "https://scrape.iproyal.com/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['WEB_SCRAPER_API_TOKEN']}"},
    json={"source": "universal", "url": "https://example.com"},
    timeout=90,
)

if response.ok:
    result = response.json()["results"][0]
    print(result["status_code"], result["job_id"])
    html = result["content"]
else:
    error = response.json()["error"]
    print(response.status_code, error["code"], error["message"])
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
const response = await fetch('https://scrape.iproyal.com/v1/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.WEB_SCRAPER_API_TOKEN}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ source: 'universal', url: 'https://example.com' }),
  signal: AbortSignal.timeout(90_000),
});

const body = await response.json();

if (response.ok) {
  const [result] = body.results;
  console.log(result.status_code, result.job_id);
  const html = result.content;
} else {
  console.error(response.status, body.error.code, body.error.message);
}
```

{% endtab %}

{% tab title="Java" %}

```java
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Map;

public class ScrapeExample {
    public static void main(String[] args) throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        String payload = mapper.writeValueAsString(Map.of(
            "source", "universal",
            "url", "https://example.com"
        ));

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://scrape.iproyal.com/v1/scrape"))
            .header("Authorization", "Bearer " + System.getenv("WEB_SCRAPER_API_TOKEN"))
            .header("Content-Type", "application/json")
            .timeout(Duration.ofSeconds(90))
            .POST(HttpRequest.BodyPublishers.ofString(payload))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());
        JsonNode body = mapper.readTree(response.body());

        if (response.statusCode() == 200) {
            JsonNode result = body.get("results").get(0);
            System.out.println(result.get("status_code").asInt() + " " + result.get("job_id").asText());
            String html = result.get("content").asText();
        } else {
            JsonNode error = body.get("error");
            System.out.println(response.statusCode() + " " + error.get("code").asText() + ": " + error.get("message").asText());
        }
    }
}
```

{% endtab %}

{% tab title="Go" %}

```go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"time"
)

type apiResponse struct {
	Results []struct {
		Content    json.RawMessage `json:"content"`
		JobID      string          `json:"job_id"`
		StatusCode int             `json:"status_code"`
	} `json:"results"`
	Error struct {
		Code    string `json:"code"`
		Message string `json:"message"`
	} `json:"error"`
}

func main() {
	payload, _ := json.Marshal(map[string]any{
		"source": "universal",
		"url":    "https://example.com",
	})

	req, _ := http.NewRequest(http.MethodPost, "https://scrape.iproyal.com/v1/scrape", bytes.NewReader(payload))
	req.Header.Set("Authorization", "Bearer "+os.Getenv("WEB_SCRAPER_API_TOKEN"))
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{Timeout: 90 * time.Second}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println("request failed:", err)
		return
	}
	defer resp.Body.Close()

	var body apiResponse
	if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
		fmt.Println("unexpected response:", err)
		return
	}

	if resp.StatusCode != http.StatusOK {
		fmt.Println(resp.StatusCode, body.Error.Code, body.Error.Message)
		return
	}

	result := body.Results[0]
	fmt.Println(result.StatusCode, result.JobID)

	var html string
	json.Unmarshal(result.Content, &html)
}
```

{% endtab %}

{% tab title="C#" %}

```csharp
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(90) };
client.DefaultRequestHeaders.Authorization =
    new AuthenticationHeaderValue("Bearer", Environment.GetEnvironmentVariable("WEB_SCRAPER_API_TOKEN"));

var response = await client.PostAsJsonAsync(
    "https://scrape.iproyal.com/v1/scrape",
    new { source = "universal", url = "https://example.com" });

using var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync());

if (response.IsSuccessStatusCode)
{
    var result = body.RootElement.GetProperty("results")[0];
    Console.WriteLine($"{result.GetProperty("status_code").GetInt32()} {result.GetProperty("job_id").GetString()}");
    var html = result.GetProperty("content").GetString();
}
else
{
    var error = body.RootElement.GetProperty("error");
    Console.WriteLine($"{(int)response.StatusCode} {error.GetProperty("code").GetString()}: {error.GetProperty("message").GetString()}");
}
```

{% endtab %}
{% endtabs %}

### Sources

#### Web Page (`universal`)

Returns any public web page. Pages load in a real browser, so content that JavaScript adds is included. `content` holds the page HTML as a string.

```json
{ "source": "universal", "url": "https://example.com/products/42", "geo_location": "gb" }
```

**Structured Data**

Set `parse` to `true`, and IPRoyal extracts structured data from the page. `content` then holds a JSON object instead of HTML. Its structure depends on the page.

```json
{ "source": "universal", "url": "https://example.com/products/42", "parse": true }
```

If the page has no structured data to extract, the request fails with `422 no_data_extracted` and is not charged. Retry without `parse` to get the HTML.

`parse` works only with `universal`. Sending it with another source returns `400 invalid_request`.

**Markdown**

Set `markdown` to `true`, and it returns the page as Markdown instead of HTML. You get the whole page, including navigation, header and footer. To keep only the main content, also set `main_content_only` to `true`. `markdown` can't be combined with `parse`.

```json
{
  "source": "universal",
  "url": "https://example.com/blog/getting-started",
  "markdown": true
}
```

```json
{
  "source": "universal",
  "url": "https://example.com/blog/getting-started",
  "markdown": true,
  "main_content_only": true
}
```

#### Google Search (`google`)

Returns one page of Google search results. `content` holds the results as a JSON object, and `url` in the response is `null`.

Use `geo_location` to see the results a searcher in that country would see.

```json
{ "source": "google", "query": "running shoes", "geo_location": "us" }
```

**Full Result URLs**

Google reveals result links a few at a time, and the last ones can arrive up to about 40 seconds after the search. By default, IPRoyal doesn't wait for them: a result whose link isn't ready yet gets the site's home page as its `url`, or a Google redirect link if no site is shown. Set `full_urls` to `true` to wait until every result has its real URL. The request takes longer but costs the same.

```json
{
  "source": "google",
  "query": "running shoes",
  "full_urls": true
}
```

Only the links inside `content` change. The result's `url` stays `null` for Google. The response examples below are shortened to one organic result.

Without `full_urls`, a result whose link wasn't ready yet:

```json
{
  "results": [
    {
      "content": {
        "keyword": "running shoes",
        "organic": [
          {
            "position": 3,
            "title": "Trail Running Shoes",
            "url": "https://example.com/",
            "domain": "example.com"
          }
        ]
      },
     ....
    }
  ]
}
```

With `full_urls`:

```json
{
  "results": [
    {
      "content": {
        "keyword": "running shoes",
        "organic": [
          {
            "position": 3,
            "title": "Trail Running Shoes",
            "url": "https://example.com/running-shoes/trail",
            "domain": "example.com"
          }
        ]
      },
     ....
    }
  ]
}
```

#### Image (`image`)

Downloads an image from its URL. Use it for images that a website does not let you download directly. It does not take screenshots of web pages.

Use `geo_location` to download the image as a visitor from that country.

```json
{ "source": "image", "url": "https://example.com/images/photo.jpg", "geo_location": "us" }
```

`content` holds the image file as a Base64 string. Decode it to get the file. The cURL example uses `jq`.

{% tabs %}
{% tab title="cURL" %}

```bash
curl -sS -X POST https://scrape.iproyal.com/v1/scrape \
  -H "Authorization: Bearer $WEB_SCRAPER_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"source": "image", "url": "https://example.com/images/photo.jpg"}' \
  --max-time 90 \
  | jq -r '.results[0].content' \
  | base64 --decode > photo.jpg
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php

$ch = curl_init('https://scrape.iproyal.com/v1/scrape');

curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 90,
    CURLOPT_HTTPHEADER => [
        'Authorization: Bearer ' . getenv('WEB_SCRAPER_API_TOKEN'),
        'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS => json_encode([
        'source' => 'image',
        'url' => 'https://example.com/images/photo.jpg',
    ]),
]);

$body = json_decode(curl_exec($ch), true);

file_put_contents('photo.jpg', base64_decode($body['results'][0]['content']));
```

{% endtab %}

{% tab title="Python" %}

```python
import base64
import os

import requests

response = requests.post(
    "https://scrape.iproyal.com/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['WEB_SCRAPER_API_TOKEN']}"},
    json={"source": "image", "url": "https://example.com/images/photo.jpg"},
    timeout=90,
)
response.raise_for_status()

with open("photo.jpg", "wb") as file:
    file.write(base64.b64decode(response.json()["results"][0]["content"]))
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
import { writeFile } from 'node:fs/promises';

const response = await fetch('https://scrape.iproyal.com/v1/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.WEB_SCRAPER_API_TOKEN}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ source: 'image', url: 'https://example.com/images/photo.jpg' }),
  signal: AbortSignal.timeout(90_000),
});

const { results } = await response.json();
await writeFile('photo.jpg', Buffer.from(results[0].content, 'base64'));
```

{% endtab %}

{% tab title="Java" %}

```java
import com.fasterxml.jackson.databind.ObjectMapper;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Base64;
import java.util.Map;

public class ImageExample {
    public static void main(String[] args) throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        String payload = mapper.writeValueAsString(Map.of(
            "source", "image",
            "url", "https://example.com/images/photo.jpg"
        ));

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://scrape.iproyal.com/v1/scrape"))
            .header("Authorization", "Bearer " + System.getenv("WEB_SCRAPER_API_TOKEN"))
            .header("Content-Type", "application/json")
            .timeout(Duration.ofSeconds(90))
            .POST(HttpRequest.BodyPublishers.ofString(payload))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());
        String content = mapper.readTree(response.body()).get("results").get(0).get("content").asText();

        Files.write(Path.of("photo.jpg"), Base64.getDecoder().decode(content));
    }
}
```

{% endtab %}

{% tab title="Go" %}

```go
package main

import (
	"bytes"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"time"
)

func main() {
	payload, _ := json.Marshal(map[string]any{
		"source": "image",
		"url":    "https://example.com/images/photo.jpg",
	})

	req, _ := http.NewRequest(http.MethodPost, "https://scrape.iproyal.com/v1/scrape", bytes.NewReader(payload))
	req.Header.Set("Authorization", "Bearer "+os.Getenv("WEB_SCRAPER_API_TOKEN"))
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{Timeout: 90 * time.Second}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println("request failed:", err)
		return
	}
	defer resp.Body.Close()

	var body struct {
		Results []struct {
			Content string `json:"content"`
		} `json:"results"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&body); err != nil || len(body.Results) == 0 {
		fmt.Println("unexpected response, HTTP", resp.StatusCode)
		return
	}

	image, err := base64.StdEncoding.DecodeString(body.Results[0].Content)
	if err != nil {
		fmt.Println("invalid Base64:", err)
		return
	}

	if err := os.WriteFile("photo.jpg", image, 0o644); err != nil {
		fmt.Println("could not save the file:", err)
	}
}
```

{% endtab %}

{% tab title="C#" %}

```csharp
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(90) };
client.DefaultRequestHeaders.Authorization =
    new AuthenticationHeaderValue("Bearer", Environment.GetEnvironmentVariable("WEB_SCRAPER_API_TOKEN"));

var response = await client.PostAsJsonAsync(
    "https://scrape.iproyal.com/v1/scrape",
    new { source = "image", url = "https://example.com/images/photo.jpg" });
response.EnsureSuccessStatusCode();

using var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
var content = body.RootElement.GetProperty("results")[0].GetProperty("content").GetString()!;

await File.WriteAllBytesAsync("photo.jpg", Convert.FromBase64String(content));
```

{% endtab %}
{% endtabs %}

If the page at the URL has no image, the request fails with `404 not_found` and is not charged.

### Response

```json
{
  "results": [
    {
      "content": "<!doctype html><html lang=\"en\">...</html>",
      "created_at": "2026-10-06 09:15:02",
      "updated_at": "2026-10-06 09:15:07",
      "page": 1,
      "url": "https://example.com",
      "job_id": "dc86c742-9843-4593-a796-1d90db73f5e5",
      "status_code": 200
    }
  ]
}
```

| Field         | Type             | Description                                                                                                                                                        |
| ------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `results`     | array            | The result of the request. It always holds one item.                                                                                                               |
| `content`     | string or object | HTML string for `universal`, Markdown string for `universal` with `markdown`, JSON object for `universal` with `parse` and for `google`, Base64 string for `image` |
| `created_at`  | string           | When processing started, in UTC, as `YYYY-MM-DD HH:MM:SS`                                                                                                          |
| `updated_at`  | string           | When processing finished, in UTC                                                                                                                                   |
| `page`        | integer          | Always `1`                                                                                                                                                         |
| `url`         | string or null   | The URL you requested. `null` for `google`.                                                                                                                        |
| `job_id`      | string           | Unique ID of the request. The same ID appears in [Usage History](https://docs.iproyal.com/scraping-tools/web-scraper-api-beta/api/usage-history.md).               |
| `status_code` | integer          | The HTTP status the target website returned                                                                                                                        |

#### When the Target Page Does Not Exist

If the target website answers with `404` or `410`, IPRoyal still delivers its page. You receive HTTP `200`, and `status_code` shows `404` or `410`. The request is charged, because the page was retrieved.

Check `status_code` before you use `content`.

### Response Time

Response time depends on the target website. In rare cases, a request takes up to 75 seconds. Google requests with `full_urls` take longer, because they wait for every result's real URL. Set your HTTP client timeout to at least 90 seconds.

### Errors

| HTTP Status    | `error.code`                              | When                                                 |
| -------------- | ----------------------------------------- | ---------------------------------------------------- |
| `400`          | `invalid_request`                         | A field is missing or not valid                      |
| `401`          | `invalid_token`                           | The token is missing or not recognized               |
| `402`          | `insufficient_credit`                     | Your credit balance is empty                         |
| `403`          | `account_suspended` or `account_disabled` | Your account cannot make requests                    |
| `404`          | `not_found`                               | No image was found at the URL (`image` only)         |
| `413`          | `payload_too_large`                       | The request body is larger than 100 KB               |
| `422`          | `blocked`                                 | The target website blocked the request               |
| `422`          | `no_data_extracted`                       | `parse` found no structured data                     |
| `429`          | `concurrency_limit_exceeded`              | Too many of your scrape requests are running at once |
| `500` or `503` | `internal_error`                          | A temporary problem on IPRoyal's side                |
| `502`          | `upstream_unavailable`                    | The scrape could not be completed                    |
| `503`          | `capacity_exceeded`                       | The service is at full capacity                      |

Requests that end in an error are not charged. [API](https://docs.iproyal.com/scraping-tools/web-scraper-api-beta/api.md) explains what to do for each code.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.iproyal.com/scraping-tools/web-scraper-api-beta/api/scraping.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
