Web Connect reference

This reference documents the configuration options of Web Connect in Adverity.

Authorization methods

There are various authorization methods available when adding a Web Connect authorization. All required fields for each authorization method are marked with an asterisk * in the Adverity user interface. The available authorization methods are the following:

Use Web Connect for APIs with basic authorization (username and password), static bearer tokens, cookie sessions, and header authentication.
Use Web Connect (Bearer) for APIs that require you to obtain a Bearer Token that is retrieved after a call to an authorization endpoint.
Use Web Connect (OAuth) for APIs that use the secure authorization Oauth (Open Authorization).
Use Web Connect (OAuth2 Authorization Code) for APIs that use the secure authorization Open Authorization 2.0 (OAuth 2.0 ).

Understanding Web Connect configuration fields

This section documents the most important fields in the Web Connect configuration page.

Request

URL

Enter the URL of the API, including the required parameters and values.

Value tables parsed in request URL

Parse a value table in the request URL. This is typically used for lists of IDs, where you would need to manually change the ID in the URL between every call. The syntax you can use is similar to the following:
```
{vt(valuetablename)}
```
In this code, vt signifies that the placeholder in curly brackets is a value table. Replace valuetablename with the name of your value table.

For more information on automatically populating value tables, see Automatically populate a value table.
Start and end parameters

With regular API calls, you can hard code the date range for which you request the data, usually through URL or body parameters (for example, with the date_from and date_to parameters). To dynamically change the date range of the fetch, include the {start} and {end} parameters in the URL or the request body. Part of the request URL will be similar to the following:
```
from={start}&to={end}
```
With this configuration, you can change the fetch date range using the date range selector as in other connectors.

To enable custom date format options, edit them in the Date format field below.

Request method

Select either the GET or POST HTTP request method, and configure the request header in JSON format. POST requests allow you to enter an HTTP request body.

Date format

Edit the date format required by the API. To use the Date format field, include {start} and {end} parameters in the URL or request body.

Most APIs use the ISO8601 format (YYYY-MM-DD), but the options may be specific for an individual API, for example DD/MM/YYYY or MMDDYYYY.

%m returns the month as a number, from 01 to 12. This is similar to the regularly used mm format.

%d returns the day as a number, from 01 to 31. This is similar to the regularly used dd format.

%Y returns the year in four-digit format. This is similar to the regularly used yyyy format.

To define a date format similar to DD/MM/YYYY, enter %d/%m/%Y into the Date format field.

For more information on available formats, see Unix reference.

Navigation URLs

Include a list of URLs that are called before fetching the data. Utilize this option if you need to initiate a request before you fetch data.

Pagination Options

API pagination divides the responses into several pages. To return a large amount of data in smaller pages, APIs make use of pagination. For example, instead of 10,000 lines of JSON, you can have 10 pages with 1,000 lines of JSON on each page. To retrieve all the data from an API, all pages are fetched one after another. These pages are then combined into one extract. Adverity's Web Connect can fetch all pages for various API pagination options.

For a more general explanation of pagination options, see Everything You Need to Know About API Pagination.

Select from the following options:

Page & PageSize Pagination

Specify the page you want to display in the Page parameter and set the number of items for each page to display in the PageSize parameter.

For example, there are 10 items in total, and you are interested in items 6, 7, and 8. If you set the PageSize parameter to 3, the pages are as follows:

Page parameter 0 returns items 0, 1, and 2.
Page parameter 1 returns items 3, 4, and 5.
Page parameter 2 returns items 6, 7, and 8.

Hence, setting the Page parameter to 2 and the PageSize parameter to 3 returns the items of initial interest (6, 7, and 8). However, in this manner, you retrieve only one page per request.

To retrieve all pages automatically, follow these steps:

In Paging Type, select Page & PageSize Pagination.
In Offset Parameter Keyword, set the offset parameter specified by the API (for example, page, offset, skip). For GET requests, you may need to include this parameter in the request URL as well. In POST requests, you may need to include this parameter in the request URL or HTML request body.
In Limit Parameter Keyword, include a parameter which sets a limit to how many items you retrieve on each page (for example, PageSize, limit, or take). For GET requests, you may need to include this parameter as well in the request URL. In POST requests, you may need to include this parameter in the request URL or HTML request body.
In Increment, set how many items to retrieve on each page. For example, if you use 100, each page includes 100 rows of data. It is recommended to set increment number high, as fewer rows for each page can mean many API calls. For example, if you set the increment to 1 and retrieve 100 rows of data, each row is on a different page, generating 100 API calls.

Page Pagination

This paging type is similar to Page & PageSize Pagination. However, in Page Pagination, you cannot change the PageSize parameter. You can only define the Page parameter.

Offset & Limit Paging

The Limit Paging parameter allows you to limit the number of rows returned in a query. The Offset parameter allows you to omit a specified number of rows before the beginning of the result. Using both limit and offset skips both rows as well as limits the rows returned.

For example, there are 10 items in total, and you are interested in items 6, 7, and 8. To retrieve all pages automatically, follow these steps:

Set the Limit Paging parameter to value 3. This configuration displays only 3 items.
Set the Offset parameter to value 6, since you need to skip the first 6 items.

Hence, setting the Limit Paging parameter to 3 and the Offset parameter to value 6 returns the items of initial interest (6, 7, and 8). However, in this manner, you retrieve only one page per request.

To retrieve all pages automatically, follow these steps:

In Paging Type, select Offset & Limit Paging.
In Offset Parameter Keyword, set the offset parameter specified by the API (for example, page, offset, skip). For GET requests, you may need to include this parameter in the request URL as well. In POST requests, you may need to include this parameter in the request URL or HTML request body.
In Limit Parameter Keyword, include a parameter that limits how many items you retrieve on each page (for example, PageSize, limit, or take). For GET requests, you may need to include this parameter as well in the request URL. In POST requests, you may need to include this parameter in the request URL or HTML request body.
In Increment, set how many items to retrieve in each page. For example, if you use 100, each page includes 100 rows of data. It is recommended to set increment number high, as less rows for each page can mean many API calls. For example, if you set the increment to 1 and retrieve 100 rows of data, each row is on a different page, generating 100 API calls.

JSON:API Cursor Paging

The cursor paging option returns a pointer to a specific result within the retrieved data. The retrieved cursor is a string value.

To retrieve all pages automatically, follow these steps:

In JSON:API Cursor Parameter Keyword, enter the parameter which is the cursor pointer for the specific API. If this parameter is called cursor, enter cursor.
In JSONPath, enter the JSONPath of the cursor parameter. For more information, see Using JSONPath

JSON:API Link Paging

Link pagination returns a URL to the next page in the result of the request. Make a new request to this URL to fetch the next page.

To retrieve all pages automatically, enter the JSONPath of the next page parameter in the JSONPath field (the value of this parameter is usually a URL). For more information, see Using JSONPath.

Response

Zip match

If you are fetching data in a compressed file format (for example, zip) that contains multiple CSV files, you can define a regular expression to configure which file within the compressed file is fetched. As a result, the fetch retrieves just a file you define uncompressed.

For example, the following URL from the Austrian government's website lets you download various CSV files in one compressed file:

https://covid19-dashboard.ages.at/data/data.zip

Enter CovidFaelle_Altersgruppe.csv in the Zip match field and fetch the data.

As a result, you obtain the CovidFaelle_Altersgruppe.csv uncompressed file.

Archive Password

If your data is in an encrypted zip file, provide the password in the Archive Password field to decrypt it automatically.

Parser

Adverity currently supports the following parsers:

CSV (with comma delimiter)
CSV (with semicolon delimiter)
TSV (tabs)
Excel
JSON (strict)
HTML
XML
JSON

Difference between JSON and JSON (strict): A valid response (HTML code 200) can be empty, or the response does not contain the specified JSONPath expression data. In such cases, Adverity raises an error. In some cases, you may not desire to see these errors. Depending on which JSON parser you select, these errors are either raised or not in the following way:

JSON ignores errors.
JSON (strict) raises an error when a specified JSONPath does not contain data or a valid response (code 200) is empty.

Encoding

Adverity supports the following encoding types:

Auto-detect
UTF-8
UTF-8 (with BOM)
UTF-16
UTF-16 (Little Endian)
ISO-8859-1
ISO-8859-9
ISO-8859-15
Shift-JIS
Code page 437
Windows-1250
Windows-1251
Windows-1252
UTF-8 (with embedded garbage)
ISO-8859-1 (with embedded garbage)

Data key

Define which part of the response holds data necessary for you.

For JSON responses, define a JSONPath. For more information, see Using JSONPath.
For XML and HTML parsers, provide the XPath for rows. For more information, see the W3 XPath documentation.
For XLSX parsers, this is the name of the sheet that you are importing.

To further request only specific columns, use the Column mapping described below.

Column mapping

In combination with Data key, the Column mapping option allows you to request only specific columns from the API. To define which columns of a JSON or XML response are extracted with column mapping, use one of the following:

For JSON responses, define a JSONPath. For more information, see Using JSONPath.
For XML responses, use XPath pairs of "column-name": "expression". For more information, see the W3 XPath documentation.

Appendix

Using JSONPath

The JSONPath enables you to specify where a particular parameter or its value is located. This means that you can filter for one specific value in a JSON response or filter for specific parameters and their values. This is a general feature available for any text formatted in JSON with correct syntax. See the example below to help you understand this concept:

Visit JSONPath.com.

Paste the following code snippet into the left window displayed on the JSONPath.com website:

{ "store": {
    "book": [ 
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      { "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      { "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

To only filter for all book titles in the response, enter the following code to the JSONPath Syntax field at the top of the page:
```
$.store.book[*]
```

As a result, you used the JSONPath to filter for a specific part of the response. For more information, see JSON with JSONPath from REST API Tutorials.