While web services are often the most efficient way to access large amounts of data on a regular and automatic basis, they do have capacity limitations.

Sometimes if you try to request large amounts of data from large LDS layers such as NZ Primary Parcels, you may encounter a HTTP 504 timeout error.

A timeout error occurs because the server has taken more than 300 seconds to return your response. You may be experiencing this timeout error if the web service request takes an extraordinary amount of time and still does not return any data or if you receive an error message.

If this occurs you have a couple of options:

Apply a filter to limit the amount of data in your request

The easiest way to limit the amount of data you are requesting from your web service query is to use one of the filter constraints capabilities featured in some GIS applications interfaces. These will let you limit the size of your web service request by either setting a limit on the maximum number of features that will be returned in the request, setting a scale dependency or other constraints. See our Using LDS Web Services guide for more information and instructions on these capabilities.

Figure 1: Using a Search Envelope in ArcGIS/FME to filter a WFS request.

XML filter for WFS screenshot

Figure 2: Using a complex XML filter for a WFS LDS layer in ArcGIS/FME.

If your client does not have an interface to easily apply the filter, another way to limit the amount of data in your web service request is to add a filter expression to your query to apply an attribute or bounding box filter. LDS WFS filters can be run by two different methods:

In this example we filter the Primary parcels layer to only return land parcels, that is, where the intent is not Road or Hydro.

CQL Filter

Note the HTTP "cql_filter" parameter:

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772&cql_filter=topology_type='Primary' AND parcel_intent <> 'Hydro' AND parcel_intent <> 'Road'

OGC Filter

Note the HTTP "filter" parameter:

https://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&VERSION=2.0&REQUEST=GetFeature&typeNames=layer-772&filter=<?xml version="1.0"?> 
<Filter xmlns="http://www.opengis.net/ogc" xmlns:v="http://data.linz.govt.nz/ns/v" xmlns:gml="http://www.opengis.net/gml">
<And>
<PropertyIsEqualTo>
<PropertyName>topology_type</PropertyName>
<Literal>Primary</Literal>
</PropertyIsEqualTo>
<PropertyIsNotEqualTo>
<PropertyName>parcel_intent</PropertyName>
<Literal>Hydro</Literal>
</PropertyIsNotEqualTo>
<PropertyIsNotEqualTo>
<PropertyName>parcel_intent</PropertyName>
<Literal>Road</Literal>
</PropertyIsNotEqualTo>
<BBOX>
<PropertyName>shape</PropertyName>
<gml:Box srsName="urn:x-ogc:def:crs:EPSG:4167">
<gml:coordinates>-36.764156,174.975230 -36.817467,175.023120</gml:coordinates>
</gml:Box>
</BBOX>
</And>
</Filter>

Use paging to request features/data in batches

Another way to avoid timeouts on web services requests to is request data in batches rather than downloading whole, large datasets in a single request.

WFS paging

WFS paging allows you to request the data in smaller batches using the STARTINDEX and COUNT parameters. We recommend that you request no more than 250,000 features in a request.

In this example, we request features in batches of 10,000:

1. Fetch first 0-9,999 features from NZ Primary parcels:

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=0&COUNT=10000

2. Then fetch the next 10,000-19,999 feature:

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=10000&COUNT=10000

3. Continue to request the data in batches until complete.

GDAL/OGR paging

Client Software such as GDAL/OGR supports paging using the OGR_WFS_PAGING_ALLOWED, OGR_WFS_PAGE_SIZE configuration options. For further information please see the GDAL/OGR paging documentation.

Note: The paging feature is WFS 2.0 feature and it not part of WFS 1.0.0 & 1.1.0 standards, however LDS support supports the features as a vendor extension.

Reference Category: Web services
Reference Tags: WFS error, Timeout, Paging, WFS filter
Last Updated: 
18 March 2019