Web services are often an efficient way to access data on a regular and automatic basis, but HTTP 504 timeout errors can occur if you request large amounts of data from large LDS layers.

Sometimes if you try to request large amounts of data from large LDS layers such as NZ Primary Parcels, you may encounter an HTTP 504 timeout error.

A timeout error occurs when the server takes more than 300 seconds to return your response. You may experience this timeout error if the web service request takes an extraordinary amount of time and still does not return any data, or if you receive an error message.

If this occurs you have a couple of options:

Apply a filter to limit the amount of data in your request

The easiest way to limit the amount of data you are requesting is to use a filter constraint capability featured in some GIS applications interfaces. This will let you limit the size of your web service request by setting a limit on the maximum number of features that will be returned in the request, setting a scale dependency, or setting other constraints. 

Screenshot of an XML filter being coded

Using a complex XML filter for a WFS LDS layer in ArcGIS/FME.

You can also limit the amount of data in your web service request by adding a filter expression to your query to apply an attribute or bounding box filter. LDS WFS filters can be run by 2 different methods:

In this example we filter the Primary parcels layer to only return land parcels (that is, where the intent is not Road or Hydro).

CQL filter

Note the HTTP 'cql_filter' parameter:

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772&cql_filter=topology_type='Primary' AND parcel_intent <> 'Hydro' AND parcel_intent <> 'Road'

OGC filter

Note the HTTP 'filter' parameter:

https://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&VERSION=2.0&REQUEST=GetFeature&typeNames=layer-772&filter=<?xml version="1.0"?>  <Filter xmlns="http://www.opengis.net/ogc" xmlns:v="http://data.linz.govt.nz/ns/v" xmlns:gml="http://www.opengis.net/gml"> <And> <PropertyIsEqualTo> <PropertyName>topology_type</PropertyName> <Literal>Primary</Literal> </PropertyIsEqualTo> <PropertyIsNotEqualTo> <PropertyName>parcel_intent</PropertyName> <Literal>Hydro</Literal> </PropertyIsNotEqualTo> <PropertyIsNotEqualTo> <PropertyName>parcel_intent</PropertyName> <Literal>Road</Literal> </PropertyIsNotEqualTo> <BBOX> <PropertyName>shape</PropertyName> <gml:Box srsName="urn:x-ogc:def:crs:EPSG:4167"> <gml:coordinates>-36.764156,174.975230 -36.817467,175.023120</gml:coordinates> </gml:Box> </BBOX> </And> </Filter>

Use paging to request features/data in batches

Another way to avoid timeouts on web services requests to is request data in batches rather than downloading the entire large dataset in a single request.

WFS paging

WFS paging allows you to request the data in smaller batches using the STARTINDEX and COUNT parameters. We recommend you request no more than 250,000 features in a request.

In this example, we request features in batches of 10,000:

1. Fetch the first 10,000 features (0 to 9,999) from NZ Primary parcels:

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=0&COUNT=10000

2. Then fetch the next 10,000 features (10,000 to 19,999):

http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=10000&COUNT=10000

3. Continue to request the data in batches until complete.

GDAL/OGR paging

Client Software such as GDAL/OGR supports paging using the OGR_WFS_PAGING_ALLOWED, OGR_WFS_PAGE_SIZE configuration options. For further information please see the GDAL/OGR paging documentation.

Note that the paging feature is a WFS 2.0 feature. It is not part of the WFS 1.0.0 or 1.1.0 standards, but LDS supports the features as a vendor extension.