Sometimes if you try to request large amounts of data from large LDS layers such as NZ Primary Parcels, you may encounter an HTTP 504 timeout error.
A timeout error occurs when the server takes more than 300 seconds to return your response. You may experience this timeout error if the web service request takes an extraordinary amount of time and still does not return any data, or if you receive an error message.
If this occurs you have a couple of options:
- Apply a filter to limit the amount of data in your request
- Use paging to request features/data in batches
Apply a filter to limit the amount of data in your request
The easiest way to limit the amount of data you are requesting is to use a filter constraint capability featured in some GIS applications interfaces. This will let you limit the size of your web service request by setting a limit on the maximum number of features that will be returned in the request, setting a scale dependency, or setting other constraints.
Using a complex XML filter for a WFS LDS layer in ArcGIS/FME.
You can also limit the amount of data in your web service request by adding a filter expression to your query to apply an attribute or bounding box filter. LDS WFS filters can be run by 2 different methods:
In this example we filter the Primary parcels layer to only return land parcels (that is, where the intent is not Road or Hydro).
CQL filter
Note the HTTP 'cql_filter' parameter:
http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772&cql_filter=topology_type='Primary' AND parcel_intent <> 'Hydro' AND parcel_intent <> 'Road'
OGC filter
Note the HTTP 'filter' parameter:
https://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&VERSION=2.0&REQUEST=GetFeature&typeNames=layer-772&filter=<?xml version="1.0"?>
<Filter xmlns="http://www.opengis.net/ogc" xmlns:v="http://data.linz.govt.nz/ns/v" xmlns:gml="http://www.opengis.net/gml">
<And>
<PropertyIsEqualTo>
<PropertyName>topology_type</PropertyName>
<Literal>Primary</Literal>
</PropertyIsEqualTo>
<PropertyIsNotEqualTo>
<PropertyName>parcel_intent</PropertyName>
<Literal>Hydro</Literal>
</PropertyIsNotEqualTo>
<PropertyIsNotEqualTo>
<PropertyName>parcel_intent</PropertyName>
<Literal>Road</Literal>
</PropertyIsNotEqualTo>
<BBOX>
<PropertyName>shape</PropertyName>
<gml:Box srsName="urn:x-ogc:def:crs:EPSG:4167">
<gml:coordinates>-36.764156,174.975230 -36.817467,175.023120</gml:coordinates>
</gml:Box>
</BBOX>
</And>
</Filter>
Use paging to request features/data in batches
Another way to avoid timeouts on web services requests to is request data in batches rather than downloading the entire large dataset in a single request.
WFS paging
WFS paging allows you to request the data in smaller batches using the STARTINDEX and COUNT parameters. We recommend you request no more than 250,000 features in a request.
In this example, we request features in batches of 10,000:
1. Fetch the first 10,000 features (0 to 9,999) from NZ Primary parcels:
http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=0&COUNT=10000
2. Then fetch the next 10,000 features (10,000 to 19,999):
http://data.linz.govt.nz/services;key=<your-API-key>/wfs?service=WFS&version=2.0.0&request=GetFeature&typeNames=layer-772& STARTINDEX=10000&COUNT=10000
3. Continue to request the data in batches until complete.
GDAL/OGR paging
Client Software such as GDAL/OGR supports paging using the OGR_WFS_PAGING_ALLOWED, OGR_WFS_PAGE_SIZE configuration options. For further information please see the GDAL/OGR paging documentation.
Note that the paging feature is a WFS 2.0 feature. It is not part of the WFS 1.0.0 or 1.1.0 standards, but LDS supports the features as a vendor extension.