Problem: Requests to WFS services using version 2.0 result in requests timing out or exceedingly slow response times.
Affects: WFS version 2.0 only. 1.0 and 1.1 are unaffected.
Description: Requests are exceedingly slow or generating timeout errors due to an ArcGIS Server 10.31 bug that causes WFS requests to ignore the enforced service feature limits (aka maximum record count) and try to return the entire dataset in one request.
This problem typically only occurs for larger datasets with hundreds of thousands of records, or for smaller datasets with complex or dense geometries (e.g. nature landform datasets).
A fix for this bug is being investigated.
For more information please see All Software: Requests Timing Out.
Workaround: There are three approaches to working around this issue:
- Use the ArcGIS REST API - Both MapServer and FeatureServer endpoints expose an API that is largely feature equivalent to WFS 2.0, as well as including a range of more advanced functionality for querying and filtering datasets. Importantly in this case - software using the ArcGIS REST API implement paging (making multiple smaller requests to download a dataset) and the the API respects the service feature limits in place.
This is the recommended approach if your software supports it.
If using FME, please refer to our How To: Download Geospatial Data using FME article for an approach that avoids using WFS. We're planning to write articles similar to this one for some of these alternative tools, so if you're interested in using other tools to automate downloading data please get in touch.
- Use result paging in WFS 2.0 - WFS 2.0 supports paging of query results - i.e. making many smaller individual requests to fetch the results of a query. For example, if you're making a WFS query that would ordinarily return 200,000 features at once, paging can split that up into 8 separate queries of 25,000 features each that are then combined back together into a single resultset.
This avoids this specific issue of exceeding ArcGIS Server's capacity to serve features, and is generally a more reliable and mature approach to downloading feature data. Support for WFS 2.0 paging varies between GIS software, so please check the functionality and documentation of your software.
At the time of writing (August, 2017) we are in the process of rolling out support for paging to WFS services provided through services.slip.wa.gov.au. To determine if a service supports paging make a GetCapabilities request and verify that the ImplementsResultPaging constraint is set to TRUE.
For example, FME 2017 implements both the Count and Max Features parameters in the WFS Service Reader. If both of these parameters are set to the number of features to return per request then ArcGIS Server honours the limits.
The Count parameter was only made available in FME 2017, so this approach does not work for older versions of FME. Please refer to our How To: Download Geospatial Data using FME article for an approach that avoids using WFS.
- Increase client timeout limits - In scenarios where time is not a factor it may be possible to simply increase the network timeout setting in the software you're using WFS with. At the time of writing (July 2017) the timeout limit on the WFS services is set to 10 minutes.
If using QGIS, please refer to our QGIS: WFS Timeout Issues article for workarounds.
If use FME with the WFS Reader, please ensure your Transfer Timeout Length and Connection Timeout Length Reader parameters have been set to 600 (seconds), rather than the much more conservative 90 (seconds) as is the default.
- Fallback to WFS 1.0/1.1 - In some scenarios it may be possible to fallback to using older versions of WFS that do respect the service feature limits in place. However, as WFS 1.x does not support paging you will need to implement this yourself - typically by breaking your area of interest up into a range of smaller query areas.
This approach is not recommended due to the added complexity and overhead.