One of the easiest and most powerful ways to download geospatial data from data.wa.gov.au is through Safe Software's FME software.
What's FME?
It's essentially an ETL (Extract, Transform, and Load) tool for working with geospatial datasets, web services, and more. At the core of FME is a powerful visual programming language that enables users to automate connecting to data, filter and transform it, and output it to any one of over 350 different data formats.
FME is free for home and personal use.
The available APIs
There are three main APIs that can be used to access data.wa.gov.au's geospatial data -
WFS is a web service used for retrieving raw geospatial datasets and running queries against a dataset (e.g. "Tell me all of the points in this area where the type is red"). While vendor implementations differ, most communication with WFS services is conducted using XML-based formats, such as GML. WFS is part of the suite of international standards developed by the Open Geospatial Consortium.
The ArcGIS REST API has very similar functionality to that available in WFS, but presents a more modern RESTful interface. The ArcGIS REST API speaks JSON, with geometry being represented in ESRI-JSON (not to be confused with GeoJSON). The ArcGIS REST API is a proprietary API, part of the ESRI technology stack, and is widely utilised throughout the ESRI ecosystem.
Many datasets are also available to download in their entirety as ESRI Shapefiles. Shapefile have a long history in the geospatial industry and, in spite of dating from the late 1990s, are still one of the most common ways of packaging and exchanging geospatial data. Thanks to their longevity, there are many tools and platforms exist that can read Shapefiles - including web-based tools like Ogre.
Automating geospatial data downloads using the ArcGIS REST API
Using the ArcGIS REST API is currently the recommended means of automatically downloading geospatial datasets from data.wa.gov.au.
- Available for all datasets - The ArcGIS REST API is available on both ESRI MapServer and FeatureServer APIs. WFS is only available on specific endpoints and must be specially published for each service.
- Supports paging - The ArcGIS REST API supports paging requests for data over multiple requests i.e. If your query to the API will result 500,000 features being downloaded, then FME will automatically break your query up into smaller, more manageable chunks. This is particularly useful for larger datasets and avoids the issue of timeout errors from any one request taking too long.
- Note: WFS v2.0 also supports paging, but at the time of writing we are aware of a bug in ArcGIS Server 10.31 that breaks the implementation of paging with WFS v2.0. For more information see Troubleshooting - All Software: WFS 2.0 Timeouts & Slow Response Times.
- Note: WFS v2.0 also supports paging, but at the time of writing we are aware of a bug in ArcGIS Server 10.31 that breaks the implementation of paging with WFS v2.0. For more information see Troubleshooting - All Software: WFS 2.0 Timeouts & Slow Response Times.
- Powerful querying options - The ArcGIS REST API supports a range of standard querying and filtering options, such as an SQL-like WHERE clause; filtering features by a bounding box; and querying features that fall within, are contained within, intersect, or fall within a given distance of (and more) a given point, envelope, polygon, or line.
Alternatives to FME
There are a range of other tools that can be used in place of FME, including:
- ogr2ogr - The open source command-line tool for querying and converting geospatial data. For more about downloading data with FME see our article How To: Download Geospatial Data using GDAL and ogr2ogr.
- ArcREST - ESRI's Python client library for interacting with the ArcGIS REST API.
We're planning to write articles similar to this one for some of these alternative tools, so if you're interested in using other tools to automate downloading data please get in touch.
Example FME Workbench
Requirements
The workbench was built and tested in FME 2016.1, but is also compatible with FME 2015 and 2017.
The Readers used in the workbench require FME Professional Edition.
The Workbench
We've put together a simple FME Workbench that will download land parcel boundaries (Cadastre Address (LGATE-002)) using a variety of spatial and attribute filters to filter the data being downloaded.
FME Workbench demonstrating three different approaches to querying the ArcGIS REST API.
Download the workbench from our FME repository on GitHub.
The workbench contains four examples of ArcGIS Feature Service Readers that have been connected to the Public Property and Planning Service MapServer API endpoint at https://services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Property_and_Planning_WFS/MapServer.
Example #1: Download features where the 'Locality' field is 'Perth'
A simple query using an SQL-like WHERE clause to find all features where any land parcel's locality_name field is PERTH. The WHERE clause could be made considerably more complex and supports all of the common operators (=, !=, <, >, LIKE, et cetera).
Example #2: Download features within a given Polygon
An example of a common query to find all features intersecting or within a given polygon.
Example #3: Download features within a given Bounding Box
An example of a common query to find all features within a given bounding box.
Example #4: Download features within 1km of central Perth
A more complex query that demonstrates the more advanced functionality of the ArcGIS REST API by finding all land parcels that are within 1km of a given point in central Perth.
Tip: Always read the manual
If you're interested in building on this example workbench we strongly recommend that you check out:
- The FME documentation on the ArcGIS Feature Service Reader for more information on the available parameters.
- The ArcGIS REST API documentation on the MapServer/Layer API endpoint. It contains detailed information on the querying capabilities and parameters you can make use of.
- For more information about specifying the ESRI-JSON 'input geometries' please refer to ArcGIS Server: Geometry Objects.
Important Note: Specifying the Reader Coordinate System
By default the ArcGIS Feature Service Reader requests data in EPSG:4283 (GDA94), but assumes the features downloaded are in the CRS that the SLIP service was published in - typically EPSG:102100/EPSG:3857 Web Mercator.
This will break any reprojection (et cetera) you wish to do, so it's important to specifically set the 'Coordinate System' parameter on the ArcGIS Feature Service Reader to 'EPSG:4283'.
Tip: Speeding up downloads with larger page sizes (features per request)
Our workbench has the page size (the number of features that each request will download) set to 10,000. Depending on local conditions (the speed of your network, the hardware you're FME running on, et cetera) you may be able to increase the page size to reduce the time it takes to download features.
In our testing we've found that a page size of around 50,000 strikes the best balance between query size and download size.
The maximum page size available will depend on how the MapServer service has been configured. 1,000 in the default, but you can expect to find page sizes up to 100,000 on some services. You can find out how the service you're using has been configured by visiting its API endpoint and looking for the MaxRecordCount line.
Tip: Using secured services
If you're using a secured service that require a username and password to access you'll need to provide your credentials in the Reader Properties.
Authentication Type: Basic
Username: Your email address
Password: Your password
Find out more about getting an account to access geospatial data by reading out article How do I create an account with SLIP to access geospatial data?
Tip: Requests timing out
If you receive errors indicating that the service has timed out (e.g. HTTP 504 Gateway Timeout, 502 Bad Gateway) there are typically two solutions. For more information please see All Software: Requests Timing Out.
Issue: Unable to edit Reader Parameters for secured services
FME 2017.1 and earlier have a minor bug on their 'Edit Parameters' dialog for ArcGIS Feature Service Readers. When attempting to use the 'Edit Parameters' dialog after adding the Reader the 'Parameters' section will be marked as invalid and the user, password, or token fields will be flagged as invalid (red background).
This appears to be occurring because FME is not using the 'Authentication Type' field to determine which fields are required.
If you would like to raise this issue with Safe Software the tracking number is #PR73037.
Workaround: The 'Navigator' pane does not suffer from the same issue, so you can simple avoid the 'Edit Parameters' dialog and edit each parameter individually.
Issue: "Invalid argument specification for attribute"
Problem: Attempting to edit the 'Feature Type Properties' for a Reader throws an "Invalid argument specification for attribute" error box.
Affects: All Versions
Description: The error appears when trying to save edits to any fields on the 'Feature Type Properties' window. This is due to an issue present on a small number of layers in several services published by SLIP where all attributes of the 'string' data type have a length of 0.
At the time of writing (July, 2017) SLIP teams are aware of the issue and are taking steps to identify the issue and republish the affected services.
Update, November 2017 - Most of the affected layers have now been republished with the correct field lengths. Please report any others you may run across.
Workaround: The 'Navigator' pane does not suffer from the same issue, so you can simple avoid the 'Feature Type Properties' dialog and edit each parameter individually.
