One of the easiest and quickest ways for developers and data professionals to download geospatial data from data.wa.gov.au is through the ogr2ogr command-line tool, part of the GDAL software library.
What is GDAL?
GDAL (the Geospatial Data Abstraction Library) is a powerful open source project for reading and writing vector and raster geospatial data. As a software library it provides both command-line tools for data translation and processing, and API wrappers around its C/C++ API for a range of programming languages (such as Python and Java). GDAL underpins a lot of the desktop and server-based software in the geospatial industry due to its maturity and comprehensive functionality.
GDAL is a free and open source software project licensed under the X/MIT license by the Open Source Geospatial Foundation.
APIs available
There are three main APIs that can be used to access data.wa.gov.au's geospatial data -
WFS is a web service used for retrieving raw geospatial datasets and running queries against a dataset (e.g. "Tell me all of the points in this area where the type is red"). While vendor implementations differ, most communication with WFS services is conducted using XML-based formats, such as GML. WFS is part of the suite of international standards developed by the Open Geospatial Consortium.
The ArcGIS REST API has very similar functionality to that available in WFS, but presents a more modern RESTful interface. The ArcGIS REST API speaks JSON, with geometry being represented in ESRI-JSON (not to be confused with GeoJSON). The ArcGIS REST API is a proprietary API, part of the ESRI technology stack, and is widely utilised throughout the ESRI ecosystem.
Many datasets are also available to download in their entirety as ESRI Shapefiles. Shapefile have a long history in the geospatial industry and, in spite of dating from the late 1990s, are still one of the most common ways of packaging and exchanging geospatial data. Thanks to their longevity, there are many tools and platforms exist that can read Shapefiles - including web-based tools like Ogre.
Downloading geospatial data using ogr2ogr
Using the ArcGIS REST API is currently the recommended means of automatically downloading geospatial datasets from data.wa.gov.au.
-
Available for all datasets - The ArcGIS REST API is available on both ESRI MapServer and FeatureServer APIs. WFS is only available on specific endpoints and must be specially published for each service.
-
Supports paging - The ArcGIS REST API supports paging requests for data over multiple requests i.e. If your query to the API will result 500,000 features being downloaded, then FME will automatically break your query up into smaller, more manageable chunks. This is particularly useful for larger datasets and avoids the issue of timeout errors from any one request taking too long.
- Powerful querying options - The ArcGIS REST API supports a range of standard querying and filtering options, such as an SQL-like WHERE clause; filtering features by a bounding box; and querying features that fall within, are contained within, intersect, or fall within a given distance of (and more) a given point, envelope, polygon, or line.
Alternatives to ogr2ogr
There are a range of other tools that can be used in place of OGR, including:
-
FME - An ETL (Extract, Transform, and Load) tool for geospatial data with a powerful visual programming language. For more about downloading data with FME see our article How To: Download Geospatial Data Snapshots.
- ArcREST - ESRI's Python client library for interacting with the ArcGIS REST API.
We're planning to write articles similar to this one for some of these alternative tools, so if you're interested in using other tools to automate downloading data please get in touch.
Getting setup
Requirements
These examples were tested with GDAL 2.1.1 on macOS and Windows, but are also compatible with GDAL 1.10.0 and later.
Installing GDAL and ogr2ogr
Instructions for installing GDAL are beyond the scope of this article. Refer to the GDAL/OGR Binaries for downloads of pre-built binaries and installers for Linux, macOS, and Windows. OSGGeo4W is the recommended way of installing GDAL on Windows.
About the examples
We've put together a selection of simple examples that will use ogr2ogr to download land parcel boundaries (Cadastre Address (LGATE-002)) using a variety of spatial and attribute filters to filter the data being downloaded.
The four examples have been connected to the Public Property and Planning Service ArcGIS REST API MapServer endpoint at https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Property_and_Planning_WFS/MapServer.
ogr2ogr Explained
ogr2ogr is a command-line utility provided by GDAL for translating and converting vector geospatial data. It's most often used for converting from one format to another, but it can also speak to a variety of web services - including the ArcGIS REST API, as we'll demonstrate here, as well as WFS and WMS.
The following examples all use a similar set of parameters and flags, but before we dive into the examples it pays to explain what's each parameter is doing. For example -
ogr2ogr -f GeoJSON CadastreAddress_Example1.json
"https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?where=locality+%3D+%27PERTH%27&outfields=*&f=json" -s_srs EPSG:3857 -t_srs EPSG:4326
-
ogr2ogr -f GeoJSON CadastreAddress_Example1.json: Tells ogr2ogr to save the result of the query as a GeoJSON file called CadastreAddress_Example1.json.
The functionality to access the ArcGIS REST API is part of ogr2ogr's GeoJSON driver, but thanks to the flexibility of ogr2ogr we can translate data to any format - not just GeoJSON. Other data formats are supported by simply changing this parameter e.g. -f "ESRI Shapefile" CadastreAddress_Example1.shp would output to an ESRI Shapefile.
- "https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?where=locality+%3D+%27PERTH%27&outfields=*&f=json": The URL for the API call to the ArcGIS REST API.
- -s_srs EPSG:3857 -t_srs EPSG:4326: Lastly, we tell ogr2ogr to expect that the source coordinate system is Web Mercator (EPSG:3857) and that it should translate it to latitude and longitude (EPSG:4326 - WGS84).
Read on beyond the examples for further links to documentation to help with using ogr2ogr and constructing queries and API calls, as well as workarounds for a few common issues we've experienced.
API Explorer
For each example we've included a link to an interactive API query interface to allow you to play and experiment with different sorts of queries.
Since some of our examples return tens of thousands of features we've set a limit of 5 on the number of features to return on each example.
Examples
Example #1: Download features where the 'Locality' field is 'Perth'
A simple query using an SQL-like WHERE clause to find all features where any land parcel's locality field is PERTH. The WHERE clause could be made considerably more complex and supports all of the common operators (=, !=, <, >, LIKE, et cetera).
ogr2ogr -f GeoJSON CadastreAddress_Example1.json "https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?where=locality+%3D+%27PERTH%27&outfields=*&f=json" -s_srs EPSG:3857 -t_srs EPSG:4326
Parameters -
- WHERE Clause: locality = 'PERTH'
- OUTFIELDS: *
Click the image below to access an interactive API query interface for this example.
Example #2: Download features within a given Polygon
An example of a common query to find all features intersecting or within a given polygon.
ogr2ogr -f GeoJSON CadastreAddress_Example4.json "https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?geometry=%7B%0D%0A++"rings"%3A%5B%5B%5B115.567663961342%2C-33.6174479261691%5D%2C%5B115.583159110819%2C-33.6220283429028%5D%2C%5B115.587865465648%2C-33.6659900561571%5D%2C%5B115.6077408301%2C-33.6728148671717%5D%2C%5B115.732476136582%2C-33.6634886308144%5D%2C%5B115.732297741742%2C-33.6925006748829%5D%2C%5B115.767473222753%2C-33.7009065642987%5D%2C%5B115.764016316325%2C-33.7969305274894%5D%2C%5B115.73607944692%2C-33.7969286275913%5D%2C%5B115.736089896359%2C-33.811789378754%5D%2C%5B115.399265686606%2C-33.8051453374518%5D%2C%5B115.37303387792%2C-33.829174535644%5D%2C%5B115.315213456062%2C-33.839732827953%5D%2C%5B115.297822138473%2C-33.8187417902751%5D%2C%5B114.900705739486%2C-33.8190032777153%5D%2C%5B114.931029439647%2C-33.7865647814667%5D%2C%5B114.915503053611%2C-33.6846299979263%5D%2C%5B114.961686935133%2C-33.6174088944398%5D%2C%5B114.942821939263%2C-33.5201830641187%5D%2C%5B114.98579889065%2C-33.478891112763%5D%2C%5B115.088078058653%2C-33.4998861039053%5D%2C%5B115.136058894493%2C-33.5361888950176%5D%2C%5B115.162848896022%2C-33.5811250045192%5D%2C%5B115.225093062874%2C-33.6029611056781%5D%2C%5B115.364451116377%2C-33.5846169333332%5D%2C%5B115.410261360594%2C-33.5452615368935%5D%2C%5B115.491691913425%2C-33.6113244150256%5D%2C%5B115.549184672718%2C-33.6026086326786%5D%2C%5B115.567663961342%2C-33.6174479261691%5D%5D%5D%2C%0D%0A++"spatialReference"%3A%7B"wkid"%3A4326%7D%0D%0A%7D&geometryType=esriGeometryPolygon&outfields=*&f=json" -s_srs EPSG:3857 -t_srs EPSG:4326
Parameters -
- INPUT GEOMETRY: A simplified polygon describing the shape of the local government area of Busselton.
- INPUT GEOMETRY TYPE: Polygon
- SPATIAL RELATIONSHIP: Intersects
- OUTFIELDS: *
Click the image below to access an interactive API query interface for this example.
Example #3: Download features within a given Bounding Box
An example of a common query to find all features within a given bounding box.
ogr2ogr -f GeoJSON CadastreAddress_Example3.json "https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?geometry=%7B%0D%0A++"xmin"%3A+115.8292007446289%2C%0D%0A++"ymin"%3A+-31.99584774425183%2C%0D%0A++"xmax"%3A+115.93065261840819%2C%0D%0A++"ymax"%3A+-31.929874625779956%2C%0D%0A++"spatialReference"+%3A+%7B"wkid"+%3A+4283%7D%0D%0A%7D&geometryType=esriGeometryEnvelope&outfields=*&f=json" -s_srs EPSG:3857 -t_srs EPSG:4326
Parameters -
- INPUT GEOMETRY: The latitude and longitude of a bounding box over central Perth.
- INPUT GEOMETRY TYPE: Envelope
- SPATIAL RELATIONSHIP: Intersects
- OUTFIELDS: *
Click the image below to access an interactive API query interface for this example.
Example #4: Download features within 1km of central Perth
A more complex query that demonstrates the more advanced functionality of the ArcGIS REST API by finding all land parcels that are within 1km of a given point in central Perth.
ogr2ogr -f GeoJSON CadastreAddress_Example2.json "https://public-services.slip.wa.gov.au/public/rest/services/SLIP_Public_Services/Places_and_Addresses_WFS/MapServer/2/query?geometry=%7B%0D%0A+"x"%3A+115.86055846038161%2C%0D%0A+"y"%3A+-31.955887684225114%2C%0D%0A++"spatialReference"+%3A+%7B"wkid"+%3A+4326%7D%0D%0A%7D&geometryType=esriGeometryPoint&inSR=&spatialRel=esriSpatialRelContains&distance=1000&units=esriSRUnit_Meter&outfields=*&f=json" -s_srs EPSG:3857 -t_srs EPSG:4326
Parameters -
- INPUT GEOMETRY: The latitude and longitude of a point.
- INPUT GEOMETRY TYPE: Point
- SPATIAL RELATIONSHIP: Contains
- DISTANCE: 1000
- UNITS: Metres
- OUTFIELDS: *
Click the image below to access an interactive API query interface for this example.
NB: The query interface does not support the Distance and Units parameters - you will need to append distance=1000&units=esriSRUnit_Meter to the URL for this example to work.
Troubleshooting and tips
Tip: Always read the manual
If you're interested in building on these examples we strongly recommend that you check out:
- GDAL's documentation for ogr2ogr detailing all of the possible parameters that can be passed to it. If you're new to GDAL, you may also find Geospatial Wandering: GDAL / OGR cheatsheet a useful resource for getting started with GDAL and ogr2ogr.
- The ArcGIS REST API documentation on the MapServer/Layer API endpoint. It contains detailed information on the querying capabilities and parameters you can make use of.
- For more information about specifying the ESRI-JSON 'input geometries' (e.g. Point, Polygon, et cetera) please refer to ArcGIS Server: Geometry Objects.
Issue: SSL Certificate Problem: Unable to get local issuer certificate
In some scenarios you may receive an error from ogr2ogr when trying to run these examples.
To work around this you can pass the GDAL config parameter --config GDAL_HTTP_UNSAFESSL YES as part of your ogr2ogr command. For more information about this issue see GDAL_HTTP_UNSAFESSL and this thread [gdal-dev]: libcurl and the certificates and Windows.
Issue: EPSG PCS/GCS code 102100 not found in EPSG support files
In running these examples you may notice an ERROR being logged asking if EPSG:102100 is a valid EPSG coordinate system.
Don't worry about it - this is a warning generated by GDAL if you don't have the ESRI-specific coordinate system EPSG:102100 available on your system. EPSG:102100 represents the Web Mercator projection popularised by Google Maps and used by many web-based mapping tools.
Fortunately, we've already worked around this in our examples by passing -s_srs EPSG:3857 -t_srs EPSG:4326 to ogr2ogr.
This tells ogr2ogr to expect that the source coordinate system is Web Mercator (EPSG:3857) and that it should translate it to latitude and longitude (EPSG:4326 - WGS84). This works because EPSG:3857 and EPSG:102100 are mathematically identical.
Tip: Using secured services
If you're using a secured service that requires you to provide a username and password to access it you'll need to provide your credentials as additional config parameters to ogr2ogr.
- GDAL_HTTP_AUTH: "BASIC" instructs GDAL to use HTTP Basic Authentication.)
- GDAL_HTTP_USERPWD: Provide your credentials in the format "username:password". (Your username is your email address.)
For production applications we recommend setting these as environment variables on your system, rather than passing them via the --config flag as shown here. This also removes the need for you to deal with escaping special characters in passwords, and allows other commands in the GDAL suite (e.g. ogrinfo) to use the same environment variables.
Refer to GDAL: Configuration Options for more information about the other config parameters available. The "Setting ConfigOptions" section contains guidance on how to set environment variables.
Find out more about getting an account to access geospatial data by reading out article How do I create an account with SLIP to access geospatial data?
Tip: Encoding spaces and special characters correctly in queries
If you're using a query that includes spaces or other special characters you'll need to apply URL encoding (aka Percent encoding) to convert them to values that SLIP will understand.
For example, if your query is -
where=lga_name = 'JOONDALUP, CITY OF'
Then we'll need to properly encode the equals sign, the single quote characters, and all of the spaces.
You can apply URL encoding by-hand based on the reference on Wikipedia linked to above, but the most effective way is to use a handy web-based tool to encode your query. For example, https://meyerweb.com/eric/tools/dencoder/.
You only need to encode the portion of your query after the "where=" component.
Tip: ogr2gui - The GUI for ogr2ogr
If you like GUIs more than the command-line you might be interested in ogr2gui - a separate project not under the GDAL banner that provides a simple Windows GUI around the ogr2ogr command-line interface.
NB: At this stage ogr2gui only supports interacting with WFS APIs - so our examples here using the ArcGIS REST API won't work.