Much of the geospatial data you'll find on Data WA is provided through SLIP—the Shared Location Information Platform—which provides you with a range of different ways for accessing geospatial data, including APIs and data snapshots.

This article will focus on accessing data snapshots - with an emphasis on automating the downloading of data snapshots for data professionals and developers.

Formats

At the time of writing, data snapshots are available in the following supported formats:

• File Geodatabase
• Shapefile (for snapshots up to 2GB)
• Geopackage
• GeoJSON

Accessing Data Snapshots

Data snapshots for geospatial data are discoverable via Data WA alongside the other resources associated with each dataset.

Selecting Download will open the SLIP login page. Once you’ve logged in, you’ll see the list of available options for the format selected.

Options include GDA94 and GDA2020 datums, and could also include different clipped extents, or public and restricted views. However you’ll only see the download options that you have permission to access.

Once you’ve chosen an option, select Download to start the download process or use Show Link if you intend to set up a script (see Automating data snapshot downloads below).

Authentication Explained

All dataset snapshots created by SLIP—even those for completely open and public datasets—require you to register an account with SLIP in order to access them. For more information about why we do this see When would I need a SLIP login?

If you’re simply accessing a SLIP data snapshot in your web browser, you’ll be asked to login to your SLIP account. After logging in, you’ll see the download options for the format you selected. If a dataset has multiple views with different restrictions, then you’ll only be able to see the options that you have permission to access.

Selecting the Download action will trigger your browser to start the download. To avoid making you login twice (once to see the options, and once to trigger the download) the Download action works by generating a signed URL for temporary access to the data snapshot extract. This means you can’t copy the URL behind the action button and use it for scripting or other automatic functions.

Tip: If you want to bookmark your favourite data snapshot and bypass Data WA, bookmark the options page for the format you selected, e.g. https://data-downloads.slip.wa.gov.au/LGATE-233/Shapefile.

To allow automating of data snapshot downloads, a separate Show Link action is available. This URL is designed for scripting, and uses an authentication system based on the OAuth 2.0 Authorization Code Grant.

If you are looking to automate or script access to data snapshots, then it pays to know a little about how SLIP's authentication system works. For more in-depth information refer to our article How does SLIP’s authentication system work?

Automating data snapshot downloads

How you'll go about automating downloads of data snapshots will vary depending on your background, skillset, and the tools at your disposal. We've put together examples for two pieces of software that are relevant to many data professionals and developers, however the logic and basic steps should be transferrable to other scripting methods.

• FME - A powerful ETL (Extract, Transform, and Load) tool for working with geospatial datasets and APIs.
• Python - The popular programming language for data professionals.

Whichever scripting method you choose, it will be necessary to use the Direct Download Link which is accessible via the Show Link button.

The URL should begin with https://direct-download.slip.wa.gov.au/.

Option 1: FME

This workbench has been tested on FME 2019.2.1.0

Reference: Download the workbench from our FME repository on GitHub.

In short, our workbench is accepting four Published Parameters:

• SOURCE_URL: The direct download link of the data snapshot to download.
• SLIP_USERNAME: The email address you signed up to SLIP with.
• SLIP_PASSWORD: Your password for SLIP.
• Token Request URL: https://sso.slip.wa.gov.au/as/token.oauth2

and downloading the LGATE-233 WA GDA2020 snapshot, e.g. LGA_Boundaries_LGATE_233_WA_GDA2020_Public_Shapefile.zip

Option 2: Python

Our Python example is using the popular Requests library for fetching our ZIP file.

In short, our python is accepting five Published Parameters:

• FolderName: The SLIP Service Name, e.g. Boundaries.
• FileName: the name of dataset, e.g. LGA_Boundaries_LGATE_233_WA_GDA2020_Public_Shapefile.zip
• UserID: The email address you signed up to SLIP with.
• PASSWORD: Your password for SLIP.
• Token Request URL: https://sso.slip.wa.gov.au/as/token.oauth2

The FolderName and FileName can be found in the Direct Download Link, i.e. https://direct-download.slip.wa.gov.au/datadownload/FolderName/FileName

import requests, json, getpass
  
folderName=input("Enter folder name:")
fileName=input("Enter file name:")
userId=input("Enter user name:")
password = getpass.getpass()

tokenRequestUrl = "https://sso.slip.wa.gov.au/as/token.oauth2"
tokenRequestHeaders = { 'Authorization' : 'Basic ZGlyZWN0LWRvd25sb2Fk'}
tokenRequestForm={"grant_type": "password", "username":userId, "password":password}
tokenResponse = requests.post(tokenRequestUrl, data=tokenRequestForm, headers=tokenRequestHeaders)
accessToken=json.loads(tokenResponse.text)["access_token"]
if tokenResponse.status_code == 200:
    dataDownloadRequestUrl = "https://direct-download.slip.wa.gov.au/datadownload/{0}/{1}".format(folderName, fileName)
    print("Downloading file from URL: " + dataDownloadRequestUrl)
    dataDownloadRequestHeaders = { 'Authorization' : 'Bearer ' + accessToken}
    dataDownloadResponse = requests.get(dataDownloadRequestUrl, headers=dataDownloadRequestHeaders)
    if dataDownloadResponse.status_code == 200:
        with open(fileName, 'wb') as f:
            f.write(dataDownloadResponse.content)
    else:
        print("Error download file with error " + str(dataDownloadResponse.status_code) + "-" + dataDownloadResponse.text)
else:
    print("Error getting token: " + str(tokenResponse.status_code) + "-" + tokenResponse.text)