In this article
- Publishing vs uploading data
- Uploading your data
In preparing data for publishing data creators should ensure that their data meets the technical requirements of the Data.wa.gov.au platform. The specifics of these requirements vary depending on the type of data you have, and where you're publishing the data to. Broadly speaking, there are three methods for publishing and uploading your data -
- Publish and upload your data directly into Data.wa.gov.au.
- Publish on Data.wa.gov.au, but upload your data to another system and publish a link to it on Data.wa.gov.au
- [Geospatial Data Only] Publish on Data.wa.gov.au, but upload your data to SLIP Self-Service and have its data services automatically harvested back to Data.wa.gov.au.
Publishing vs uploading data
Before we continue and explain the three methods in more detail it's important to understand the difference between publishing data and uploading data.
Publishing Data: Simply refers to making your data discoverable by publishing a metadata record, and any supporting materials, publicly on Data.wa.gov.au. Publishing data does not mean the data is publicly accessible, just that the existence of the data has been publicised. This is the minimum requirement agencies need to take to be compliant with the Open Data Policy.
Uploading Data: Refers to uploading and storing a copy of your data somewhere in order to make it available for others to use. Where you choose to store your data will vary between datasets based on its type, size, classification, and other factors. Typically, the system you are storing your data in will also take care of applying any security controls to the data. Once uploaded, the link to the place you are storing your data needs to be added to Data.wa.gov.au.
Uploading your data
Directly to data.wa.gov.au
The technology powering data.wa.gov.au - called CKAN - is much more than just a traditional metadata catalogue. In addition to making metadata records discoverable and easily understandable it offers -
- The ability to store and serve files that are uploaded to it - this allows your to attach supporting material for your data (PDFs, documents, and images) as well as uploading and hosting your data itself.
- Automatic creation of APIs from tabular datasets (e.g. spreadsheets) uploaded to it - this allows agencies to quickly and easily create an API for datasets without having to invest in their own IT infrastructure and tools.
- A comprehensive API for searching for datasets and publishing updates - this allows agencies to automate the process of publishing data and metadata updates, and for third parties to integrate directly with the platform.
Suitable for
Storing all types and formats of data.
File formats
There are no restrictions on the file formats that may be uploaded to data.wa.gov.au.
Limits
Individual files of up to approximately 1GB are supported. If you need to publish larger files, or a lot of data at once, get in touch with the Open Data Team to discuss your needs.
To another system
If you already have your data uploaded and stored elsewhere there's no need to create another copy by uploading it to data.wa.gov.au unless you would like to make use of the benefits discussed above. If your data is already -
- Uploaded to your agency's website,
- stored in a separate storage system such as Amazon S3 or Azure Storage, or
- available via a service or API that your agency is running,
Then you need only add a link to that dataset to data.wa.gov.au.
Suitable for
Storing all types and formats of data.
File formats
Dependant upon system.
Limits
Dependant upon system.
[Geospatial Data] SLIP data upload tool
The SLIP Self-Service platform is part of the data infrastructure that powers data.wa.gov.au. It provides a single place to store and serve geospatial data through a range of data services including APIs and data snapshots.
Suitable for
Geospatial data formats only.
File formats
ESRI File Geodatabase and Shapefile.
Limits
1GB directly loaded into the system, however if you load directly to your bucket in the database then there is no limit