Compare commits
No commits in common. '04db1e01466734d18da434ae3b9f48bb866d69ed' and '7ab7f8f493bc8fc079162e0feea3cbbd2be8f83f' have entirely different histories.
04db1e0146
...
7ab7f8f493
@ -1 +0,0 @@
|
|||||||
.ipynb_checkpoints
|
|
||||||
@ -1,106 +0,0 @@
|
|||||||
{
|
|
||||||
"cells": [
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "3b5fe448-682e-4168-9f0f-ddd4e147b59d",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"# Challenge I: Search and process data from DIAS (Copernicus Data and Information Access Services)\n",
|
|
||||||
"*WRS jupyter hackathon, 30th of June 2021*, R. Rietbroek\n",
|
|
||||||
"\n",
|
|
||||||
"The overall goal of this challenge is threefold:\n",
|
|
||||||
"1. Learn how to establish a connection with a DIAS server and execute a search request\n",
|
|
||||||
"2. Download a dataset\n",
|
|
||||||
"3. Apply a post-processing operation on such a dataset\n",
|
|
||||||
"\n",
|
|
||||||
"DIAS currently consists of 5 different data providers, which hosts a variety of data and provide access to it:\n",
|
|
||||||
"\n",
|
|
||||||
"<a href=\"https://creodias.eu/\" target=\"_blank\">\n",
|
|
||||||
" <img src=\"https://eo4society.esa.int/wp-content/uploads/2018/11/creodias_logo-e1543314673171.png\" />\n",
|
|
||||||
"</a>\n",
|
|
||||||
"\n",
|
|
||||||
"<a href=\"https://mundiwebservices.com\" target=\"_blank\">\n",
|
|
||||||
" <img src=\"https://eo4society.esa.int/wp-content/uploads/2018/11/mundi_logo-e1543314714627.png\" />\n",
|
|
||||||
"</a>\n",
|
|
||||||
"\n",
|
|
||||||
"<a href=\"https://www.onda-dias.eu/cms/\" target=\"_blank\">\n",
|
|
||||||
" <img src=\"https://eo4society.esa.int/wp-content/uploads/2018/11/onda_logo-e1543314731347.png\" />\n",
|
|
||||||
"</a>\n",
|
|
||||||
"\n",
|
|
||||||
"<a href=\"https://sobloo.eu\" target=\"_blank\">\n",
|
|
||||||
" <img src=\"https://eo4society.esa.int/wp-content/uploads/2018/11/sobloo_logo-e1543314744507.png\" />\n",
|
|
||||||
"</a>\n",
|
|
||||||
"\n",
|
|
||||||
"<a href=\"https://wekeo.eu\" target=\"_blank\">\n",
|
|
||||||
" <img src=\"https://eo4society.esa.int/wp-content/uploads/2018/11/wekeo_logo-e1543314756229.png\" />\n",
|
|
||||||
"</a>\n",
|
|
||||||
"\n",
|
|
||||||
"The different providers generally provide a graphical website to search for and access data. However, they also provide scriptable ways which allow automated access (so-called [REST-API's](https://en.wikipedia.org/wiki/Representational_state_transfer) in webserver speak). In essence, these API's allow the server to be passed additional arguments (append by using `?parameter=Value1¶meter2=Value2` which can be interpreted and processed by the server and a response can be send back. For example [https://catalogue.onda-dias.eu/opensearch/OpenSearch?instrumentShortName=MSI](https://catalogue.onda-dias.eu/opensearch/OpenSearch?instrumentShortName=MSI), will return a machine readable xml document containing the search hits for the term `MSI`. \n",
|
|
||||||
"\n",
|
|
||||||
"In this challenge, you will try to make use of the automated way to search for and access the data, in order to facilitate automated scripts. \n",
|
|
||||||
"\n",
|
|
||||||
"Unfortunately, the way to access the servers is not standardized and searches and download requests may take several forms depending on the provider. Luckily there is [progress on a python module called eodag](https://pypi.org/project/eodag/) which provides a more uniform way of accessing the the providers. You're encouraged to make use of that package."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "ecc297bd-ce71-4099-8674-be4262138eda",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Challenge statement\n",
|
|
||||||
"Supplement this notebook with functionality which (1) allow search queries to the datasets, (2) Downloading an appropriate subset of the data, (3) create a simple visualization of the downloaded data\n",
|
|
||||||
"\n",
|
|
||||||
"## Tips and tricks\n",
|
|
||||||
"* Try to find interesting datasets on the graphical webinterfaces of the servers, and see if those can also be found in a programmatic way.\n",
|
|
||||||
"* Try out with 'light' datasets first in order to avoid repeated downloading of large files. Try to see if *subsetting* is possible (downloading only parts of the dataset)\n",
|
|
||||||
"* There are many python code snippets in the crib's folder `public/resources/Python-Data-Science-Handbook` which may be of use\n",
|
|
||||||
"* From a security standpoint: Don't hardcode usernames and passwords in your jupyter notebook. For example, you can query for the user's input (see below) or use separate files which contain confidential information"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 9,
|
|
||||||
"id": "34d416dd-a9bc-48e1-8cec-8d443a3373a0",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdin",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"Please enter username roelof\n",
|
|
||||||
"Please enter password ·····\n"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"#example how to query the user for sensitive information\n",
|
|
||||||
"from getpass import getpass\n",
|
|
||||||
"credentials={}\n",
|
|
||||||
"credentials[\"user\"]=input(\"Please enter username\")\n",
|
|
||||||
"credentials[\"pass\"]=getpass(\"Please enter password\")\n",
|
|
||||||
"# Note that this information is 'volatile': when the notebook shuts down the values of credentials are lost and not stored in the notebook"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"kernelspec": {
|
|
||||||
"display_name": "Python",
|
|
||||||
"language": "python",
|
|
||||||
"name": "python3"
|
|
||||||
},
|
|
||||||
"language_info": {
|
|
||||||
"codemirror_mode": {
|
|
||||||
"name": "ipython",
|
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.8.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 5
|
|
||||||
}
|
|
||||||
File diff suppressed because one or more lines are too long
@ -1,40 +0,0 @@
|
|||||||
{
|
|
||||||
"cells": [
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "4fc5da8a-9e7a-4571-a286-c89cf81b2283",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"# Welcome to the WRS hackathon\n",
|
|
||||||
"\n",
|
|
||||||
"## Goals & expectations\n",
|
|
||||||
"\n",
|
|
||||||
"## Schedule\n",
|
|
||||||
"\n",
|
|
||||||
"\n",
|
|
||||||
"\n"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"kernelspec": {
|
|
||||||
"display_name": "Python",
|
|
||||||
"language": "python",
|
|
||||||
"name": "python3"
|
|
||||||
},
|
|
||||||
"language_info": {
|
|
||||||
"codemirror_mode": {
|
|
||||||
"name": "ipython",
|
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.8.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 5
|
|
||||||
}
|
|
||||||
Loading…
Reference in New Issue