GRASS GSoC 2024 EODAG Support: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
(Added initial timeline)
 
(68 intermediate revisions by 2 users not shown)
Line 13: Line 13:
|}
|}


__TOC__


== Abstract ==
= Abstract =
The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.
The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.


== Timeline ==
= Timeline =
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 25: Line 24:
| '''Bonding Period''' || May 1 - May 26 ||  
| '''Bonding Period''' || May 1 - May 26 ||  
# Researching
# Researching
# GRASS documentation reading
# Reading GRASS documentation
# Further familiarizing with the GRASS development environment
# Further familiarizing with the GRASS development environment
# Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
# Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
  | style="background: #ffdead;" | In Progress
  || OK<br>OK<br>OK<br>OK
|-
|-
| rowspan="12" | '''Official Coding Period'''
| rowspan="12" | '''Official Coding Period'''
Line 34: Line 33:
# Write an initial i.eo.download addon module using USGS provider and Landsat data  
# Write an initial i.eo.download addon module using USGS provider and Landsat data  
# Limited working time due to final exams
# Limited working time due to final exams
  ||
  || OK
|-
|-
| '''Week 2''' (June 2 - June 8) ||  
| '''Week 2''' (June 2 - June 8) ||  
# Work on getting credentials on runtime.
# Add printing results and saving it in GeoJSON format.
# Support choosing the provider by the user
# <s>Add support for GeoJSON as an AOI parameter.</s>
# Add downloading products from a file by IDs.
# Work on feedback
# Work on feedback
# Limited working time due to final exams
# Limited working time due to final exams
  ||
  || OK<br><br>OK<br>OK
|-
|-
| '''Week 3''' (June 9 - June 15) ||  
| '''Week 3''' (June 9 - June 15) ||  
Line 47: Line 47:
# Work on feedback
# Work on feedback
# Almost done with final exams
# Almost done with final exams
  ||
  || OK<br>OK
|-
|-
| '''Week 4''' (June 16 - June 22) ||  
| '''Week 4''' (June 16 - June 22) ||  
# Add support for AOI (Area of Interest) with '''shapely'''
# Add support for Grass Map as an AOI parameter.
# Add filtering by AOI relation type (intersect, contain, and within).
# Work on getting credentials on runtime.
# Testing, refactoring and bug fixing
# Testing, refactoring and bug fixing
# Work on feedback
# Work on feedback
  ||
  ||OK<br>OK<br>POSTPONED<br>OK<br>OK<br>
|-
|-
| '''Week 5''' (June 23 - June 29) ||  
| '''Week 5''' (June 23 - June 29) ||  
# Add generic support for [https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/4_search.html#Queryables Queryables]
# Add generic support for [https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/4_search.html#Queryables Queryables]
# Add (to be chosen) features according to the new supported Queryables.
# Work on feedback
# Work on feedback
||
||OK<br>OK<br>OK
|-
|-
| '''Week 6''' (June 30 - July 6) ||  
| '''Week 6''' (June 30 - July 6) ||  
# Add [https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/5_serialize_deserialize.html Serialization/Deserialization] for SearchResults
# Testing, refactoring and bug fixing
# Testing, refactoring and bug fixing
# Work on feedback
# Work on feedback
  ||
  || OK<br>OK
|-
|-
| style="background: #ffdead;"| '''Week 7''' (July 7 - July 13)  
| style="background: #ffdead;"| '''Week 7''' (July 7 - July 13)  
Line 70: Line 72:
# Pre-evaluation reformatting and documenting
# Pre-evaluation reformatting and documenting
# '''Mid-term Evaluation: July 12'''
# '''Mid-term Evaluation: July 12'''
||
||OK<br>OK
|-
|-
| '''Week 8''' (July 14 - July 20) ||
| '''Week 8''' (July 14 - July 20) ||
# Reimplement [https://github.com/HamedElgizery/grass-addons/tree/grass8/src/imagery/i.sentinel i.sentinel.download]
# Reimplement [https://github.com/HamedElgizery/grass-addons/tree/grass8/src/imagery/i.landsat/i.landsat.download i.landsat.download]
 
# Work on feedback
# Work on feedback
  ||
  || OK<br>OK
|-
|-
| '''Week 9''' (July 21 - July 27) ||  
| '''Week 9''' (July 21 - July 27) ||  
# Testing, refactoring and bug fixing
# Testing, refactoring and bug fixing
# Work on feedback
# Work on feedback
  ||
  || OK<br>OK
|-
|-
| '''Week 10''' (July 28 - August 3) ||  
| '''Week 10''' (July 28 - August 3) ||  
# Reimplement [https://github.com/HamedElgizery/grass-addons/tree/grass8/src/imagery/i.landsat/i.landsat.download i.landsat.download]
# Reimplement [https://github.com/HamedElgizery/grass-addons/tree/grass8/src/imagery/i.sentinel i.sentinel.download]
# Work on feedback
# Work on feedback
  ||
  || OK<br>OK
|-
|-
| '''Week 11''' (August 4 - August 10) ||
| '''Week 11''' (August 4 - August 10) ||
# Testing, refactoring and bug fixing
# Testing, refactoring and bug fixing
# Work on feedback
# Work on feedback
 
  ||OK<br>OK
  ||
|-
|-
| '''Week 12''' (August 11 - August 17) ||  
| '''Week 12''' (August 11 - August 17) ||  
# Work on feedback
# Work on feedback
# Final reformatting and documenting
# Final reformatting and documenting
  ||
  ||OK<br>OK
|-
|  '''Evaluation Period''' || August 26 - September 2 || Submit code and final evaluation || OK
|}
 
= Reports =
== Community Bonding Period ==
'''What did I get done this week?'''
* Finished Wiki setup.
* Throughly read EODAG documentation, and got familiar with its capabilities.
* Got familiar with the contribution guidelines.
* Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
* Setup development environment.
 
'''What do I plan on doing next week?'''
* I plan on designing and building the initial structure for the i.eodag module.
 
'''Am I blocked on anything?'''
:I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.
'''Meetings attended during the community bonding period:'''
* May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
* May 23rd 2024: Discussing module structure and features.
 
== Week 1 ==
 
'''What did I get done this week?'''
* Built the initial version of i.eodag, found in [https://github.com/OSGeo/grass-addons/pull/1090 #1090].
* Added feature for searching & downloading by product (aka. dataset) type.
* Added the following parameter for searching:
** Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
** AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
** Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
* Added feature for downloading products by their IDs.
* Added option for the user to specify their custom [https://eodag.readthedocs.io/en/stable/getting_started_guide/configure.html EODAG config file].
 
'''What do I plan on doing next week?'''
* Add cloud coverage parameter for searching.
* Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
* Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
* Add an option to download products from a list of IDs saved in a text file.
* Add handling of AOI as GeoJSON.
 
'''Am I blocked on anything?'''
:Nothing, except the previously mentioned exams, but everything is going well.
 
== Week 2 ==
 
'''What did I get done this week?'''
* Added search filtering for datasets with cloud coverage.
* Added a method to download datasets, using a text file with a list of IDs.
* Added printing metadata about the datasets found after searching.
* All the mentioned above are addressed in [https://github.com/OSGeo/grass-addons/pull/1097 #1097].
 
'''What do I plan on doing next week?'''
* Continue working on adding AOI with Vector maps, and possibly GeoJSON.
* Add an initial test suite, for the i.eodag module.
 
'''Am I blocked on anything?'''
:No, and I will be done with my final exams by Wednesday this week.
 
== Week 3 ==
 
'''What did I get done this week?'''
* Added support for setting the AOI based on a Vector Map, included in [https://github.com/OSGeo/grass-addons/pull/1097 #1097].
* Added support for two ways of searching:
** General Search: searches through all the available providers for results.
** Restricted Search: only searches through a predetermined – set by the user – provider.
* Started working on a testsuite, for the i.eodag module.
* i.eodag manual can be found [https://github.com/OSGeo/grass-addons/pull/1097 here].
 
'''What do I plan on doing next week?'''
* Wrap up the initial testsuite.
* Add filtering & sorting for search results.
* Attend – online – and give a brief presentation about the i.eodag module, for the [[GRASS_Community_Meeting_Prague_2024#Monday,_June_17|GRASS Community Meeting Prague 2024]].
 
'''Am I blocked on anything?'''
:I am not currently blocked on anything.
 
== Week 4 ==
 
'''What did I get done this week?'''
* Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
* Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
* Added filtering for the search results using [https://eodag.readthedocs.io/en/latest/plugins_reference/crunch.html EODAG crunch], currently supported methods of filtering:
** Start/End dates.
** Cloud Cover.
** Area Relation (contains - intersects - within).
** Minimum overlap (Percentage of AOI covered by the scene).
* Added Docstring for the module functions.
* All the above changes can be found in [https://github.com/OSGeo/grass-addons/pull/1104 #1104].
* Attended – online – and gave a brief presentation about the i.eodag module, for the [[GRASS_Community_Meeting_Prague_2024#Monday,_June_17|GRASS Community Meeting Prague 2024]].
* The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR [https://github.com/HamedElgizery/grass-addons/pull/11 #11].
 
'''What do I plan on doing next week?'''
* Extend the query parameters.
* Test the module and look into the products that might need case handling.
* Add a way to get credentials on runtime (esp. [https://creodias.eu/ Creodias] as it requires OTP for authentication).
 
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 5 ==
 
'''What did I get done this week?'''
* Added the “query” option, to allow extra search parameters [https://github.com/OSGeo/grass-addons/pull/1136 #1136].
* Added the “print” option allowing the listing of all relevant EODAG metadata [https://github.com/OSGeo/grass-addons/pull/1136 #1136]:
** Current EODAG configuration.
** Available products.
** Available products for a given provider.
** Available providers.
** Available providers that offer a specific product.
** Queryables for a given provider and/or product.
  Note: Queryables from this list can be used in the query option.
* Added the “footprint” option to save the found scenes footprints in a vector map, with the name given in the “footprint” option [https://github.com/OSGeo/grass-addons/pull/1136 #1136].
* Added options to save/print results in JSON format [https://github.com/OSGeo/grass-addons/pull/1104 #1104].
'''What do I plan on doing next week?'''
* Continue working on adding a way to get credentials on runtime, as I haven’t received access to Creodias yet.
* Merging the above-mentioned PRs, after further discussions with mentors.
* Further testing and edge cases handling.
 
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 6 ==
 
'''What did I get done this week?'''
 
* Created this PR [https://github.com/OSGeo/grass-addons/pull/1136 #1136] to add more features including query, print, limit, and footprints.
* Removed saving to JSON, and limiting it only to GeoJSON files.
* Removed “extract” and “delete archive” flags, and instead used the values from the EODAG YAML config file.
* Done more testing on i.eodag.
* Worked on code refactoring and feedback.
* Added use cases with the “print” option to i.eodag [https://github.com/OSGeo/grass-addons/blob/grass8/src/imagery/i.eodag/i.eodag.html manual].
'''What do I plan on doing next week?'''
* Start investigating possible ways to reimplement [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.landsat/i.landsat.download i.landsat.download].
* Adjust i.eodag according to how things turn out when reimplementing [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.landsat/i.landsat.download i.landsat.download].
* Describe more use cases of i.eodag in the [https://github.com/OSGeo/grass-addons/blob/grass8/src/imagery/i.eodag/i.eodag.html manual].
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 7 ==
 
'''What did I get done this week?'''
 
This week’s work was mostly focused on querying and updating the manual:
* Added operators to the query option.
* Added the option to use multiple values per queryable, to emulate the OR relation.
* Added the option to use multiple queryables, to emulate the AND relation.
* Added “querying” section to the manual with examples.
* Added different use cases of i.eodag to the [https://github.com/OSGeo/grass-addons/blob/grass8/src/imagery/i.eodag/i.eodag.html manual].
* Code refactoring.
* The above changes can be found in this PR [https://github.com/OSGeo/grass-addons/pull/1142 #1142].
'''What do I plan on doing next week?'''
* Further refactoring and working on feedback, as needed.
* Continue planning for [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.landsat/i.landsat.download i.landsat.download], and starting with an initial reimplementation.
 
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 8 ==
 
'''What did I get done this week?'''
 
* Reimplemented i.landsat.download.
* Supported Planetary Computer as a data source (USGS will be added as well once it is up again and the eodag fix reaches production [https://github.com/CS-SI/eodag/issues/1252 eodag#1252]).
* Updated i.landsat.download manual.
* Above changes can be found here [https://github.com/OSGeo/grass-addons/pull/1144 #1144].
'''What do I plan on doing next week?'''
* Add USGS support to i.landsat.download (couldn’t implement it yet because USGS is down, or there is some other problem).
* Address this issue [https://github.com/HamedElgizery/grass-addons/issues/17 #17]:
** Add testsuite.
** Add a method for providing credentials manually on runtime.
** Update list_queryables function with more useful info.
* Start experimenting with sentinel products, and doing necessary research for week 10 which should be the start of reimplementing [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.sentinel/i.sentinel.download i.sentinel.download ].
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 9 ==
 
'''What did I get done this week?'''
 
* Added USGS as a datasource to i.landsat.download.
* Refactored i.landsat.download and the reimplementation is merged here [https://github.com/OSGeo/grass-addons/pull/1144 #1144].
* Added three new options to i.eodag [https://github.com/OSGeo/grass-addons/pull/1145 #1145]:
** Pattern: To filter products using regular expressions, based on their ID.
** Wait: Wait time in minutes before retrying to download data.
** Timeout: If download fails, maximum time before stop retrying to download.
'''What do I plan on doing next week?'''
* Start working on [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.sentinel/i.sentinel.download i.sentinel.download].
* Address as much as possible of the issues mentioned here [https://github.com/HamedElgizery/grass-addons/issues/17 #17]. (Couldn’t work on them last week, because of multiple travelling for a coding competition, ECPC - Egyptian Collegiate Programming Contest, but I will address them in parallel with i.sentinel.download).
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 10 ==
 
'''What did I get done this week?'''
 
* Created an initial implementation of i.sentinel.download to replace sentinelsat and landsatxplore libraries by eodag [https://github.com/OSGeo/grass-addons/pull/1150 #1150]. To be completed in the upcoming week.
* Added new flags to i.eodag [https://github.com/OSGeo/grass-addons/pull/1152 #1152]:
** -b flag: to use the AOI borders instead of the bounding box.
** -s flag: to skip already downloaded products from the search result list.
** Added “footprint” sorting criteria to i.eodag to sort scenes according to their id lexicographically.
* Added the settings option back to i.landsat.download, to set credentials [https://github.com/OSGeo/grass-addons/pull/1156 #1156], which is also reimplemented in i.sentinel.download [https://github.com/OSGeo/grass-addons/pull/1150 #1150].
'''What do I plan on doing next week?'''
* Continue working on i.sentinel.download.
* Add a testsuite for i.edoag [https://github.com/HamedElgizery/grass-addons/pull/11 #11].
* Work on merging the current open PRs.
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 11 ==
 
'''What did I get done this week?'''
* Added testsuite to i.eodag [https://github.com/OSGeo/grass-addons/pull/1163 #1163], to be completed in the upcoming week.
* Put back the “settings” option to i.landsat.download that was initially removed after reimplementation. [https://github.com/OSGeo/grass-addons/pull/1156 #1156]
* Added -p flag to i.eodag to print a query summary and exit. [https://github.com/OSGeo/grass-addons/pull/1167 #1167]
* Started working on this PR [https://github.com/OSGeo/grass-addons/pull/1165 #1165] to update i.eodag to be compatible with the, to be released, EODAG 3.0.0, in response to the  [https://github.com/CS-SI/eodag/releases EODAG 3 beta versions.]
* Added Creodias provider support to i.eodag, by supporting authentication with TOTP. [https://github.com/OSGeo/grass-addons/pull/1159 #1159]
* Updated i.sentinel.download manual after the reimplementation and continued working on the i.sentinel.download open PR. [https://github.com/OSGeo/grass-addons/pull/1150 #1150]
 
'''What do I plan on doing next week?'''
* Continue working on merging the current open PRs, and adjusting according to feedback.
* Refactoring and documentation.
 
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
== Week 12 ==
 
'''What did I get done this week?'''
* Continued working on the i.eodag testsuite [https://github.com/OSGeo/grass-addons/pull/1163 #1163]:
** Modified tests to be run only if the connection to providers of concern can be established.
** Implemented more generic and robust tests, that will be more resilient to changes from the provider side which shouldn’t fail the tests.
* Handled the edge case of when a sorting parameter is missing in one scene or more. [https://github.com/OSGeo/grass-addons/pull/1170 #1170]
* Wrapped up this PR [https://github.com/OSGeo/grass-addons/pull/1165 #1165] to ensure the compatibility of i.eodag with [https://github.com/CS-SI/eodag/releases EODAG 3.]
* Continued working on the i.sentinel.download reimplementation, and mapped the sleep and retry options from i.sentinel.download to the wait and timeout options from i.eodag. [https://github.com/OSGeo/grass-addons/pull/1150 #1150]
 
'''What do I plan on doing next week?'''
* Continue working on merging the current open PRs, and adjusting according to feedback.
* Write and submit the final report.
 
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
 
=Final Report=
'''Title''': Add EODAG Support to GRASS GIS
 
'''Community''': GRASS GIS - OSGeo
== Abstract ==
This project introduces a new extension for GRASS GIS, <code>i.eodag</code>, which integrates the EODAG API with GRASS GIS, serving as a central portal for downloading Earth Observation data and products. Before this project, downloading such data required three separate modules: <code>i.landsat.download</code>, <code>i.sentinel.download</code>, and <code>i.modis.download</code>. Each module relied on different APIs, leading to significant maintenance efforts. The <code>i.eodag</code> extension simplifies this process by providing a unified source for all geospatial products, reducing the maintenance burden to a single API—the [https://eodag.readthedocs.io EODAG API]. Additionally, <code>i.landsat.download</code> and <code>i.sentinel.download</code> have been reimplemented to internally use <code>i.eodag</code>, with their logic abstracted to improve modularity and maintainability.
 
== The state of integration BEFORE the start of GSoC ==
Before this project, all of MODIS, Landsat and Sentinel download tools relied on different APIs, necessitating significant maintenance to accommodate for updates from both the APIs themselves and their respective providers. Specifically:
 
*[https://grass.osgeo.org/grass84/manuals/addons/i.landsat.download.html i.landsat.download]:
** Utilized the [https://github.com/yannforget/landsatxplore landsatxplore] Python library to download Landsat data from USGS. This library is no longer maintained by the original author.
*[https://grass.osgeo.org/grass84/manuals/addons/i.sentinel.download.html i.sentinel.download] utilized three distinct data sources:
** [https://github.com/sentinelsat/sentinelsat?tab=readme-ov-file sentinelsat] for downloading Sentinel data from the [https://scihub.copernicus.eu/ Copernicus Open Access Hub]. However, after the data migration to the [https://dataspace.copernicus.eu/ Copernicus Data Space Ecosystem], Sentinel products were no longer available through this source, rendering portions of the previous implementation non-functional.
** [https://github.com/yannforget/landsatxplore landsatxplore] to access Sentinel data from [https://earthexplorer.usgs.gov/ USGS].
** Manual data requests via the [https://cloud.google.com/solutions/geospatial?hl=en Google Cloud Storage] API.
 
== The state of integration AFTER GSoC ==
#I implemented a new module, [https://www.google.com/url?q=https://grass.osgeo.org/grass84/manuals/addons/i.eodag.html&sa=D&source=docs&ust=1724462440593459&usg=AOvVaw3a3jNg0Qah1KVNbas67qFB <code>i.eodag</code>], which serves as a unified portal for downloading a variety of Earth Observation data products, extending beyond just Sentinel and Landsat. This integration allows GRASS GIS to rely on a single API, '''EODAG''', instead of managing multiple APIs like '''[https://github.com/yannforget/landsatxplore landsatxplore]''', '''[https://github.com/sentinelsat/sentinelsat sentinelsat]''', and '''[https://cloud.google.com/solutions/geospatial?hl=en Google Cloud Storage]'''. Some of the features implemented in i.eodag:
##Supports generic searching without specifying a provider.
##Supports limiting searching to a single provider.
##Supports simplified search result listing.
##Supports extended search result listing, printing scene metadata in '''JSON'''.
##Downloads scenes to a specified directory.
##Enables searching for scenes by their IDs.
##Saves search results in a '''GeoJSON''' file, including all scenes' metadata.
##Reads previously generated '''GeoJSON''' files for further filtering.
##Downloads scenes saved in a '''GeoJSON''' file.
##Reads a text file with a list of IDs and searches for them.
##Supports '''TOTP''' Authentication for [https://creodias.eu/ Creodias] provider.
##Offers the possibility to use either a pre-existing '''Vector Map''' or the '''current computational region''' as the '''Area of Interest'''.
##Includes an option to skip '''fully''' downloaded scenes by using the '''MD5 Hash'''.
##Provides an option to filter scenes using '''RegEx'''.
##Provides an option to filter scenes using their metadata (e.g., orbitDirection), with filters varying depending on product type and provider.
##Supports querying/filtering with both '''OR''' and '''AND''' relations, as well as '''relational operators''' (e.g., 'lt', 'gt', etc.).
##Supports printing of relevant EODAG metadata e.g. available providers, available products and their corresponding queryables (aka. products' metadata).
##For the full list of features and options, as well as usage examples, please refer to the [https://grass.osgeo.org/grass84/manuals/addons/i.eodag.html i.eodag manual.]
#I added a comprehensive test suite for <code>i.eodag</code>, featuring resilient and robust test cases. These tests are designed to be flexible, accommodating different EODAG versions and future developments.
#I reimplemented <code>i.landsat.download</code> using <code>i.eodag</code>, while maintaining backward compatibility. The reimplementation abstracted the logic, making the tool more modular and extensible for future updates.
#I reimplemented <code>i.sentinel.download</code> using <code>i.eodag</code>, reducing the need for maintaining multiple APIs. This reimplementation simplified and abstracted the logic effectively.
 
== Conclusion ==
To wrap it up, I’m glad to have successfully achieved the three main targets of the project, with the invaluable support from my mentors!
* Integrated '''EODAG API''' to GRASS GIS through a new tool <code>i.eodag</code>, serving as a unified portal for downloading a variety of Earth Observation data products.
* Reimplemented <code>i.landsat.download</code> using the new <code>i.eodag</code> tool.
* Reimplemented <code>i.sentinel.download</code> using the new <code>i.eodag</code> tool.
 
I am incredibly grateful for the support I received this summer and for the opportunity to contribute to GRASS GIS through GSoC 2024. This experience has been rewarding, and I am excited to continue enhancing <code>i.eodag</code>. While we've made significant progress, there are still exciting opportunities for further improvement, as outlined in the Future Work section below.
 
== Future Work ==
* Investigate the possibility of the integration of the <code>i.eodag</code> tool into <code>i.modis.download</code> too. This could be achieved by either adding <code>i.eodag</code> as an option or reimplementing <code>i.modis.download</code> to rely on <code>i.eodag</code> as the primary source for '''MODIS''' data.
 
* Investigate how '''EODAG product types''' can be matched to GRASS GIS '''image band libraries'''. [https://github.com/HamedElgizery/grass-addons/issues/7 Read More]
 
* Consider adding an option for adding labels for downloaded scenes with the <code>r.support</code> tool.
 
* There is room for improvement in data querying and filtering, particularly for i.sentinel.download, which supports over '''20 products''', each with its own set of auxiliary metadata. A useful enhancement would be to provide users with an option to retrieve a list of available filters for each product type, enabling more refined searches. Although the filtering method is already implemented and functional, adding a method to access the metadata (queryables / filters) list for each product type would be beneficial. Since <code>i.eodag</code> currently offers metadata lists for each product type, this feature can be implemented in <code>i.sentinel.download</code> by leveraging the <code>i.eodag</code> tool.
 
* The process of listing queryable options in <code>i.eodag</code> could be enhanced by extracting them directly from a data product instance rather than relying solely on the '''EODAG API,''' which may sometimes be outdated.
 
== Links ==
 
=== Pull Requests Log ===
{| class="wikitable"
|-
! Pull Request !! Description !! Date !! Status
|-
| [https://github.com/OSGeo/grass-addons/pull/1090 #1090]
|| i.eodag: add i.eodag module
|| May 27, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1097 #1097]
|| i.eodag: add files support
|| June 9, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1104 #1104]
||i.eodag: remove pandas dependency and save results in json/geojson files
|| June 16, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1136 #1136]
|| i.eodag: add query, print, and footprints options
|| July 2, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1142 #1142]
|| i.eodag: add operators to queryables and allow multiple values
|| July 11, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1144 #1144]
|| i.landsat.download: change API from landsatxplore to EODAG using i.eodag
|| July 17, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1145 #1145]
|| i.eodag: add timeout & wait options
|| July 18, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1150 #1150]
|| i.sentinel.download: use EODAG as a unified API for downloading sentinel products
|| July 31, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1152 #1152]
|| i.eodag: additional options and modifications for i.sentinel.download
|| August 1, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1156 #1156]
|| i.landsat.download: put back settings option for credentials
|| August 4, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1159 #1159]
|| i.eodag: add dynamic setting for Creodias OTP
|| August 6, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1163 #1163]
|| i.eodag: add testsuite
|| August 8, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1165 #1165]
|| i.eodag: update i.eodag to be compatible with EODAG 3
|| August 8, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1167 #1167]
|| i.eodag: add -p flag to print query summary
|| August 9, 2024
|| <span style="color:#8256d0">Merged</span>
|-
| [https://github.com/OSGeo/grass-addons/pull/1170 #1170]
|| i.eodag: handle cases when sorting parameters are not available
|| August 14, 2024
|| <span style="color:#8256d0">Merged</span>
|}
 
=== Documentation & GitHub Fork ===
{| class="wikitable"
! Ttile
! Link
|-
| OSGeo Wiki Page
| [https://grasswiki.osgeo.org/w/index.php?title=GRASS_GSoC_2024_EODAG_Support Add EODAG Support to GRASS GIS]
|-
| Github Fork
| [https://github.com/HamedElgizery/grass-addons GitHub Fork]
|-
| EODAG API
| [https://eodag.readthedocs.io/en/stable/ EODAG]
|-
| i.eodag
| [https://grass.osgeo.org/grass84/manuals/addons/i.eodag.html i.eodag manual]
|-
| i.sentinel.download
| [https://grass.osgeo.org/grass84/manuals/addons/i.sentinel.download.html i.sentinel.download manual]
|-
| i.landsat.download
| [https://grass.osgeo.org/grass84/manuals/addons/i.sentinel.download.html i.landsat.download manual]
|}
 
== Images ==
 
{| style="width: 100%; text-align: center;"
|-
|[[File:Screenshot from 2024-08-23 16-50-43.png|thumb|center|alt=Sentinel-2 Scene downloaded|Sentinel-2 Scene downloaded with i.sentinel.download]]
|
[[File:Sample for Sentinel Scene Downloaded with i.eodag.png|thumb|center|alt=Sample for Sentinel Scene Downloaded|Sample for Sentinel Scene Downloaded with i.eodag]]
|[[File:LandsatC2L2Sample.png|thumb|center|alt=Landsat 9 Scene downloaded|Landsat-9 Scene downloaded with i.landsat.download]]
|}
{| style="width: 100%; text-align: center;"
|-
|[[File:I.eodag gui example.png|thumb|center|alt=i.eodag GUI Sample Workflow|i.eodag GUI Sample Workflow]]
|[[File:Search for Landsat Scenes with i.eodag.png|thumb|center|alt=Search Result of i.eodag|Search Result of i.eodag GUI Sample Workflow]]
|[[File:Sentinel Download Sample.png|thumb|center|alt=Downloading a Sentinel Scene |Downloading a Sentinel Scene by ID using i.sentinel.download in Terminal]]
|}
{| style="width: 100%; text-align: center;"
|-
|-
| '''Evaluation Period''' || August 26 - September 2 || Submit code and final evaluation ||  
|[[File:Searching for Landsat Scene.gif|thumb|center|alt=Searching for Landsat Scenes|Searching for Landsat Scenes in Planetary Computer from terminal]]
|[[File:Download Planetary Computer.gif|thumb|center|alt=GIF installing a scene from planetary computer.|Installing a Landsat Scene from Planetary Computer]]
|
[[File:Screenshot from 2024-08-23 18-47-12 (1).png|thumb|center|alt=19 test cases|19 test cases written in test suite]]
|}
|}


== Reports ==


{{GSoC}}
{{GSoC}}

Latest revision as of 12:43, 7 September 2024

Accepted Google Summer of Code 2024 project.

Student Name: Hamed Ashraf Elgizery, AASTMT, Smart-Village, Giza, Egypt
Organization: OSGeo - Open Source Geospatial Foundation
Mentor Name: Luca Delucchi, Veronica Andreo, Stefan Blumentrath
GitHub Fork: View Fork
GSoC Proposal View Proposal


Abstract

The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.

Timeline

Period Timeline Tasks Status
Bonding Period May 1 - May 26
  1. Researching
  2. Reading GRASS documentation
  3. Further familiarizing with the GRASS development environment
  4. Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
 OK
OK
OK
OK
Official Coding Period Week 1 (May 27 - June 1)
  1. Write an initial i.eo.download addon module using USGS provider and Landsat data
  2. Limited working time due to final exams
 OK
Week 2 (June 2 - June 8)
  1. Add printing results and saving it in GeoJSON format.
  2. Add support for GeoJSON as an AOI parameter.
  3. Add downloading products from a file by IDs.
  4. Work on feedback
  5. Limited working time due to final exams
 OK

OK
OK
Week 3 (June 9 - June 15)
  1. Add initial test cases and documentation / manual
  2. Work on feedback
  3. Almost done with final exams
 OK
OK
Week 4 (June 16 - June 22)
  1. Add support for Grass Map as an AOI parameter.
  2. Add filtering by AOI relation type (intersect, contain, and within).
  3. Work on getting credentials on runtime.
  4. Testing, refactoring and bug fixing
  5. Work on feedback
 OK
OK
POSTPONED
OK
OK
Week 5 (June 23 - June 29)
  1. Add generic support for Queryables
  2. Add (to be chosen) features according to the new supported Queryables.
  3. Work on feedback
 OK
OK
OK
Week 6 (June 30 - July 6)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 7 (July 7 - July 13)
  1. Pre-evaluation reformatting and documenting
  2. Mid-term Evaluation: July 12
 OK
OK
Week 8 (July 14 - July 20)
  1. Reimplement i.landsat.download
  1. Work on feedback
 OK
OK
Week 9 (July 21 - July 27)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 10 (July 28 - August 3)
  1. Reimplement i.sentinel.download
  2. Work on feedback
 OK
OK
Week 11 (August 4 - August 10)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 12 (August 11 - August 17)
  1. Work on feedback
  2. Final reformatting and documenting
 OK
OK
Evaluation Period August 26 - September 2 Submit code and final evaluation OK

Reports

Community Bonding Period

What did I get done this week?

  • Finished Wiki setup.
  • Throughly read EODAG documentation, and got familiar with its capabilities.
  • Got familiar with the contribution guidelines.
  • Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
  • Setup development environment.

What do I plan on doing next week?

  • I plan on designing and building the initial structure for the i.eodag module.

Am I blocked on anything?

I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.

Meetings attended during the community bonding period:

  • May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
  • May 23rd 2024: Discussing module structure and features.

Week 1

What did I get done this week?

  • Built the initial version of i.eodag, found in #1090.
  • Added feature for searching & downloading by product (aka. dataset) type.
  • Added the following parameter for searching:
    • Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
    • AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
    • Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
  • Added feature for downloading products by their IDs.
  • Added option for the user to specify their custom EODAG config file.

What do I plan on doing next week?

  • Add cloud coverage parameter for searching.
  • Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
  • Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
  • Add an option to download products from a list of IDs saved in a text file.
  • Add handling of AOI as GeoJSON.

Am I blocked on anything?

Nothing, except the previously mentioned exams, but everything is going well.

Week 2

What did I get done this week?

  • Added search filtering for datasets with cloud coverage.
  • Added a method to download datasets, using a text file with a list of IDs.
  • Added printing metadata about the datasets found after searching.
  • All the mentioned above are addressed in #1097.

What do I plan on doing next week?

  • Continue working on adding AOI with Vector maps, and possibly GeoJSON.
  • Add an initial test suite, for the i.eodag module.

Am I blocked on anything?

No, and I will be done with my final exams by Wednesday this week.

Week 3

What did I get done this week?

  • Added support for setting the AOI based on a Vector Map, included in #1097.
  • Added support for two ways of searching:
    • General Search: searches through all the available providers for results.
    • Restricted Search: only searches through a predetermined – set by the user – provider.
  • Started working on a testsuite, for the i.eodag module.
  • i.eodag manual can be found here.

What do I plan on doing next week?

  • Wrap up the initial testsuite.
  • Add filtering & sorting for search results.
  • Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.

Am I blocked on anything?

I am not currently blocked on anything.

Week 4

What did I get done this week?

  • Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
  • Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
  • Added filtering for the search results using EODAG crunch, currently supported methods of filtering:
    • Start/End dates.
    • Cloud Cover.
    • Area Relation (contains - intersects - within).
    • Minimum overlap (Percentage of AOI covered by the scene).
  • Added Docstring for the module functions.
  • All the above changes can be found in #1104.
  • Attended – online – and gave a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
  • The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR #11.

What do I plan on doing next week?

  • Extend the query parameters.
  • Test the module and look into the products that might need case handling.
  • Add a way to get credentials on runtime (esp. Creodias as it requires OTP for authentication).

Am I blocked on anything?

I am not currently blocked by anything.

Week 5

What did I get done this week?

  • Added the “query” option, to allow extra search parameters #1136.
  • Added the “print” option allowing the listing of all relevant EODAG metadata #1136:
    • Current EODAG configuration.
    • Available products.
    • Available products for a given provider.
    • Available providers.
    • Available providers that offer a specific product.
    • Queryables for a given provider and/or product.
 Note: Queryables from this list can be used in the query option.
  • Added the “footprint” option to save the found scenes footprints in a vector map, with the name given in the “footprint” option #1136.
  • Added options to save/print results in JSON format #1104.

What do I plan on doing next week?

  • Continue working on adding a way to get credentials on runtime, as I haven’t received access to Creodias yet.
  • Merging the above-mentioned PRs, after further discussions with mentors.
  • Further testing and edge cases handling.

Am I blocked on anything?

I am not currently blocked by anything.

Week 6

What did I get done this week?

  • Created this PR #1136 to add more features including query, print, limit, and footprints.
  • Removed saving to JSON, and limiting it only to GeoJSON files.
  • Removed “extract” and “delete archive” flags, and instead used the values from the EODAG YAML config file.
  • Done more testing on i.eodag.
  • Worked on code refactoring and feedback.
  • Added use cases with the “print” option to i.eodag manual.

What do I plan on doing next week?

Am I blocked on anything?

I am not currently blocked by anything.

Week 7

What did I get done this week?

This week’s work was mostly focused on querying and updating the manual:

  • Added operators to the query option.
  • Added the option to use multiple values per queryable, to emulate the OR relation.
  • Added the option to use multiple queryables, to emulate the AND relation.
  • Added “querying” section to the manual with examples.
  • Added different use cases of i.eodag to the manual.
  • Code refactoring.
  • The above changes can be found in this PR #1142.

What do I plan on doing next week?

  • Further refactoring and working on feedback, as needed.
  • Continue planning for i.landsat.download, and starting with an initial reimplementation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 8

What did I get done this week?

  • Reimplemented i.landsat.download.
  • Supported Planetary Computer as a data source (USGS will be added as well once it is up again and the eodag fix reaches production eodag#1252).
  • Updated i.landsat.download manual.
  • Above changes can be found here #1144.

What do I plan on doing next week?

  • Add USGS support to i.landsat.download (couldn’t implement it yet because USGS is down, or there is some other problem).
  • Address this issue #17:
    • Add testsuite.
    • Add a method for providing credentials manually on runtime.
    • Update list_queryables function with more useful info.
  • Start experimenting with sentinel products, and doing necessary research for week 10 which should be the start of reimplementing i.sentinel.download .

Am I blocked on anything?

I am not currently blocked by anything.

Week 9

What did I get done this week?

  • Added USGS as a datasource to i.landsat.download.
  • Refactored i.landsat.download and the reimplementation is merged here #1144.
  • Added three new options to i.eodag #1145:
    • Pattern: To filter products using regular expressions, based on their ID.
    • Wait: Wait time in minutes before retrying to download data.
    • Timeout: If download fails, maximum time before stop retrying to download.

What do I plan on doing next week?

  • Start working on i.sentinel.download.
  • Address as much as possible of the issues mentioned here #17. (Couldn’t work on them last week, because of multiple travelling for a coding competition, ECPC - Egyptian Collegiate Programming Contest, but I will address them in parallel with i.sentinel.download).

Am I blocked on anything?

I am not currently blocked by anything.

Week 10

What did I get done this week?

  • Created an initial implementation of i.sentinel.download to replace sentinelsat and landsatxplore libraries by eodag #1150. To be completed in the upcoming week.
  • Added new flags to i.eodag #1152:
    • -b flag: to use the AOI borders instead of the bounding box.
    • -s flag: to skip already downloaded products from the search result list.
    • Added “footprint” sorting criteria to i.eodag to sort scenes according to their id lexicographically.
  • Added the settings option back to i.landsat.download, to set credentials #1156, which is also reimplemented in i.sentinel.download #1150.

What do I plan on doing next week?

  • Continue working on i.sentinel.download.
  • Add a testsuite for i.edoag #11.
  • Work on merging the current open PRs.

Am I blocked on anything?

I am not currently blocked by anything.

Week 11

What did I get done this week?

  • Added testsuite to i.eodag #1163, to be completed in the upcoming week.
  • Put back the “settings” option to i.landsat.download that was initially removed after reimplementation. #1156
  • Added -p flag to i.eodag to print a query summary and exit. #1167
  • Started working on this PR #1165 to update i.eodag to be compatible with the, to be released, EODAG 3.0.0, in response to the EODAG 3 beta versions.
  • Added Creodias provider support to i.eodag, by supporting authentication with TOTP. #1159
  • Updated i.sentinel.download manual after the reimplementation and continued working on the i.sentinel.download open PR. #1150

What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Refactoring and documentation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 12

What did I get done this week?

  • Continued working on the i.eodag testsuite #1163:
    • Modified tests to be run only if the connection to providers of concern can be established.
    • Implemented more generic and robust tests, that will be more resilient to changes from the provider side which shouldn’t fail the tests.
  • Handled the edge case of when a sorting parameter is missing in one scene or more. #1170
  • Wrapped up this PR #1165 to ensure the compatibility of i.eodag with EODAG 3.
  • Continued working on the i.sentinel.download reimplementation, and mapped the sleep and retry options from i.sentinel.download to the wait and timeout options from i.eodag. #1150

What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Write and submit the final report.

Am I blocked on anything?

I am not currently blocked by anything.

Final Report

Title: Add EODAG Support to GRASS GIS

Community: GRASS GIS - OSGeo

Abstract

This project introduces a new extension for GRASS GIS, i.eodag, which integrates the EODAG API with GRASS GIS, serving as a central portal for downloading Earth Observation data and products. Before this project, downloading such data required three separate modules: i.landsat.download, i.sentinel.download, and i.modis.download. Each module relied on different APIs, leading to significant maintenance efforts. The i.eodag extension simplifies this process by providing a unified source for all geospatial products, reducing the maintenance burden to a single API—the EODAG API. Additionally, i.landsat.download and i.sentinel.download have been reimplemented to internally use i.eodag, with their logic abstracted to improve modularity and maintainability.

The state of integration BEFORE the start of GSoC

Before this project, all of MODIS, Landsat and Sentinel download tools relied on different APIs, necessitating significant maintenance to accommodate for updates from both the APIs themselves and their respective providers. Specifically:

The state of integration AFTER GSoC

  1. I implemented a new module, i.eodag, which serves as a unified portal for downloading a variety of Earth Observation data products, extending beyond just Sentinel and Landsat. This integration allows GRASS GIS to rely on a single API, EODAG, instead of managing multiple APIs like landsatxplore, sentinelsat, and Google Cloud Storage. Some of the features implemented in i.eodag:
    1. Supports generic searching without specifying a provider.
    2. Supports limiting searching to a single provider.
    3. Supports simplified search result listing.
    4. Supports extended search result listing, printing scene metadata in JSON.
    5. Downloads scenes to a specified directory.
    6. Enables searching for scenes by their IDs.
    7. Saves search results in a GeoJSON file, including all scenes' metadata.
    8. Reads previously generated GeoJSON files for further filtering.
    9. Downloads scenes saved in a GeoJSON file.
    10. Reads a text file with a list of IDs and searches for them.
    11. Supports TOTP Authentication for Creodias provider.
    12. Offers the possibility to use either a pre-existing Vector Map or the current computational region as the Area of Interest.
    13. Includes an option to skip fully downloaded scenes by using the MD5 Hash.
    14. Provides an option to filter scenes using RegEx.
    15. Provides an option to filter scenes using their metadata (e.g., orbitDirection), with filters varying depending on product type and provider.
    16. Supports querying/filtering with both OR and AND relations, as well as relational operators (e.g., 'lt', 'gt', etc.).
    17. Supports printing of relevant EODAG metadata e.g. available providers, available products and their corresponding queryables (aka. products' metadata).
    18. For the full list of features and options, as well as usage examples, please refer to the i.eodag manual.
  2. I added a comprehensive test suite for i.eodag, featuring resilient and robust test cases. These tests are designed to be flexible, accommodating different EODAG versions and future developments.
  3. I reimplemented i.landsat.download using i.eodag, while maintaining backward compatibility. The reimplementation abstracted the logic, making the tool more modular and extensible for future updates.
  4. I reimplemented i.sentinel.download using i.eodag, reducing the need for maintaining multiple APIs. This reimplementation simplified and abstracted the logic effectively.

Conclusion

To wrap it up, I’m glad to have successfully achieved the three main targets of the project, with the invaluable support from my mentors!

  • Integrated EODAG API to GRASS GIS through a new tool i.eodag, serving as a unified portal for downloading a variety of Earth Observation data products.
  • Reimplemented i.landsat.download using the new i.eodag tool.
  • Reimplemented i.sentinel.download using the new i.eodag tool.

I am incredibly grateful for the support I received this summer and for the opportunity to contribute to GRASS GIS through GSoC 2024. This experience has been rewarding, and I am excited to continue enhancing i.eodag. While we've made significant progress, there are still exciting opportunities for further improvement, as outlined in the Future Work section below.

Future Work

  • Investigate the possibility of the integration of the i.eodag tool into i.modis.download too. This could be achieved by either adding i.eodag as an option or reimplementing i.modis.download to rely on i.eodag as the primary source for MODIS data.
  • Investigate how EODAG product types can be matched to GRASS GIS image band libraries. Read More
  • Consider adding an option for adding labels for downloaded scenes with the r.support tool.
  • There is room for improvement in data querying and filtering, particularly for i.sentinel.download, which supports over 20 products, each with its own set of auxiliary metadata. A useful enhancement would be to provide users with an option to retrieve a list of available filters for each product type, enabling more refined searches. Although the filtering method is already implemented and functional, adding a method to access the metadata (queryables / filters) list for each product type would be beneficial. Since i.eodag currently offers metadata lists for each product type, this feature can be implemented in i.sentinel.download by leveraging the i.eodag tool.
  • The process of listing queryable options in i.eodag could be enhanced by extracting them directly from a data product instance rather than relying solely on the EODAG API, which may sometimes be outdated.

Links

Pull Requests Log

Pull Request Description Date Status
#1090 i.eodag: add i.eodag module May 27, 2024 Merged
#1097 i.eodag: add files support June 9, 2024 Merged
#1104 i.eodag: remove pandas dependency and save results in json/geojson files June 16, 2024 Merged
#1136 i.eodag: add query, print, and footprints options July 2, 2024 Merged
#1142 i.eodag: add operators to queryables and allow multiple values July 11, 2024 Merged
#1144 i.landsat.download: change API from landsatxplore to EODAG using i.eodag July 17, 2024 Merged
#1145 i.eodag: add timeout & wait options July 18, 2024 Merged
#1150 i.sentinel.download: use EODAG as a unified API for downloading sentinel products July 31, 2024 Merged
#1152 i.eodag: additional options and modifications for i.sentinel.download August 1, 2024 Merged
#1156 i.landsat.download: put back settings option for credentials August 4, 2024 Merged
#1159 i.eodag: add dynamic setting for Creodias OTP August 6, 2024 Merged
#1163 i.eodag: add testsuite August 8, 2024 Merged
#1165 i.eodag: update i.eodag to be compatible with EODAG 3 August 8, 2024 Merged
#1167 i.eodag: add -p flag to print query summary August 9, 2024 Merged
#1170 i.eodag: handle cases when sorting parameters are not available August 14, 2024 Merged

Documentation & GitHub Fork

Ttile Link
OSGeo Wiki Page Add EODAG Support to GRASS GIS
Github Fork GitHub Fork
EODAG API EODAG
i.eodag i.eodag manual
i.sentinel.download i.sentinel.download manual
i.landsat.download i.landsat.download manual

Images

Sentinel-2 Scene downloaded
Sentinel-2 Scene downloaded with i.sentinel.download
Sample for Sentinel Scene Downloaded
Sample for Sentinel Scene Downloaded with i.eodag
Landsat 9 Scene downloaded
Landsat-9 Scene downloaded with i.landsat.download
i.eodag GUI Sample Workflow
i.eodag GUI Sample Workflow
Search Result of i.eodag
Search Result of i.eodag GUI Sample Workflow
Downloading a Sentinel Scene
Downloading a Sentinel Scene by ID using i.sentinel.download in Terminal
Searching for Landsat Scenes
Searching for Landsat Scenes in Planetary Computer from terminal
GIF installing a scene from planetary computer.
Installing a Landsat Scene from Planetary Computer
19 test cases
19 test cases written in test suite
Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=GRASS_GSoC_2024_EODAG_Support&oldid=27950"