GRASS GSoC 2024 EODAG Support

From GRASS-Wiki
Revision as of 00:29, 24 August 2024 by Hamed5001 (talk | contribs) (Apply suggestions from review)
Jump to navigation Jump to search

Accepted Google Summer of Code 2024 project.

Student Name: Hamed Ashraf Elgizery, AASTMT, Smart-Village, Giza, Egypt
Organization: OSGeo - Open Source Geospatial Foundation
Mentor Name: Luca Delucchi, Veronica Andreo, Stefan Blumentrath
GitHub Fork: View Fork
GSoC Proposal View Proposal


Abstract

The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.

Timeline

Period Timeline Tasks Status
Bonding Period May 1 - May 26
  1. Researching
  2. Reading GRASS documentation
  3. Further familiarizing with the GRASS development environment
  4. Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
 OK
OK
OK
OK
Official Coding Period Week 1 (May 27 - June 1)
  1. Write an initial i.eo.download addon module using USGS provider and Landsat data
  2. Limited working time due to final exams
 OK
Week 2 (June 2 - June 8)
  1. Add printing results and saving it in GeoJSON format.
  2. Add support for GeoJSON as an AOI parameter.
  3. Add downloading products from a file by IDs.
  4. Work on feedback
  5. Limited working time due to final exams
 OK

OK
OK
Week 3 (June 9 - June 15)
  1. Add initial test cases and documentation / manual
  2. Work on feedback
  3. Almost done with final exams
 OK
OK
Week 4 (June 16 - June 22)
  1. Add support for Grass Map as an AOI parameter.
  2. Add filtering by AOI relation type (intersect, contain, and within).
  3. Work on getting credentials on runtime.
  4. Testing, refactoring and bug fixing
  5. Work on feedback
 OK
OK
POSTPONED
OK
OK
Week 5 (June 23 - June 29)
  1. Add generic support for Queryables
  2. Add (to be chosen) features according to the new supported Queryables.
  3. Work on feedback
 OK
OK
OK
Week 6 (June 30 - July 6)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 7 (July 7 - July 13)
  1. Pre-evaluation reformatting and documenting
  2. Mid-term Evaluation: July 12
 OK
OK
Week 8 (July 14 - July 20)
  1. Reimplement i.landsat.download
  1. Work on feedback
 OK
OK
Week 9 (July 21 - July 27)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 10 (July 28 - August 3)
  1. Reimplement i.sentinel.download
  2. Work on feedback
 OK
OK
Week 11 (August 4 - August 10)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
 OK
OK
Week 12 (August 11 - August 17)
  1. Work on feedback
  2. Final reformatting and documenting
 OK
OK
Evaluation Period August 26 - September 2 Submit code and final evaluation In Progress

Reports

Community Bonding Period

What did I get done this week?

  • Finished Wiki setup.
  • Throughly read EODAG documentation, and got familiar with its capabilities.
  • Got familiar with the contribution guidelines.
  • Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
  • Setup development environment.

What do I plan on doing next week?

  • I plan on designing and building the initial structure for the i.eodag module.

Am I blocked on anything?

I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.

Meetings attended during the community bonding period:

  • May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
  • May 23rd 2024: Discussing module structure and features.

Week 1

What did I get done this week?

  • Built the initial version of i.eodag, found in #1090.
  • Added feature for searching & downloading by product (aka. dataset) type.
  • Added the following parameter for searching:
    • Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
    • AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
    • Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
  • Added feature for downloading products by their IDs.
  • Added option for the user to specify their custom EODAG config file.

What do I plan on doing next week?

  • Add cloud coverage parameter for searching.
  • Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
  • Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
  • Add an option to download products from a list of IDs saved in a text file.
  • Add handling of AOI as GeoJSON.

Am I blocked on anything?

Nothing, except the previously mentioned exams, but everything is going well.

Week 2

What did I get done this week?

  • Added search filtering for datasets with cloud coverage.
  • Added a method to download datasets, using a text file with a list of IDs.
  • Added printing metadata about the datasets found after searching.
  • All the mentioned above are addressed in #1097.

What do I plan on doing next week?

  • Continue working on adding AOI with Vector maps, and possibly GeoJSON.
  • Add an initial test suite, for the i.eodag module.

Am I blocked on anything?

No, and I will be done with my final exams by Wednesday this week.

Week 3

What did I get done this week?

  • Added support for setting the AOI based on a Vector Map, included in #1097.
  • Added support for two ways of searching:
    • General Search: searches through all the available providers for results.
    • Restricted Search: only searches through a predetermined – set by the user – provider.
  • Started working on a testsuite, for the i.eodag module.
  • i.eodag manual can be found here.

What do I plan on doing next week?

  • Wrap up the initial testsuite.
  • Add filtering & sorting for search results.
  • Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.

Am I blocked on anything?

I am not currently blocked on anything.

Week 4

What did I get done this week?

  • Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
  • Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
  • Added filtering for the search results using EODAG crunch, currently supported methods of filtering:
    • Start/End dates.
    • Cloud Cover.
    • Area Relation (contains - intersects - within).
    • Minimum overlap (Percentage of AOI covered by the scene).
  • Added Docstring for the module functions.
  • All the above changes can be found in #1104.
  • Attended – online – and gave a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
  • The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR #11.

What do I plan on doing next week?

  • Extend the query parameters.
  • Test the module and look into the products that might need case handling.
  • Add a way to get credentials on runtime (esp. Creodias as it requires OTP for authentication).

Am I blocked on anything?

I am not currently blocked by anything.

Week 5

What did I get done this week?

  • Added the “query” option, to allow extra search parameters #1136.
  • Added the “print” option allowing the listing of all relevant EODAG metadata #1136:
    • Current EODAG configuration.
    • Available products.
    • Available products for a given provider.
    • Available providers.
    • Available providers that offer a specific product.
    • Queryables for a given provider and/or product.
 Note: Queryables from this list can be used in the query option.
  • Added the “footprint” option to save the found scenes footprints in a vector map, with the name given in the “footprint” option #1136.
  • Added options to save/print results in JSON format #1104.

What do I plan on doing next week?

  • Continue working on adding a way to get credentials on runtime, as I haven’t received access to Creodias yet.
  • Merging the above-mentioned PRs, after further discussions with mentors.
  • Further testing and edge cases handling.

Am I blocked on anything?

I am not currently blocked by anything.

Week 6

What did I get done this week?

  • Created this PR #1136 to add more features including query, print, limit, and footprints.
  • Removed saving to JSON, and limiting it only to GeoJSON files.
  • Removed “extract” and “delete archive” flags, and instead used the values from the EODAG YAML config file.
  • Done more testing on i.eodag.
  • Worked on code refactoring and feedback.
  • Added use cases with the “print” option to i.eodag manual.

What do I plan on doing next week?

Am I blocked on anything?

I am not currently blocked by anything.

Week 7

What did I get done this week?

This week’s work was mostly focused on querying and updating the manual:

  • Added operators to the query option.
  • Added the option to use multiple values per queryable, to emulate the OR relation.
  • Added the option to use multiple queryables, to emulate the AND relation.
  • Added “querying” section to the manual with examples.
  • Added different use cases of i.eodag to the manual.
  • Code refactoring.
  • The above changes can be found in this PR #1142.

What do I plan on doing next week?

  • Further refactoring and working on feedback, as needed.
  • Continue planning for i.landsat.download, and starting with an initial reimplementation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 8

What did I get done this week?

  • Reimplemented i.landsat.download.
  • Supported Planetary Computer as a data source (USGS will be added as well once it is up again and the eodag fix reaches production eodag#1252).
  • Updated i.landsat.download manual.
  • Above changes can be found here #1144.

What do I plan on doing next week?

  • Add USGS support to i.landsat.download (couldn’t implement it yet because USGS is down, or there is some other problem).
  • Address this issue #17:
    • Add testsuite.
    • Add a method for providing credentials manually on runtime.
    • Update list_queryables function with more useful info.
  • Start experimenting with sentinel products, and doing necessary research for week 10 which should be the start of reimplementing i.sentinel.download .

Am I blocked on anything?

I am not currently blocked by anything.

Week 9

What did I get done this week?

  • Added USGS as a datasource to i.landsat.download.
  • Refactored i.landsat.download and the reimplementation is merged here #1144.
  • Added three new options to i.eodag #1145:
    • Pattern: To filter products using regular expressions, based on their ID.
    • Wait: Wait time in minutes before retrying to download data.
    • Timeout: If download fails, maximum time before stop retrying to download.

What do I plan on doing next week?

  • Start working on i.sentinel.download.
  • Address as much as possible of the issues mentioned here #17. (Couldn’t work on them last week, because of multiple travelling for a coding competition, ECPC - Egyptian Collegiate Programming Contest, but I will address them in parallel with i.sentinel.download).

Am I blocked on anything?

I am not currently blocked by anything.

Week 10

What did I get done this week?

  • Created an initial implementation of i.sentinel.download to replace sentinelsat and landsatxplore libraries by eodag #1150. To be completed in the upcoming week.
  • Added new flags to i.eodag #1152:
    • -b flag: to use the AOI borders instead of the bounding box.
    • -s flag: to skip already downloaded products from the search result list.
    • Added “footprint” sorting criteria to i.eodag to sort scenes according to their id lexicographically.
  • Added the settings option back to i.landsat.download, to set credentials #1156, which is also reimplemented in i.sentinel.download #1150.

What do I plan on doing next week?

  • Continue working on i.sentinel.download.
  • Add a testsuite for i.edoag #11.
  • Work on merging the current open PRs.

Am I blocked on anything?

I am not currently blocked by anything.

Week 11

What did I get done this week?

  • Added testsuite to i.eodag #1163, to be completed in the upcoming week.
  • Put back the “settings” option to i.landsat.download that was initially removed after reimplementation. #1156
  • Added -p flag to i.eodag to print a query summary and exit. #1167
  • Started working on this PR #1165 to update i.eodag to be compatible with the, to be released, EODAG 3.0.0, in response to the EODAG 3 beta versions.
  • Added Creodias provider support to i.eodag, by supporting authentication with TOTP. #1159
  • Updated i.sentinel.download manual after the reimplementation and continued working on the i.sentinel.download open PR. #1150

What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Refactoring and documentation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 12

What did I get done this week?

  • Continued working on the i.eodag testsuite #1163:
    • Modified tests to be run only if the connection to providers of concern can be established.
    • Implemented more generic and robust tests, that will be more resilient to changes from the provider side which shouldn’t fail the tests.
  • Handled the edge case of when a sorting parameter is missing in one scene or more. #1170
  • Wrapped up this PR #1165 to ensure the compatibility of i.eodag with EODAG 3.
  • Continued working on the i.sentinel.download reimplementation, and mapped the sleep and retry options from i.sentinel.download to the wait and timeout options from i.eodag. #1150

What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Write and submit the final report.

Am I blocked on anything?

I am not currently blocked by anything.

Final Report

Title: Add EODAG Support to GRASS GIS

Community: GRASS GIS - OSGeo

Abstract

This project introduces a new extension for GRASS GIS, i.eodag, which integrates the EODAG API with GRASS GIS, serving as a central portal for downloading Earth Observation data and products. Before this project, downloading such data required three separate modules: i.landsat.download, i.sentinel.download, and i.modis.download. Each module relied on different APIs, leading to significant maintenance efforts. The i.eodag extension simplifies this process by providing a unified source for all geospatial products, reducing the maintenance burden to a single API—the EODAG API. Additionally, i.landsat.download and i.sentinel.download have been reimplemented to internally use i.eodag, with their logic abstracted to improve modularity and maintainability.

The state of integration BEFORE the start of GSoC

Before this project, all of MODIS, Landsat and Sentinel download tools relied on different APIs, necessitating significant maintenance to accommodate for updates from both the APIs themselves and their respective providers. Specifically:

The state of integration AFTER GSoC

  1. Implemented a new module, i.eodag, which serves as a unified portal for downloading a variety of geospatial data products, extending beyond just Sentinel and Landsat. This integration allows GRASS GIS to rely on a single API, EODAG, instead of managing multiple APIs like Landsatxplore, Sentinelsat, and Google Cloud Storage.
    1. Supports generic searching without specifying a provider.
    2. Supports limiting searching to a single provider.
    3. Supports simplified search result listing.
    4. Supports extended search result listing, printing scene metadata in JSON.
    5. Downloads scenes to a specified directory.
    6. Enables searching for scenes by their IDs.
    7. Saves search results in a GeoJSON file, including all scenes' metadata.
    8. Reads previously generated GeoJSON files for further filtering.
    9. Downloads scenes saved in a GeoJSON file.
    10. Reads a text file with a list of IDs and searches for them.
    11. Supports TOTP Authentication for Creodias provider.
    12. Offers the possibility to use either a pre-existing Vector Map or the current computational region as the Area of Interest.
    13. Includes an option to skip fully downloaded scenes by using the MD5 Hash.
    14. Provides an option to filter scenes using RegEx.
    15. Provides an option to filter scenes using their metadata (e.g., orbitDirection), with filters varying depending on product type and provider.
    16. Supports querying/filtering with both OR and AND relations, as well as relational operators (e.g., 'lt', 'gt', etc.).
    17. Supports printing of relevant EODAG metadata e.g. available providers, available products and their corresponding queryables (aka. products' metadata).
    18. For the full list of features and options, please refer to the i.eodag manual.
  2. Added a comprehensive test suite for i.eodag, featuring resilient and robust test cases. These tests are designed to be flexible, accommodating different EODAG versions and future developments.
  3. Reimplemented i.landsat.download using i.eodag, while maintaining backward compatibility. The reimplementation abstracted the logic, making the modules more modular and extensible for future updates.
  4. Reimplemented i.sentinel.download using the new i.eodag subpackage, reducing the need for maintaining multiple APIs. This reimplementation simplified and abstracted the logic effectively. Some

Conclusion

To wrap it up, I’m glad to have successfully achieved the three main targets of the project, with the invaluable support from my mentors!

  • Integrated EODAG API to GRASS GIS through new the i.eodag sub-package, serving as a unified portal for downloading a variety of geospatial data products.
  • Reimplemented i.landsat.download using the new i.eodag sub-package.
  • Reimplemented i.sentinel.download using the new i.eodag sub-package.

I am incredibly grateful for the support I received this summer and for the opportunity to contribute to GRASS GIS through GSoC 2024. This experience has been rewarding, and I am excited to continue enhancing i.eodag. While we've made significant progress, there are still exciting opportunities for further improvement, as outlined in the Future Work section below.

Future Work

  • There is potential to integrate the i.eodag sub-package into the i.modis.download sub-package. This could be achieved by either adding i.eodag as an option or reimplementing i.modis.download to rely on i.eodag as the primary source for MODIS geospatial data.
  • Investigate how EODAG product types can be matched to GRASS GIS image band libraries. Read More
  • There is room for improvement in data querying and filtering, particularly for i.sentinel.download, which supports over 20 products, each with its own set of auxiliary metadata. A useful enhancement would be to provide users with an option to retrieve a list of available filters for each product type, enabling more refined searches. Although the filtering method is already implemented and functional, adding a method to access the metadata (queryables / filters) list for each product type would be beneficial. Since i.eodag currently offers metadata lists for each product type, this feature can be implemented in i.sentinel.download by leveraging the i.eodag sub-package.
  • The process of listing queryable options in i.eodag could be enhanced by extracting them directly from a scene instance rather than relying solely on the EODAG API, which may sometimes be outdated.

Links

Pull Requests Log

Pull Request Description Date Status
#1090 i.eodag: add i.eodag module May 27, 2024 Merged
#1097 i.eodag: add files support June 9, 2024 Merged
#1104 i.eodag: remove pandas dependency and save results in json/geojson files June 16, 2024 Merged
#1136 i.eodag: add query, print, and footprints options July 2, 2024 Merged
#1142 i.eodag: add operators to queryables and allow multiple values July 11, 2024 Merged
#1144 i.landsat.download: change API from landsatxplore to EODAG using i.eodag July 17, 2024 Merged
#1145 i.eodag: add timeout & wait options July 18, 2024 Merged
#1150 i.sentinel.download: use EODAG as a unified API for downloading sentinel products July 31, 2024 Merged
#1152 i.eodag: additional options and modifications for i.sentinel.download August 1, 2024 Merged
#1156 i.landsat.download: put back settings option for credentials August 4, 2024 Merged
#1159 i.eodag: add dynamic setting for Creodias OTP August 6, 2024 Merged
#1163 i.eodag: add testsuite August 8, 2024 Merged
#1165 i.eodag: update i.eodag to be compatible with EODAG 3 August 8, 2024 Merged
#1167 i.eodag: add -p flag to print query summary August 9, 2024 Merged
#1170 i.eodag: handle cases when sorting parameters are not available August 14, 2024 Merged

Documentation & GitHub Fork

Ttile Link
OSGeo Wiki Page Add EODAG Support to GRASS GIS
Github Fork GitHub Fork
EODAG API EODAG
i.eodag i.eodag manual
i.sentinel.download i.sentinel.download manual
i.landsat.download i.landsat.download manual

Images

Sentinel-2 Scene downloaded
Sentinel-2 Scene downloaded with i.sentinel.download
Sample for Sentinel Scene Downloaded
Sample for Sentinel Scene Downloaded with i.eodag
Landsat 9 Scene downloaded
Landsat-9 Scene downloaded with i.landsat.download
i.eodag GUI Sample Workflow
i.eodag GUI Sample Workflow
Search Result of i.eodag
Search Result of i.eodag GUI Sample Workflow
Downloading a Sentinel Scene
Downloading a Sentinel Scene by ID using i.sentinel.download in Terminal
Searching for Landsat Scenes
Searching for Landsat Scenes in Planetary Computer from terminal
GIF installing a scene from planetary computer.
Installing a Landsat Scene from Planetary Computer
19 test cases
19 test cases written in test suite
Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=GRASS_GSoC_2024_EODAG_Support&oldid=27900"