GRASS GSoC 2024 EODAG Support: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
(Update Week 12)
(Add a draft of "Abstract" of final report)
Line 100: Line 100:
  ||OK<br>OK
  ||OK<br>OK
|-
|-
|  '''Evaluation Period''' || August 26 - September 2 || Submit code and final evaluation ||  
|  '''Evaluation Period''' || August 26 - September 2 || Submit code and final evaluation || In Progress
|}
|}


Line 218: Line 218:


== Week 1 ==
== Week 1 ==
Following is the report for Week 1.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 241: Line 240:


== Week 2 ==
== Week 2 ==
Following is the report for Week 2 report.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 257: Line 255:


== Week 3 ==
== Week 3 ==
Here is my weekly report for Week 3.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 276: Line 273:


== Week 4 ==
== Week 4 ==
Here is my weekly report for Week 4.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 300: Line 296:


== Week 5 ==
== Week 5 ==
Here is my weekly report for Week 5.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 323: Line 318:


== Week 6 ==
== Week 6 ==
Here is my weekly report for Week 6.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 341: Line 335:


== Week 7 ==
== Week 7 ==
Here is my weekly report for Week 7.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 361: Line 354:


== Week 8 ==
== Week 8 ==
Here is my weekly report for Week 8.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 380: Line 372:


== Week 9 ==
== Week 9 ==
Here is my weekly report for Week 9.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 397: Line 388:


== Week 10 ==
== Week 10 ==
Here is my weekly report for Week 10.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 415: Line 405:


== Week 11 ==
== Week 11 ==
Here is my weekly report for Week 11.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 433: Line 422:


== Week 12 ==
== Week 12 ==
Here is my weekly report for Week 12.


'''What did I get done this week?'''
'''What did I get done this week?'''
Line 450: Line 438:
'''Am I blocked on anything?'''
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
:I am not currently blocked by anything.
=Final Report=
'''Title''': Add EODAG Support to GRASS GIS
'''Community''': GRASS GIS - OSGeo
== Abstract ==
This project introduces a new subpackage for GRASS GIS, <code>i.eodag</code> that integrates the EODAG API with GRASS GIS acting as a portal for downloading geospatial scenes through GRASS GIS. Previously, to download geospatial scenes that had to be done through three main modules, <code>i.landsat.download</code>, <code>i.sentinel.download</code>, and <code>i.modis.download</code>, and each of them had to use different APIs, which resulted in a lot of effort that had to be put into maintaining the different APIs. So here comes <code>i.eodag</code> to act as the main source of all geospatial products, and instead of maintaining all the different APIs, now only the EODAG API has to be sufficiently maintained. Moreover, <code>i.landsat.download</code> & <code>i.sentinel.download</code> have also been reimplemented, while abstracting all the logic, using the new <code>i.eodag</code> subpackage.
== The state of integration BEFORE the start of GSoC ===
== The state of integration AFTER GSoC ==
== Future Work ==
== Links ==
== Images ==
== Media ==
{{GSoC}}
{{GSoC}}

Revision as of 01:45, 21 August 2024

Accepted Google Summer of Code 2024 project.

Student Name: Hamed Ashraf Elgizery, AASTMT, Smart-Village, Giza, Egypt
Organization: OSGeo - Open Source Geospatial Foundation
Mentor Name: Luca Delucchi, Veronica Andreo, Stefan Blumentrath
GitHub Fork: View Fork
GSoC Proposal View Proposal


Abstract

The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.

Timeline

Period Timeline Tasks Status
Bonding Period May 1 - May 26
  1. Researching
  2. Reading GRASS documentation
  3. Further familiarizing with the GRASS development environment
  4. Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
OK
OK
OK
OK
Official Coding Period Week 1 (May 27 - June 1)
  1. Write an initial i.eo.download addon module using USGS provider and Landsat data
  2. Limited working time due to final exams
OK
Week 2 (June 2 - June 8)
  1. Add printing results and saving it in GeoJSON format.
  2. Add support for GeoJSON as an AOI parameter.
  3. Add downloading products from a file by IDs.
  4. Work on feedback
  5. Limited working time due to final exams
OK

OK
OK
Week 3 (June 9 - June 15)
  1. Add initial test cases and documentation / manual
  2. Work on feedback
  3. Almost done with final exams
OK
OK
Week 4 (June 16 - June 22)
  1. Add support for Grass Map as an AOI parameter.
  2. Add filtering by AOI relation type (intersect, contain, and within).
  3. Work on getting credentials on runtime.
  4. Testing, refactoring and bug fixing
  5. Work on feedback
OK
OK
POSTPONED
OK
OK
Week 5 (June 23 - June 29)
  1. Add generic support for Queryables
  2. Add (to be chosen) features according to the new supported Queryables.
  3. Work on feedback
OK
OK
OK
Week 6 (June 30 - July 6)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
OK
OK
Week 7 (July 7 - July 13)
  1. Pre-evaluation reformatting and documenting
  2. Mid-term Evaluation: July 12
OK
OK
Week 8 (July 14 - July 20)
  1. Reimplement i.landsat.download
  1. Work on feedback
OK
OK
Week 9 (July 21 - July 27)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
OK
OK
Week 10 (July 28 - August 3)
  1. Reimplement i.sentinel.download
  2. Work on feedback
OK
OK
Week 11 (August 4 - August 10)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
OK
OK
Week 12 (August 11 - August 17)
  1. Work on feedback
  2. Final reformatting and documenting
OK
OK
Evaluation Period August 26 - September 2 Submit code and final evaluation In Progress

Pull Requests Log

Pull Request Description Date Status
#1090 i.eodag: add i.eodag module May 27, 2024 Merged
#1097 i.eodag: add files support June 9, 2024 Merged
#1104 i.eodag: remove pandas dependency and save results in json/geojson files June 16, 2024 Merged
#11 i.eodag: add testsuite June 16, 2024 Moved to #1163
#13 i.eodag: add importing footprints June 24, 2024 Moved to #1136
#15 i.eodag: add queryables June 30, 2024 Moved to #1136
#1136 i.eodag: add query, print, and footprints options July 2, 2024 Merged
#1142 i.eodag: add operators to queryables and allow multiple values July 11, 2024 Merged
#1144 i.landsat.download: change API from landsatxplore to EODAG using i.eodag July 17, 2024 Merged
#1145 i.eodag: add timeout & wait options July 18, 2024 Merged
#1150 i.sentinel.download: use EODAG as a unified API for downloading sentinel products July 31, 2024 Open
#1152 i.eodag: additional options and modifications for i.sentinel.download August 1, 2024 Merged
#1156 i.landsat.download: put back settings option for credentials August 4, 2024 Merged
#1159 i.eodag: add dynamic setting for Creodias OTP August 6, 2024 Merged
#1163 i.eodag: add testsuite August 8, 2024 Open
#1165 i.eodag: update i.eodag to be compatible with EODAG 3 August 8, 2024 Merged
#1167 i.eodag: add -p flag to print query summary August 9, 2024 Merged
#1170 i.eodag: handle cases when sorting parameters are not available August 14, 2024 Merged

Reports

Community Bonding Period

What did I get done this week?

  • Finished Wiki setup.
  • Throughly read EODAG documentation, and got familiar with its capabilities.
  • Got familiar with the contribution guidelines.
  • Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
  • Setup development environment.

What do I plan on doing next week?

  • I plan on designing and building the initial structure for the i.eodag module.

Am I blocked on anything?

I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.

Meetings attended during the community bonding period:

  • May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
  • May 23rd 2024: Discussing module structure and features.

Week 1

What did I get done this week?

  • Built the initial version of i.eodag, found in #1090.
  • Added feature for searching & downloading by product (aka. dataset) type.
  • Added the following parameter for searching:
    • Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
    • AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
    • Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
  • Added feature for downloading products by their IDs.
  • Added option for the user to specify their custom EODAG config file.

What do I plan on doing next week?

  • Add cloud coverage parameter for searching.
  • Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
  • Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
  • Add an option to download products from a list of IDs saved in a text file.
  • Add handling of AOI as GeoJSON.

Am I blocked on anything?

Nothing, except the previously mentioned exams, but everything is going well.

Week 2

What did I get done this week?

  • Added search filtering for datasets with cloud coverage.
  • Added a method to download datasets, using a text file with a list of IDs.
  • Added printing metadata about the datasets found after searching.
  • All the mentioned above are addressed in #1097.

What do I plan on doing next week?

  • Continue working on adding AOI with Vector maps, and possibly GeoJSON.
  • Add an initial test suite, for the i.eodag module.

Am I blocked on anything?

No, and I will be done with my final exams by Wednesday this week.

Week 3

What did I get done this week?

  • Added support for setting the AOI based on a Vector Map, included in #1097.
  • Added support for two ways of searching:
    • General Search: searches through all the available providers for results.
    • Restricted Search: only searches through a predetermined – set by the user – provider.
  • Started working on a testsuite, for the i.eodag module.
  • i.eodag manual can be found here.

What do I plan on doing next week?

  • Wrap up the initial testsuite.
  • Add filtering & sorting for search results.
  • Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.

Am I blocked on anything?

I am not currently blocked on anything.

Week 4

What did I get done this week?

  • Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
  • Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
  • Added filtering for the search results using EODAG crunch, currently supported methods of filtering:
    • Start/End dates.
    • Cloud Cover.
    • Area Relation (contains - intersects - within).
    • Minimum overlap (Percentage of AOI covered by the scene).
  • Added Docstring for the module functions.
  • All the above changes can be found in #1104.
  • Attended – online – and gave a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
  • The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR #11.

What do I plan on doing next week?

  • Extend the query parameters.
  • Test the module and look into the products that might need case handling.
  • Add a way to get credentials on runtime (esp. Creodias as it requires OTP for authentication).

Am I blocked on anything?

I am not currently blocked by anything.

Week 5

What did I get done this week?

  • Added the “query” option, to allow extra search parameters #1136.
  • Added the “print” option allowing the listing of all relevant EODAG metadata #1136:
    • Current EODAG configuration.
    • Available products.
    • Available products for a given provider.
    • Available providers.
    • Available providers that offer a specific product.
    • Queryables for a given provider and/or product.
 Note: Queryables from this list can be used in the query option.
  • Added the “footprint” option to save the found scenes footprints in a vector map, with the name given in the “footprint” option #1136.
  • Added options to save/print results in JSON format #1104.

What do I plan on doing next week?

  • Continue working on adding a way to get credentials on runtime, as I haven’t received access to Creodias yet.
  • Merging the above-mentioned PRs, after further discussions with mentors.
  • Further testing and edge cases handling.

Am I blocked on anything?

I am not currently blocked by anything.

Week 6

What did I get done this week?

  • Created this PR #1136 to add more features including query, print, limit, and footprints.
  • Removed saving to JSON, and limiting it only to GeoJSON files.
  • Removed “extract” and “delete archive” flags, and instead used the values from the EODAG YAML config file.
  • Done more testing on i.eodag.
  • Worked on code refactoring and feedback.
  • Added use cases with the “print” option to i.eodag manual.

What do I plan on doing next week?

Am I blocked on anything?

I am not currently blocked by anything.

Week 7

What did I get done this week?

This week’s work was mostly focused on querying and updating the manual:

  • Added operators to the query option.
  • Added the option to use multiple values per queryable, to emulate the OR relation.
  • Added the option to use multiple queryables, to emulate the AND relation.
  • Added “querying” section to the manual with examples.
  • Added different use cases of i.eodag to the manual.
  • Code refactoring.
  • The above changes can be found in this PR #1142.

What do I plan on doing next week?

  • Further refactoring and working on feedback, as needed.
  • Continue planning for i.landsat.download, and starting with an initial reimplementation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 8

What did I get done this week?

  • Reimplemented i.landsat.download.
  • Supported Planetary Computer as a data source (USGS will be added as well once it is up again and the eodag fix reaches production eodag#1252).
  • Updated i.landsat.download manual.
  • Above changes can be found here #1144.

What do I plan on doing next week?

  • Add USGS support to i.landsat.download (couldn’t implement it yet because USGS is down, or there is some other problem).
  • Address this issue #17:
    • Add testsuite.
    • Add a method for providing credentials manually on runtime.
    • Update list_queryables function with more useful info.
  • Start experimenting with sentinel products, and doing necessary research for week 10 which should be the start of reimplementing i.sentinel.download .

Am I blocked on anything?

I am not currently blocked by anything.

Week 9

What did I get done this week?

  • Added USGS as a datasource to i.landsat.download.
  • Refactored i.landsat.download and the reimplementation is merged here #1144.
  • Added three new options to i.eodag #1145:
    • Pattern: To filter products using regular expressions, based on their ID.
    • Wait: Wait time in minutes before retrying to download data.
    • Timeout: If download fails, maximum time before stop retrying to download.

What do I plan on doing next week?

  • Start working on i.sentinel.download.
  • Address as much as possible of the issues mentioned here #17. (Couldn’t work on them last week, because of multiple travelling for a coding competition, ECPC - Egyptian Collegiate Programming Contest, but I will address them in parallel with i.sentinel.download).

Am I blocked on anything?

I am not currently blocked by anything.

Week 10

What did I get done this week?

  • Created an initial implementation of i.sentinel.download to replace sentinelsat and landsatxplore libraries by eodag #1150. To be completed in the upcoming week.
  • Added new flags to i.eodag #1152:
    • -b flag: to use the AOI borders instead of the bounding box.
    • -s flag: to skip already downloaded products from the search result list.
    • Added “footprint” sorting criteria to i.eodag to sort scenes according to their id lexicographically.
  • Added the settings option back to i.landsat.download, to set credentials #1156, which is also reimplemented in i.sentinel.download #1150.

What do I plan on doing next week?

  • Continue working on i.sentinel.download.
  • Add a testsuite for i.edoag #11.
  • Work on merging the current open PRs.

Am I blocked on anything?

I am not currently blocked by anything.

Week 11

What did I get done this week?

  • Added testsuite to i.eodag #1163, to be completed in the upcoming week.
  • Put back the “settings” option to i.landsat.download that was initially removed after reimplementation. #1156
  • Added -p flag to i.eodag to print a query summary and exit. #1167
  • Started working on this PR #1165 to update i.eodag to be compatible with the, to be released, EODAG 3.0.0, in response to the EODAG 3 beta versions.
  • Added Creodias provider support to i.eodag, by supporting authentication with TOTP. #1159
  • Updated i.sentinel.download manual after the reimplementation and continued working on the i.sentinel.download open PR. #1150

What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Refactoring and documentation.

Am I blocked on anything?

I am not currently blocked by anything.

Week 12

What did I get done this week?

  • Continued working on the i.eodag testsuite #1163:
    • Modified tests to be run only if the connection to providers of concern can be established.
    • Implemented more generic and robust tests, that will be more resilient to changes from the provider side which shouldn’t fail the tests.
  • Handled the edge case of when a sorting parameter is missing in one scene or more. #1170
  • Wrapped up this PR #1165 to ensure the compatibility of i.eodag with EODAG 3.
  • Continued working on the i.sentinel.download reimplementation, and mapped the sleep and retry options from i.sentinel.download to the wait and timeout options from i.eodag. #1150


What do I plan on doing next week?

  • Continue working on merging the current open PRs, and adjusting according to feedback.
  • Write and submit the final report.

Am I blocked on anything?

I am not currently blocked by anything.

Final Report

Title: Add EODAG Support to GRASS GIS

Community: GRASS GIS - OSGeo

Abstract

This project introduces a new subpackage for GRASS GIS, i.eodag that integrates the EODAG API with GRASS GIS acting as a portal for downloading geospatial scenes through GRASS GIS. Previously, to download geospatial scenes that had to be done through three main modules, i.landsat.download, i.sentinel.download, and i.modis.download, and each of them had to use different APIs, which resulted in a lot of effort that had to be put into maintaining the different APIs. So here comes i.eodag to act as the main source of all geospatial products, and instead of maintaining all the different APIs, now only the EODAG API has to be sufficiently maintained. Moreover, i.landsat.download & i.sentinel.download have also been reimplemented, while abstracting all the logic, using the new i.eodag subpackage.

The state of integration BEFORE the start of GSoC =

The state of integration AFTER GSoC

Future Work

Links

Images

Media