GRASS GSoC 2024 EODAG Support: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
 
(12 intermediate revisions by the same user not shown)
Line 13: Line 13:
|}
|}


__TOC__


== Abstract ==
= Abstract =
The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.
The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.


== Timeline ==
= Timeline =
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 28: Line 27:
# Further familiarizing with the GRASS development environment
# Further familiarizing with the GRASS development environment
# Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
# Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
  || Ok
  || OK
|-
|-
| rowspan="12" | '''Official Coding Period'''
| rowspan="12" | '''Official Coding Period'''
Line 34: Line 33:
# Write an initial i.eo.download addon module using USGS provider and Landsat data  
# Write an initial i.eo.download addon module using USGS provider and Landsat data  
# Limited working time due to final exams
# Limited working time due to final exams
  || Ok
  || OK
|-
|-
| '''Week 2''' (June 2 - June 8) ||  
| '''Week 2''' (June 2 - June 8) ||  
# Add printing results and saving it in GeoJSON format.
# Add printing results and saving it in GeoJSON format.
# Add support for GeoJSON as an AOI parameter.
# <s>Add support for GeoJSON as an AOI parameter.</s>
# Add downloading products from a file by IDs.
# Add downloading products from a file by IDs.
# Work on feedback
# Work on feedback
# Limited working time due to final exams
# Limited working time due to final exams
  ||
  || OK<br><br>OK<br>OK
|-
|-
| '''Week 3''' (June 9 - June 15) ||  
| '''Week 3''' (June 9 - June 15) ||  
Line 48: Line 47:
# Work on feedback
# Work on feedback
# Almost done with final exams
# Almost done with final exams
  ||
  || OK<br>OK
|-
|-
| '''Week 4''' (June 16 - June 22) ||  
| '''Week 4''' (June 16 - June 22) ||  
Line 56: Line 55:
# Testing, refactoring and bug fixing
# Testing, refactoring and bug fixing
# Work on feedback
# Work on feedback
  ||
  ||OK<br>OK<br>POSTPONED<br>OK<br>OK<br>
|-
|-
| '''Week 5''' (June 23 - June 29) ||  
| '''Week 5''' (June 23 - June 29) ||  
Line 105: Line 104:
|}
|}


== Log of Pull Requests ==
= Pull Requests Log =
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 111: Line 110:
|-
|-
| [https://github.com/OSGeo/grass-addons/pull/1090 #1090]
| [https://github.com/OSGeo/grass-addons/pull/1090 #1090]
|| Week 1: add i.eodag module  
|| i.eodag: add i.eodag module  
|| May 27th, 2024  
|| May 27th, 2024  
|| Merged
|| Merged
|-
|-
| [https://github.com/OSGeo/grass-addons/pull/1097 #1097]
| [https://github.com/OSGeo/grass-addons/pull/1097 #1097]
|| Week 2: add files support
|| i.eodag: add files support
|| June 9th, 2024  
|| June 9th, 2024  
|| Merged
|-
| [https://github.com/OSGeo/grass-addons/pull/1104 #1104]
|| i.eodag: add sorting and filtering options
|| June 16th, 2024
|| Open
|| Open
|-
| [https://github.com/HamedElgizery/grass-addons/pull/11 #11]
|| i.eodag: add testsuite
|| June 16th, 2024
|| Draft
|-
| [https://github.com/HamedElgizery/grass-addons/pull/13 #13]
|| i.eodag: add importing footprints
|| June 24th, 2024
|| Draft
|}
|}


== Reports ==
= Reports =
== Community Bonding Period ==
== Community Bonding Period ==
==== What did I get done this week? ====
'''What did I get done this week?'''
 
* Finished Wiki setup.
* Finished Wiki setup.
* Throughly read EODAG documentation, and got familiar with its capabilities.
* Throughly read EODAG documentation, and got familiar with its capabilities.
Line 131: Line 144:
* Setup development environment.
* Setup development environment.


==== What do I plan on doing next week? ====
'''What do I plan on doing next week?'''
* I plan on designing and building the initial structure for the i.eodag module.
* I plan on designing and building the initial structure for the i.eodag module.


==== Am I blocked on anything? ====
'''Am I blocked on anything?'''
:I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.
:I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.
==== Meetings attended during the community bonding period: ====
'''Meetings attended during the community bonding period:'''
* May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
* May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
* May 23rd 2024: Discussing module structure and features.
* May 23rd 2024: Discussing module structure and features.


== Week 1 ==
== Week 1 ==
Following is the week 1 report, for adding EODAG support to GRASS GIS, GSoC 2024.
Following is the report for Week 1.


==== What did I get done this week? ====
'''What did I get done this week?'''
 
* Built the initial version of i.eodag, found in [https://github.com/OSGeo/grass-addons/pull/1090 #1090].
* Built the initial version of i.eodag, found in this pull request [1].
* Added feature for searching & downloading by product (aka. dataset) type.
* Added feature for searching & downloading by product (aka. dataset) type.
* Added the following parameter for searching:
* Added the following parameter for searching:
Line 152: Line 164:
** Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
** Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
* Added feature for downloading products by their IDs.
* Added feature for downloading products by their IDs.
* Added option for the user to specify their custom EODAG config file [2].
* Added option for the user to specify their custom [https://eodag.readthedocs.io/en/stable/getting_started_guide/configure.html EODAG config file].


==== What do I plan on doing next week? ====
'''What do I plan on doing next week?'''
* Add cloud coverage parameter for searching.
* Add cloud coverage parameter for searching.
* Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
* Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
Line 161: Line 173:
* Add handling of AOI as GeoJSON.
* Add handling of AOI as GeoJSON.


==== Am I blocked on anything? ====
'''Am I blocked on anything?'''
:Nothing, except the previously mentioned exams, but everything is going well.
:Nothing, except the previously mentioned exams, but everything is going well.


== Week 2 ==
Following is the report for Week 2 report.


== Week 2 ==
'''What did I get done this week?'''
Following is the week #2 report, for adding EODAG support to GRASS GIS, GSoC 2024.
==== What did I get done this week? ====
* Added search filtering for datasets with cloud coverage.
* Added search filtering for datasets with cloud coverage.
* Added a method to download datasets, using a text file with a list of IDs.
* Added a method to download datasets, using a text file with a list of IDs.
* Added printing metadata about the datasets found after searching.
* Added printing metadata about the datasets found after searching.
* All the mentioned above are addressed in this PR [1].
* All the mentioned above are addressed in [https://github.com/OSGeo/grass-addons/pull/1097 #1097].
==== What do I plan on doing next week? ====
 
'''What do I plan on doing next week?'''
* Continue working on adding AOI with Vector maps, and possibly GeoJSON.
* Continue working on adding AOI with Vector maps, and possibly GeoJSON.
* Add an initial test suite, for the i.eodag module.
* Add an initial test suite, for the i.eodag module.
==== Am I blocked on anything? ====
 
'''Am I blocked on anything?'''
:No, and I will be done with my final exams by Wednesday this week.
:No, and I will be done with my final exams by Wednesday this week.
== Week 3 ==
== Week 3 ==
Here is my weekly report for week #3, adding EODAG support to GRASS GIS, GSoC 2024.
Here is my weekly report for Week 3.
==== What did I get done this week? ====


* Added support for setting the AOI based on a Vector Map [1].
'''What did I get done this week?'''
* Added support for setting the AOI based on a Vector Map, included in [https://github.com/OSGeo/grass-addons/pull/1097 #1097].
* Added support for two ways of searching:
* Added support for two ways of searching:
** General Search: searches through all the available providers for results.
** General Search: searches through all the available providers for results.
Line 188: Line 203:
* i.eodag manual can be found [https://github.com/OSGeo/grass-addons/pull/1097 here].
* i.eodag manual can be found [https://github.com/OSGeo/grass-addons/pull/1097 here].


==== What do I plan on doing next week? ====
'''What do I plan on doing next week?'''
 
* Wrap up the initial testsuite.
* Wrap up the initial testsuite.
* Add filtering & sorting for search results.
* Add filtering & sorting for search results.
* Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024. [2]
* Attend – online – and give a brief presentation about the i.eodag module, for the [[GRASS_Community_Meeting_Prague_2024#Monday,_June_17|GRASS Community Meeting Prague 2024]].
 
'''Am I blocked on anything?'''
:I am not currently blocked on anything.
 
== Week 4 ==
Here is my weekly report for Week 4.
 
'''What did I get done this week?'''
* Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
* Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
* Added filtering for the search results using [https://eodag.readthedocs.io/en/latest/plugins_reference/crunch.html EODAG crunch], currently supported methods of filtering:
** Start/End dates.
** Cloud Cover.
** Area Relation (contains - intersects - within).
** Minimum overlap (Percentage of AOI covered by the scene).
* Added Docstring for the module functions.
* All the above changes can be found in [https://github.com/OSGeo/grass-addons/pull/1104 #1104].
* Attended – online – and gave a brief presentation about the i.eodag module, for the [[GRASS_Community_Meeting_Prague_2024#Monday,_June_17|GRASS Community Meeting Prague 2024]].
* The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR [https://github.com/HamedElgizery/grass-addons/pull/11 #11].
 
'''What do I plan on doing next week?'''
* Extend the query parameters.
* Test the module and look into the products that might need case handling.
* Add a way to get credentials on runtime (esp. [https://creodias.eu/ Creodias] as it requires OTP for authentication).


==== Am I blocked on anything? ====
'''Am I blocked on anything?'''
:I am not currently blocked by anything.
:I am not currently blocked by anything.


{{GSoC}}
{{GSoC}}

Latest revision as of 23:11, 24 June 2024

Accepted Google Summer of Code 2024 project.

Student Name: Hamed Ashraf Elgizery, AASTMT, Smart-Village, Giza, Egypt
Organization: OSGeo - Open Source Geospatial Foundation
Mentor Name: Luca Delucchi, Veronica Andreo, Stefan Blumentrath
GitHub Fork: View Fork
GSoC Proposal View Proposal


Abstract

The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.

Timeline

Period Timeline Tasks Status
Bonding Period May 1 - May 26
  1. Researching
  2. Reading GRASS documentation
  3. Further familiarizing with the GRASS development environment
  4. Familiarizing with the EODAG API and how it can be related to GRASS GIS concepts
 OK
Official Coding Period Week 1 (May 27 - June 1)
  1. Write an initial i.eo.download addon module using USGS provider and Landsat data
  2. Limited working time due to final exams
 OK
Week 2 (June 2 - June 8)
  1. Add printing results and saving it in GeoJSON format.
  2. Add support for GeoJSON as an AOI parameter.
  3. Add downloading products from a file by IDs.
  4. Work on feedback
  5. Limited working time due to final exams
 OK

OK
OK
Week 3 (June 9 - June 15)
  1. Add initial test cases and documentation / manual
  2. Work on feedback
  3. Almost done with final exams
 OK
OK
Week 4 (June 16 - June 22)
  1. Add support for Grass Map as an AOI parameter.
  2. Add filtering by AOI relation type (intersect, contain, and within).
  3. Work on getting credentials on runtime.
  4. Testing, refactoring and bug fixing
  5. Work on feedback
 OK
OK
POSTPONED
OK
OK
Week 5 (June 23 - June 29)
  1. Add generic support for Queryables
  2. Add (to be chosen) features according to the new supported Queryables.
  3. Work on feedback
Week 6 (June 30 - July 6)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
Week 7 (July 7 - July 13)
  1. Pre-evaluation reformatting and documenting
  2. Mid-term Evaluation: July 12
Week 8 (July 14 - July 20)
  1. Reimplement i.landsat.download
  1. Work on feedback
Week 9 (July 21 - July 27)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
Week 10 (July 28 - August 3)
  1. Reimplement i.sentinel.download
  2. Work on feedback
Week 11 (August 4 - August 10)
  1. Testing, refactoring and bug fixing
  2. Work on feedback
Week 12 (August 11 - August 17)
  1. Work on feedback
  2. Final reformatting and documenting
Evaluation Period August 26 - September 2 Submit code and final evaluation

Pull Requests Log

Pull Request Description Date Status
#1090 i.eodag: add i.eodag module May 27th, 2024 Merged
#1097 i.eodag: add files support June 9th, 2024 Merged
#1104 i.eodag: add sorting and filtering options June 16th, 2024 Open
#11 i.eodag: add testsuite June 16th, 2024 Draft
#13 i.eodag: add importing footprints June 24th, 2024 Draft

Reports

Community Bonding Period

What did I get done this week?

  • Finished Wiki setup.
  • Throughly read EODAG documentation, and got familiar with its capabilities.
  • Got familiar with the contribution guidelines.
  • Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
  • Setup development environment.

What do I plan on doing next week?

  • I plan on designing and building the initial structure for the i.eodag module.

Am I blocked on anything?

I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.

Meetings attended during the community bonding period:

  • May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
  • May 23rd 2024: Discussing module structure and features.

Week 1

Following is the report for Week 1.

What did I get done this week?

  • Built the initial version of i.eodag, found in #1090.
  • Added feature for searching & downloading by product (aka. dataset) type.
  • Added the following parameter for searching:
    • Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
    • AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
    • Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
  • Added feature for downloading products by their IDs.
  • Added option for the user to specify their custom EODAG config file.

What do I plan on doing next week?

  • Add cloud coverage parameter for searching.
  • Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
  • Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
  • Add an option to download products from a list of IDs saved in a text file.
  • Add handling of AOI as GeoJSON.

Am I blocked on anything?

Nothing, except the previously mentioned exams, but everything is going well.

Week 2

Following is the report for Week 2 report.

What did I get done this week?

  • Added search filtering for datasets with cloud coverage.
  • Added a method to download datasets, using a text file with a list of IDs.
  • Added printing metadata about the datasets found after searching.
  • All the mentioned above are addressed in #1097.

What do I plan on doing next week?

  • Continue working on adding AOI with Vector maps, and possibly GeoJSON.
  • Add an initial test suite, for the i.eodag module.

Am I blocked on anything?

No, and I will be done with my final exams by Wednesday this week.

Week 3

Here is my weekly report for Week 3.

What did I get done this week?

  • Added support for setting the AOI based on a Vector Map, included in #1097.
  • Added support for two ways of searching:
    • General Search: searches through all the available providers for results.
    • Restricted Search: only searches through a predetermined – set by the user – provider.
  • Started working on a testsuite, for the i.eodag module.
  • i.eodag manual can be found here.

What do I plan on doing next week?

  • Wrap up the initial testsuite.
  • Add filtering & sorting for search results.
  • Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.

Am I blocked on anything?

I am not currently blocked on anything.

Week 4

Here is my weekly report for Week 4.

What did I get done this week?

  • Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
  • Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
  • Added filtering for the search results using EODAG crunch, currently supported methods of filtering:
    • Start/End dates.
    • Cloud Cover.
    • Area Relation (contains - intersects - within).
    • Minimum overlap (Percentage of AOI covered by the scene).
  • Added Docstring for the module functions.
  • All the above changes can be found in #1104.
  • Attended – online – and gave a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
  • The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR #11.

What do I plan on doing next week?

  • Extend the query parameters.
  • Test the module and look into the products that might need case handling.
  • Add a way to get credentials on runtime (esp. Creodias as it requires OTP for authentication).

Am I blocked on anything?

I am not currently blocked by anything.
Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=GRASS_GSoC_2024_EODAG_Support&oldid=27723"