GRASS GSoC 2024 EODAG Support: Difference between revisions
(Update for Week 11) |
mNo edit summary |
||
Line 93: | Line 93: | ||
# Testing, refactoring and bug fixing | # Testing, refactoring and bug fixing | ||
# Work on feedback | # Work on feedback | ||
||OK<br>OK | |||
|| | |||
|- | |- | ||
| '''Week 12''' (August 11 - August 17) || | | '''Week 12''' (August 11 - August 17) || |
Revision as of 20:39, 11 August 2024
Accepted Google Summer of Code 2024 project.
Student Name: | Hamed Ashraf Elgizery, AASTMT, Smart-Village, Giza, Egypt |
Organization: | OSGeo - Open Source Geospatial Foundation |
Mentor Name: | Luca Delucchi, Veronica Andreo, Stefan Blumentrath |
GitHub Fork: | View Fork |
GSoC Proposal | View Proposal |
Abstract
The project focuses on writing a Python library for GRASS GIS that utilizes EODAG API. Currently, GRASS uses different APIs for each satellite data provider. As a result, a lot of maintenance has to be done to keep up with the changes happening to all the different APIs. Thus, adding EODAG support to GRASS GIS will provide a lot of flexibility in the future development of GRASS, by decreasing the amount of maintenance needed to keep up with all the different APIs updated, and instead, sufficiently, maintain the support for EODAG library. Deliverables for this proposal are supporting Sentinel, Landsat, and MODIS datasets with EODAG, and reimplementation of both i.sentinel.download and i.landsat.download modules with the new GRASS EODAG library.
Timeline
Period | Timeline | Tasks | Status |
---|---|---|---|
Bonding Period | May 1 - May 26 |
|
OK |
Official Coding Period | Week 1 (May 27 - June 1) |
|
OK |
Week 2 (June 2 - June 8) |
|
OK OK OK | |
Week 3 (June 9 - June 15) |
|
OK OK | |
Week 4 (June 16 - June 22) |
|
OK OK POSTPONED OK OK | |
Week 5 (June 23 - June 29) |
|
OK OK OK | |
Week 6 (June 30 - July 6) |
|
OK OK | |
Week 7 (July 7 - July 13) |
|
OK OK | |
Week 8 (July 14 - July 20) |
|
OK OK | |
Week 9 (July 21 - July 27) |
|
OK OK | |
Week 10 (July 28 - August 3) |
|
OK OK | |
Week 11 (August 4 - August 10) |
|
OK OK | |
Week 12 (August 11 - August 17) |
|
||
Evaluation Period | August 26 - September 2 | Submit code and final evaluation |
Pull Requests Log
Pull Request | Description | Date | Status |
---|---|---|---|
#1090 | i.eodag: add i.eodag module | May 27, 2024 | Merged |
#1097 | i.eodag: add files support | June 9, 2024 | Merged |
#1104 | i.eodag: remove pandas dependency and save results in json/geojson files | June 16, 2024 | Merged |
#11 | i.eodag: add testsuite | June 16, 2024 | Moved to #1163 |
#13 | i.eodag: add importing footprints | June 24, 2024 | Moved to #1136 |
#15 | i.eodag: add queryables | June 30, 2024 | Moved to #1136 |
#1136 | i.eodag: add query, print, and footprints options | July 2, 2024 | Merged |
#1142 | i.eodag: add operators to queryables and allow multiple values | July 11, 2024 | Merged |
#1144 | i.landsat.download: change API from landsatxplore to EODAG using i.eodag | July 17, 2024 | Merged |
#1145 | i.eodag: add timeout & wait options | July 18, 2024 | Merged |
#1150 | i.sentinel.download: use EODAG as a unified API for downloading sentinel products | July 31, 2024 | Open |
#1152 | i.eodag: additional options and modifications for i.sentinel.download | August 1, 2024 | Merged |
#1156 | i.landsat.download: put back settings option for credentials | August 4, 2024 | Open |
#1159 | i.eodag: add dynamic setting for Creodias OTP | August 6, 2024 | Merged |
#1163 | i.eodag: add testsuite | August 8, 2024 | Draft |
#1165 | i.eodag: update i.eodag to be compatible with EODAG 3 | August 8, 2024 | Draft |
#1167 | i.eodag: add -p flag to print query summary | August 9, 2024 | Open |
Reports
Community Bonding Period
What did I get done this week?
- Finished Wiki setup.
- Throughly read EODAG documentation, and got familiar with its capabilities.
- Got familiar with the contribution guidelines.
- Created an initial add-on, with some basic features of EODAG, and went through the compilation process.
- Setup development environment.
What do I plan on doing next week?
- I plan on designing and building the initial structure for the i.eodag module.
Am I blocked on anything?
- I will be having my final exams for the next two weeks. I have started working on the module and done extra research so that I can extend my work during the exams period. I won’t stop working during that period, but I might not have as much time, so I will get some extra work done before and after the exams to catch up, and so no work will be compromised.
Meetings attended during the community bonding period:
- May 16th 2024: Introductory meeting about EODAG, and what are the expected results, and how will the add-on be used.
- May 23rd 2024: Discussing module structure and features.
Week 1
Following is the report for Week 1.
What did I get done this week?
- Built the initial version of i.eodag, found in #1090.
- Added feature for searching & downloading by product (aka. dataset) type.
- Added the following parameter for searching:
- Start time and end time. Users can specify any time that complies with ISO 8601, including adding the timezone.
- AOI. As for now, it uses the current computational region, and supporting GeoJSON will be added next week.
- Provider. Currently specifying a provider is required by the user, and search won’t be done without it. It is planned to add a way to search broadly without specifying a provider.
- Added feature for downloading products by their IDs.
- Added option for the user to specify their custom EODAG config file.
What do I plan on doing next week?
- Add cloud coverage parameter for searching.
- Add printing for relevant metadata information, regarding the products found by searching, and relevant configurations set by the user.
- Add saving results as a GeoJSON, allowing users to save the search results locally and then load them later, to perform downloading or further filtering.
- Add an option to download products from a list of IDs saved in a text file.
- Add handling of AOI as GeoJSON.
Am I blocked on anything?
- Nothing, except the previously mentioned exams, but everything is going well.
Week 2
Following is the report for Week 2 report.
What did I get done this week?
- Added search filtering for datasets with cloud coverage.
- Added a method to download datasets, using a text file with a list of IDs.
- Added printing metadata about the datasets found after searching.
- All the mentioned above are addressed in #1097.
What do I plan on doing next week?
- Continue working on adding AOI with Vector maps, and possibly GeoJSON.
- Add an initial test suite, for the i.eodag module.
Am I blocked on anything?
- No, and I will be done with my final exams by Wednesday this week.
Week 3
Here is my weekly report for Week 3.
What did I get done this week?
- Added support for setting the AOI based on a Vector Map, included in #1097.
- Added support for two ways of searching:
- General Search: searches through all the available providers for results.
- Restricted Search: only searches through a predetermined – set by the user – provider.
- Started working on a testsuite, for the i.eodag module.
- i.eodag manual can be found here.
What do I plan on doing next week?
- Wrap up the initial testsuite.
- Add filtering & sorting for search results.
- Attend – online – and give a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
Am I blocked on anything?
- I am not currently blocked on anything.
Week 4
Here is my weekly report for Week 4.
What did I get done this week?
- Removed the Pandas dependency after agreeing that removing unnecessary dependencies would make the module lighter.
- Added sorting for the search results by “Cloud Cover” & “Ingestion Date”.
- Added filtering for the search results using EODAG crunch, currently supported methods of filtering:
- Start/End dates.
- Cloud Cover.
- Area Relation (contains - intersects - within).
- Minimum overlap (Percentage of AOI covered by the scene).
- Added Docstring for the module functions.
- All the above changes can be found in #1104.
- Attended – online – and gave a brief presentation about the i.eodag module, for the GRASS Community Meeting Prague 2024.
- The testsuite is currently drafted, as it will be edited further when adding more features. Here is the draft PR #11.
What do I plan on doing next week?
- Extend the query parameters.
- Test the module and look into the products that might need case handling.
- Add a way to get credentials on runtime (esp. Creodias as it requires OTP for authentication).
Am I blocked on anything?
- I am not currently blocked by anything.
Week 5
Here is my weekly report for Week 5.
What did I get done this week?
- Added the “query” option, to allow extra search parameters #1136.
- Added the “print” option allowing the listing of all relevant EODAG metadata #1136:
- Current EODAG configuration.
- Available products.
- Available products for a given provider.
- Available providers.
- Available providers that offer a specific product.
- Queryables for a given provider and/or product.
Note: Queryables from this list can be used in the query option.
- Added the “footprint” option to save the found scenes footprints in a vector map, with the name given in the “footprint” option #1136.
- Added options to save/print results in JSON format #1104.
What do I plan on doing next week?
- Continue working on adding a way to get credentials on runtime, as I haven’t received access to Creodias yet.
- Merging the above-mentioned PRs, after further discussions with mentors.
- Further testing and edge cases handling.
Am I blocked on anything?
- I am not currently blocked by anything.
Week 6
Here is my weekly report for Week 6.
What did I get done this week?
- Created this PR #1136 to add more features including query, print, limit, and footprints.
- Removed saving to JSON, and limiting it only to GeoJSON files.
- Removed “extract” and “delete archive” flags, and instead used the values from the EODAG YAML config file.
- Done more testing on i.eodag.
- Worked on code refactoring and feedback.
- Added use cases with the “print” option to i.eodag manual.
What do I plan on doing next week?
- Start investigating possible ways to reimplement i.landsat.download.
- Adjust i.eodag according to how things turn out when reimplementing i.landsat.download.
- Describe more use cases of i.eodag in the manual.
Am I blocked on anything?
- I am not currently blocked by anything.
Week 7
Here is my weekly report for Week 7.
What did I get done this week?
This week’s work was mostly focused on querying and updating the manual:
- Added operators to the query option.
- Added the option to use multiple values per queryable, to emulate the OR relation.
- Added the option to use multiple queryables, to emulate the AND relation.
- Added “querying” section to the manual with examples.
- Added different use cases of i.eodag to the manual.
- Code refactoring.
- The above changes can be found in this PR #1142.
What do I plan on doing next week?
- Further refactoring and working on feedback, as needed.
- Continue planning for i.landsat.download, and starting with an initial reimplementation.
Am I blocked on anything?
- I am not currently blocked by anything.
Week 8
Here is my weekly report for Week 8.
What did I get done this week?
- Reimplemented i.landsat.download.
- Supported Planetary Computer as a data source (USGS will be added as well once it is up again and the eodag fix reaches production eodag#1252).
- Updated i.landsat.download manual.
- Above changes can be found here #1144.
What do I plan on doing next week?
- Add USGS support to i.landsat.download (couldn’t implement it yet because USGS is down, or there is some other problem).
- Address this issue #17:
- Add testsuite.
- Add a method for providing credentials manually on runtime.
- Update list_queryables function with more useful info.
- Start experimenting with sentinel products, and doing necessary research for week 10 which should be the start of reimplementing i.sentinel.download .
Am I blocked on anything?
- I am not currently blocked by anything.
Week 9
Here is my weekly report for Week 9.
What did I get done this week?
- Added USGS as a datasource to i.landsat.download.
- Refactored i.landsat.download and the reimplementation is merged here #1144.
- Added three new options to i.eodag #1145:
- Pattern: To filter products using regular expressions, based on their ID.
- Wait: Wait time in minutes before retrying to download data.
- Timeout: If download fails, maximum time before stop retrying to download.
What do I plan on doing next week?
- Start working on i.sentinel.download.
- Address as much as possible of the issues mentioned here #17. (Couldn’t work on them last week, because of multiple travelling for a coding competition, ECPC - Egyptian Collegiate Programming Contest, but I will address them in parallel with i.sentinel.download).
Am I blocked on anything?
- I am not currently blocked by anything.
Week 10
Here is my weekly report for Week 10.
What did I get done this week?
- Created an initial implementation of i.sentinel.download to replace sentinelsat and landsatxplore libraries by eodag #1150. To be completed in the upcoming week.
- Added new flags to i.eodag #1152:
- -b flag: to use the AOI borders instead of the bounding box.
- -s flag: to skip already downloaded products from the search result list.
- Added “footprint” sorting criteria to i.eodag to sort scenes according to their id lexicographically.
- Added the settings option back to i.landsat.download, to set credentials #1156, which is also reimplemented in i.sentinel.download #1150.
What do I plan on doing next week?
- Continue working on i.sentinel.download.
- Add a testsuite for i.edoag #11.
- Work on merging the current open PRs.
Am I blocked on anything?
- I am not currently blocked by anything.
Week 11
Here is my weekly report for Week 11.
What did I get done this week?
- Added testsuite to i.eodag #1163, to be completed in the upcoming week.
- Put back the “settings” option to i.landsat.download that was initially removed after reimplementation. #1156
- Added -p flag to i.eodag to print a query summary and exit. #1167
- Started working on this PR #1165 to update i.eodag to be compatible with the, to be released, EODAG 3.0.0, in response to the EODAG 3 beta versions.
- Added Creodias provider support to i.eodag, by supporting authentication with TOTP. #1159
- Updated i.sentinel.download manual after the reimplementation and continued working on the i.sentinel.download open PR. #1150
What do I plan on doing next week?
- Continue working on merging the current open PRs, and adjusting according to feedback.
- Refactoring and documentation.
Am I blocked on anything?
- I am not currently blocked by anything.