User:Fæ/Project list/ESA

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search
Diamond ring effect, during 22 Aug 2017 total eclipse. ESA team in Wyoming, US.
The largest file in this upload project is a Sentinel-2A image of Saint Petersburg in snow, 2017. The file is 690 MB and a resolution of 10,980 × 10,980 pixels.

This batch upload project populates ESA images (review needed) and Copernicus Sentinel Satellite Imagery. Questions can be raised at User talk:Fæ as the uploader.

The batch upload script has been converted to run automatically once a day, to upload the latest available ESA images to Commons. See petscan report for images uploaded in the last week.

European Space Agency

[edit]

Based on a suggestion at the Village Pump, a specific upload script has been designed to take information from ESA catalogue pages, initially for Copernicus Sentinel Data images. The script is in Python using Pywikibot core and BeautifulSoup. In its initial form it checks for images and will skip video.

The copyright of files is checked as being "CC BY-SA 3.0 IGO" and this appears to apply to all catalogue text. The html of the catalogue pages includes a non-visible copyright meta tag of "All rights reserved." The presumption is that this is a default but not intended to overrule the visible copyright statement. A general statement about ESA copyright is available at http://www.esa.int/spaceinimages/ESA_Multimedia/Copyright_Notice_Images.

The sample url is http://www.esa.int/spaceinimages/content/search/(offset)/0?SearchText=Copernicus+Sentinel+Data&sortBy=published where the offset number (0) is the number of the first image in the search. Grid pages are 16 images per page.

Critical data for the import is:

  • title, taken from og:title
  • date, taken from release date
  • Id, taken from Id in catalogue
  • description, from the description in the catalogue
  • copyright, from the copyright statement in the catalogue

For Commons the filename will be "File:<title> ESA<id>.<ext>" where ext is expected to be jpg, gif, PNG or TIFF. The priority for selecting a format will be TIFF > PNG > gif > jpg. Due to a known issue with the potential poor display of TIFF thumbnails for reuse on Wikipedia, where both a TIFF and a ESA supplied jpg exists, both will be uploaded.

There are a number of tags, such as mission and keywords, which are added as additional values in the information template. As the initial run is for fewer than 300 images, the complexity of metadata import is being kept simple and categorization limited to a bucket category to be manually sorted later, if desirable.

Technical points

[edit]
  • The domain for jpegs is a local link, the same as the search, but for tiffs it's a different domain so external.
  • The TIFFs may be large, so local SHA values will not be used to check for existing duplicates on Commons in advance of attempting API upload; because this requires downloading the file locally anyway. If the IDs are unique, then searching Commons for any "ESA<id>.<ext>" matches should be sufficient as a weaker and quicker test.
  • Direct uploading by url requires the relevant image hosting ESA domains to be whitelisted, this has been done on Phab:T164643.

Rejections

[edit]
[edit]

Tested: 22:54, 5 May 2017 (UTC)

Out of the 228 images in the original suggested search, the following do not (currently) contain a CC BY-SA 3.0 IGO statement:

File list
  1. http://www.esa.int/spaceinimages/Images/2017/05/Ice_speed
  2. http://www.esa.int/spaceinimages/Images/2017/04/Sentinel-1_sees_Mocoa_landslide
  3. http://www.esa.int/spaceinimages/Images/2017/04/TimeScan_product_Germany
  4. http://www.esa.int/spaceinimages/Images/2017/03/Kaikoura_quake
  5. http://www.esa.int/spaceinimages/Images/2017/03/3D_ground_displacement_using_Sentinel-1_data
  6. http://www.esa.int/spaceinimages/Images/2017/02/Shelf_deformation
  7. http://www.esa.int/spaceinimages/Images/2017/02/Halloween_Crack
  8. http://www.esa.int/spaceinimages/Images/2016/06/Iceberg_A56_June
  9. http://www.esa.int/spaceinimages/Images/2016/06/La_Reunion_Island_via_laser2
  10. http://www.esa.int/spaceinimages/Images/2016/06/La_Reunion_Island_via_laser
  11. http://www.esa.int/spaceinimages/Images/2016/06/Uprint_Calendar_-_June_2016
  12. http://www.esa.int/spaceinimages/Images/2016/03/Tracking_Tim_s_iceberg
  13. http://www.esa.int/spaceinimages/Images/2016/03/Europe_viewed_wide
  14. http://www.esa.int/spaceinimages/Images/2016/03/Ice_cracked
  15. http://www.esa.int/spaceinimages/Images/2016/03/Islands_in_red
  16. http://www.esa.int/spaceinimages/Images/2016/03/Wave_height
  17. http://www.esa.int/spaceinimages/Images/2016/03/Sea-level_track_from_Sentinel-3A
  18. http://www.esa.int/spaceinimages/Images/2016/03/Strait_of_Gibraltar_from_Sentinel-3A
  19. http://www.esa.int/spaceinimages/Images/2016/03/California_from_Sentinel-3A
  20. http://www.esa.int/spaceinimages/Images/2016/03/Iberian_Peninsula
  21. http://www.esa.int/spaceinimages/Images/2016/03/First_image_from_Sentinel-3A
  22. http://www.esa.int/spaceinimages/Images/2016/03/Uprint_Calendar_-_March_2016
  23. http://www.esa.int/spaceinimages/Images/2016/02/Barcelona_from_Sentinel-2A
  24. http://www.esa.int/spaceinimages/Images/2015/12/Dutch_mosaic
  25. http://www.esa.int/spaceinimages/Images/2015/11/Paris_in_focus
  26. http://www.esa.int/spaceinimages/Images/2015/11/Merida_Spain
  27. http://www.esa.int/spaceinimages/Images/2015/11/Montreal_Canada
  28. http://www.esa.int/spaceinimages/Images/2015/10/Manicouagan_Crater_Canada
  29. http://www.esa.int/spaceinimages/Images/2015/10/Deep_blue_Red_Sea_reefs
  30. http://www.esa.int/spaceinimages/Images/2015/08/Sentinel-2A_spots_ship_traffic
  31. http://www.esa.int/spaceinimages/Images/2015/08/Sentinel-1A_spots_ship_traffic
  32. http://www.esa.int/spaceinimages/Images/2015/08/Danube_River
  33. http://www.esa.int/spaceinimages/Images/2015/07/Sentinel-2_for_agriculture
  34. http://www.esa.int/spaceinimages/Images/2015/07/Watching_coastal_waters
  35. http://www.esa.int/spaceinimages/Images/2015/07/Monitoring_inland_water_bodies

GIFs

[edit]
GIF animation of the building of the Third Bosphorus Bridge in Istanbul, Turkey. Sentinel-2 images, 2015—2016.

GIF images frequently can fail post-upload verification, for example:

  1. http://www.esa.int/spaceinimages/Images/2017/02/Drought_to_overflow (source)
    Successfully uploaded on a re-run File:Drought to overflow ESA373404.gif
  2. http://www.esa.int/spaceinimages/Images/2017/01/Larsen_crack
    Successfully uploaded on a re-run File:Larsen crack ESA372459.gif
  3. http://www.esa.int/spaceinimages/Images/2016/09/Fire-scarred_Madeira2
    Successfully uploaded on re-run File:Fire-scarred Madeira ESA365160.gif
  4. http://www.esa.int/spaceinimages/Images/2016/08/Third_Bosphorus_Bridge_progress
    Successfully uploaded on re-run, see thumbnail on right.

The standard warning is "WARNING: API error stashfailed: This file did not pass file verification." The failure may be the rendering limits on animated GIFs, however repeating upload attempts seems to be bypassing the problem, so this may have been down to transient WMF operational issues.

Known errors

[edit]

Duplicates

[edit]

Some digitally identical duplicates have been unavoidably created. These may occur when:

An SHA1 check using the Commons API can show these duplicates, but creating the SHA1 value means downloading a local copy of the image to create the checksum, which considering the size of some images would be too 'expensive' in bandwidth and time. The ESA catalogue does not separately publish an SHA1 value.

The API does flag uploads as duplicates during the upload-from-url process, however these errors are being suppressed in order to allow for the "exists-normalized" error to be skipped; i.e. to allow both jpeg and TIFF to be uploaded under the same base filename. At the time of writing the potential solution of using Pywikibot's error trapping feature using an array in site.upload does not work, instead always causing fatal HttpConnection timeout failures.

Text missing

[edit]

These can be discovered in https://petscan.wmflabs.org/?psid=1047668. As of 24 May 2017, only one catalogue entry was found with no descriptive text, by 25th this was 26 images. These can be manually fixed by adding a description.

TIFF MIME and tiffinfo errors

[edit]

Several TIFFs have been rejected by the Commons API as having verification-error: Files of the MIME type "text/html" are not allowed to be uploaded others fail with verification-error: The uploaded file contains errors: tiffinfo command failed. The following are examples, the list may not be complete as it is ad-hoc cut & paste from a terminal:

  1. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2009/12/phobos_up_close/9558587-4-eng-GB/Phobos_up_close.tif
  2. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2009/12/raw_and_processed_images/9960948-2-eng-GB/Raw_and_processed_images.tif
  3. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2012/03/indonesian_islands/9989829-2-eng-GB/Indonesian_islands.tif
  4. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2009/04/vietnam_s_mekong_delta/9878555-3-eng-GB/Vietnam_s_Mekong_Delta.tif
  5. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2008/04/gobi_desert/9914133-2-eng-GB/Gobi_Desert.tiff
  6. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/10/california_s_raging_wildfires/10031923-2-eng-GB/California_s_raging_wildfires.tif
  7. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/09/a_2007_ice-free_portion_of_the_northwest_passage/9782601-3-eng-GB/A_2007_ice-free_portion_of_the_Northwest_Passage.tif
  8. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/09/2007_envisat_mosaic_of_arctic_ocean/9661511-3-eng-GB/2007_Envisat_mosaic_of_Arctic_Ocean.tif
  9. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/08/dean_leaving_martinique/9585654-3-eng-GB/Dean_leaving_Martinique.tif
  10. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/08/dean_off_coast_of_haiti/9610992-3-eng-GB/Dean_off_coast_of_Haiti.tif
  11. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/08/dean_approaching_yucatan_peninsula/9706869-3-eng-GB/Dean_approaching_Yucatan_Peninsula.tif
  12. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2007/02/envisat_captures_cyclone_gamede_on_23_february_2007/9717062-3-eng-GB/Envisat_captures_Cyclone_Gamede_on_23_February_2007.tif
  13. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2006/10/envisat_sees_madagascar/10084739-2-eng-GB/Envisat_sees_Madagascar.tiff
  14. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2006/10/residual_water_ice_in_vastitas_borealis_crater/9810169-3-eng-GB/Residual_water_ice_in_Vastitas_Borealis_Crater.tif
  15. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2006/09/svalbard_area_as_seen_by_envisat/10073241-2-eng-GB/Svalbard_area_as_seen_by_Envisat.tif
  16. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2006/04/beijing_blanketed_in_dust/10304870-2-eng-GB/Beijing_blanketed_in_dust.tif
  17. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2006/03/libya_montes_valley_region_on_mars/9603521-3-eng-GB/Libya_Montes_valley_region_on_Mars.tif
  18. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2005/08/envisat_optical_view_of_china_haze/9265069-5-eng-GB/Envisat_optical_view_of_China_haze.tif
  19. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2005/04/vancouver_and_seattle_seen_by_envisat/10073149-2-eng-GB/Vancouver_and_Seattle_seen_by_Envisat.tif
  20. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2005/04/the_nile_delta_and_the_sinai_peninsula/9245510-5-eng-GB/The_Nile_Delta_and_the_Sinai_Peninsula.tif
  21. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/11/mount_everest_as_seen_by_meris/10159981-2-eng-GB/Mount_Everest_as_seen_by_MERIS.tif
  22. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/10/buenos_aires_-_meris_-_17april_2004/9275944-5-eng-GB/Buenos_Aires_-_MERIS_-_17April_2004.tif
  23. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/08/close-up_of_the_southern_part_of_valles_marineris/9667275-3-eng-GB/Close-up_of_the_southern_part_of_Valles_Marineris.tiff
  24. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/08/eos_chasma_-_perspective_view/9565932-4-eng-GB/Eos_Chasma_-_perspective_view.tiff
  25. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/09/west_africa_asar_11_november_2003/9671441-3-eng-GB/West_Africa_ASAR_11_November_2003.tif
  26. http://www.esa.int/var/esa/storage/images/esa_multimedia/images/2004/09/guinea_conakry_asar_02_august_2003/9524728-4-eng-GB/Guinea_Conakry_ASAR_02_August_2003.tif

Testing by downloading a sample locally and attempting to open in image software, appears to demonstrate file corruption, and that the problem is likely to be at source rather than something created during the upload process. Failures from tiffinfo may be a question of some TIFFs using more obscure compression or odd use of the format as a wrapper. There are previous WMF Phabricator tickets relating to tiffinfo (search), and these may be worth reexamining, potentially with a view to upgrading tiffinfo if that is the root cause of the upload failures or requesting changes to the error descriptions if TIFF transcoding is needed to avoid, for example, unacceptable file formats embedded within the TIFF wrapper.

Wrongly identified images

[edit]

Some images may have been incorrectly identified, or the wrong image appears to have been added to the ESA database record.

Upload runs

[edit]

22:54, 5 May 2017 (UTC) Dry run

After adapting to PNG and GIF formats, only copyright failures have to be skipped.

07:32, 6 May 2017 (UTC) Full run populating Category:Copernicus Sentinel Satellite Imagery

The domains are not whitelisted, so the upload run relies on client-side uploads. This means it's slow!

15:16, 6 May 2017 (UTC)

Upload run adjusted so a parallel run handled jpg and png versions. All GIFs appear to be getting rejected due to verification errors. The tiff images may take a long time, it may not even complete before the domains are white-listed for url uploads.
TIFFs much over 130 MB appear to fail using client-side uploading. This may be due to time-out or other connection problems, as the uploads take over 20 minutes before failing. This should not be an issue when the upload sites are white-listed as uploads will be a magnitude faster with server-side uploads.

09:30, 8 May 2017 (UTC)

A full re-run does not appear to be uploading missing files. These are the larger TIFFs, which fail with errors like APIError: stashedfilenotfound and APIMWException: internal_api_error_UploadChunkFileException. The errors are probably due to the low-ish bandwidth upload not being up to sustaining uploads of 300 MB to 600 MB files and are likely to be resolved by use of server side uploads. UK domestic broadband deliberately throttles upload speeds, especially since UK Gov getting interested in forcing all ISPs to do more about anti-piracy.

08:36, 24 May 2017 (UTC)

With the ESA sites now white-listed, the code was re-run. This first uploaded the remaining 230MB+ size files previously rejected, then using a wider search for all CC-BY-SA-IGO licensed images. These less specific images are added to ESA images (review needed) for manual checks and categorization.
A separate 'housekeeping' routine checks through the category for jpegs and TIFFs with the same filename and cross-links them by adding thumbnails to a gallery in the information table.
During this upload the TIFF at http://www.esa.int/spaceinimages/Images/2012/03/Indonesian_islands was rejected by the API as a text/html MIME type. Later instances occurred, see Known Errors above.
Jpegs are being downloaded locally, then uploaded to Commons. This leaves them vulnerable to connection time-outs, and urlretrieve had to be stuck in an error trap after this meant restarting twice. A second attempt on the problem files appears to invariably succeed and there is no special pattern, such as size, that may be a cause apart from internet connection issues. Example File:Features in the elongated crater ESA215938.jpg.