This document is a supplement to the DOI Landing Page of the EUI Data Release 6.0 2023-01 available at DOI: https://doi.org/10.24414/z818-4163 This document contains additional technical details.
Content
“EUI Data Release 6.0 2023-01”, released on 2023-02-08 contains the Solar Orbiter/EUI L1, L2 and L3 data for the period 2020-03-03 up till 2023-11-30 (originally 2023-01-10). An image catalogue, in the form of an SQLite database of metadata for all L1 and L2 FITS files, is provided, with each image file corresponding a database row and each FITS keyword corresponding to a database column. The column ‘level’ can be used to distinguish L1 from L2 files. This release originally contained 293342 L1, 274024 L2, and 257117 L3 files. This release will continue to be appended with later incoming data, until the next full release (Release 7) is issued. On 2023-05-09, this release has been appended with data up till 2023-03-24. On 2023-08-07, this release has been appended with data up till 2023-06-20. On 2023-12-13, this release has been appended with data up till 2023-11-30.
Level 1 data
Level 1 (L1) data implies that the metadata in the FITS keywords is complete but that the images remain uncalibrated. Besides decompression, only limited, reversible operations have been applied to the images (mirroring & flipping, and 90-degree rotations) such that solar north is roughly on top and solar east roughly on the left. All the images received on the ground are distributed in the L1 dataset. This includes images with the telescope doors closed, either “darks” or “LEDs” (calibration lamp images). The first of these were received on 2020 March 3.
Level 2 data
Level 2 (L2) data are produced by the euiprep python routine. This routine takes the L1 data and rectifies instrument deviations for which corrections are available. It also updates the WCS FITS keywords to correct for spacecraft pointing instabilities. The output of these calibration steps is distributed as L2 data. The euiprep routine is in continuous evolution. The L2 data are thus a snapshot of the best available corrections at the time of the L2 data production. The git version number of the euiprep S/W used to generate these L2 files is copied in the FITS header HISTORY keyword. For this release, the git SHA (first 8 characters) is (see also FITS header VERS_CAL keyword): ‘a57d9e42’ for files with DATE-BEG before 2023-01-10 15:00:00 and ‘f434d8b2’ for files with a DATE-BEG after this time.
The euiprep routine is only applied to lossless compressed or uncompressed Level-1 solar images, starting with the ‘first light’ images on 2020 May 12. Technical images such as e.g. darks or LEDs are not processed into L2 data. The selection is done with the IMGTYPE keyword which is set to IMGTYPE=“solar image” if * both the heat shield doors and instrument doors are open, AND * no LED is on AND * if the telescope has a filter wheel, that this filter is not in a blocking position.
For HRILYA, extra conditions are added to make sure the Screen and MCP voltages were properly configured for solar imaging. For HRILYA images with exposure time less or equal than 3s (XPOSURE LE 3.0) we require
LYACSCR > 3450 AND ((GAINCOMB="low-only" AND LYACMCP> 580) OR (GAINCOMB="high-only" AND LYACMCP> 450))
For HRILYA images with exposure time larger than 3s (XPOSURE GT 3.0) we require
LYACSCR > 3450 AND ((GAINCOMB="low-only" AND LYACMCP> 480) OR (GAINCOMB="high-only" AND LYACMCP> 350))
The L2 file version number is incremented every time the L2 file is updated and may thus differ from the L1 file version number.
Level 3 data
Level 3 (L3) data corresponds to a further processing of L2 data for visualisation purposes. The L3 image files are intended for visualisation through JHelioviewer. L2 images without the keyword WAVELNTH (e.g., because a wrong filter position was selected) are excluded from processing. The L3 file version number is identical to the L2 file version number.
Data policy, citation & acknowledgements
The “EUI Data Release 6.0 2023-01” is public and can be freely downloaded from the EUI website and from the Solar Orbiter Archive. The requested citation and acknowledgements are stipulated in the EUI Data Release 6.0 2023-01 landing page. Specifically for citing the EUI instrument paper “Rochus et al, AA, 2020” and the above DOI, the following latex/bibtex bibitems are suggested:
@ARTICLE{2020A&A...642A...8R,
author = {{Rochus}, P. and {Auch{\`e}re}, F. and {Berghmans}, D. and {Harra}, L. and {Schmutz}, W. and {Sch{\"u}hle}, U. and {Addison}, P. and {Appourchaux}, T. and {Aznar Cuadrado}, R. and {Baker}, D. and {Barbay}, J. and {Bates}, D. and {BenMoussa}, A. and {Bergmann}, M. and {Beurthe}, C. and {Borgo}, B. and {Bonte}, K. and {Bouzit}, M. and {Bradley}, L. and {B{\"u}chel}, V. and {Buchlin}, E. and {B{\"u}chner}, J. and {Cab{\'e}}, F. and {Cadiergues}, L. and {Chaigneau}, M. and {Chares}, B. and {Choque Cortez}, C. and {Coker}, P. and {Condamin}, M. and {Coumar}, S. and {Curdt}, W. and {Cutler}, J. and {Davies}, D. and {Davison}, G. and {Defise}, J. -M. and {Del Zanna}, G. and {Delmotte}, F. and {Delouille}, V. and {Dolla}, L. and {Dumesnil}, C. and {D{\"u}rig}, F. and {Enge}, R. and {Fran{\c{c}}ois}, S. and {Fourmond}, J. -J. and {Gillis}, J. -M. and {Giordanengo}, B. and {Gissot}, S. and {Green}, L.~M. and {Guerreiro}, N. and {Guilbaud}, A. and {Gyo}, M. and {Haberreiter}, M. and {Hafiz}, A. and {Hailey}, M. and {Halain}, J. -P. and {Hansotte}, J. and {Hecquet}, C. and {Heerlein}, K. and {Hellin}, M. -L. and {Hemsley}, S. and {Hermans}, A. and {Hervier}, V. and {Hochedez}, J. -F. and {Houbrechts}, Y. and {Ihsan}, K. and {Jacques}, L. and {J{\'e}r{\^o}me}, A. and {Jones}, J. and {Kahle}, M. and {Kennedy}, T. and {Klaproth}, M. and {Kolleck}, M. and {Koller}, S. and {Kotsialos}, E. and {Kraaikamp}, E. and {Langer}, P. and {Lawrenson}, A. and {Le Clech'}, J. -C. and {Lenaerts}, C. and {Liebecq}, S. and {Linder}, D. and {Long}, D.~M. and {Mampaey}, B. and {Markiewicz-Innes}, D. and {Marquet}, B. and {Marsch}, E. and {Matthews}, S. and {Mazy}, E. and {Mazzoli}, A. and {Meining}, S. and {Meltchakov}, E. and {Mercier}, R. and {Meyer}, S. and {Monecke}, M. and {Monfort}, F. and {Morinaud}, G. and {Moron}, F. and {Mountney}, L. and {M{\"u}ller}, R. and {Nicula}, B. and {Parenti}, S. and {Peter}, H. and {Pfiffner}, D. and {Philippon}, A. and {Phillips}, I. and {Plesseria}, J. -Y. and {Pylyser}, E. and {Rabecki}, F. and {Ravet-Krill}, M. -F. and {Rebellato}, J. and {Renotte}, E. and {Rodriguez}, L. and {Roose}, S. and {Rosin}, J. and {Rossi}, L. and {Roth}, P. and {Rouesnel}, F. and {Roulliay}, M. and {Rousseau}, A. and {Ruane}, K. and {Scanlan}, J. and {Schlatter}, P. and {Seaton}, D.~B. and {Silliman}, K. and {Smit}, S. and {Smith}, P.~J. and {Solanki}, S.~K. and {Spescha}, M. and {Spencer}, A. and {Stegen}, K. and {Stockman}, Y. and {Szwec}, N. and {Tamiatto}, C. and {Tandy}, J. and {Teriaca}, L. and {Theobald}, C. and {Tychon}, I. and {van Driel-Gesztelyi}, L. and {Verbeeck}, C. and {Vial}, J. -C. and {Werner}, S. and {West}, M.~J. and {Westwood}, D. and {Wiegelmann}, T. and {Willis}, G. and {Winter}, B. and {Zerr}, A. and {Zhang}, X. and {Zhukov}, A.~N.},
title = "{The Solar Orbiter EUI instrument: The Extreme Ultraviolet Imager}",
journal = {\aap},
keywords = {Sun: UV radiation, Sun: transition region, Sun: corona, space vehicles: instruments, telescopes, instrumentation: high angular resolution},
year = 2020,
month = oct,
volume = {642},
eid = {A8},
pages = {A8},
doi = {10.1051/0004-6361/201936663},
adsurl = {https://ui.adsabs.harvard.edu/abs/2020A&A...642A...8R},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
@misc{euidatarelease6,
author = {{Kraaikamp}, E. and {Gissot}, S. and {Stegen}, K. and {Mampaey}, B. and {Verbeeck}, F. and {Auch{\`e}re}, F. and {Berghmans}, D. },
title = {SolO/EUI Data Release 6.0 2023-01},
howpublished = {https://doi.org/10.24414/z818-4163},
month = {January},
year = {2023},
note = {Published by Royal Observatory of Belgium (ROB)}
}
Data usage tips & tricks
General comments
The EUI FITS files (L1 and L2) are compliant with the FITS standard, the FITS World Coordinate System and the Solar Orbiter Metadata definition. EUI FITS files have been tested on popular FITS reading applications such as FITSview, SAOImageDS9 and jHelioviewer. Note that since Release 5.0, the majority of L1 and L2 EUI images are are internally compressed (tile-compression, RICE).
Like all spacecraft, Solar Orbiter experiences a certain amount of jitter while taking image sequences. In nearly all cases, this jitter has negligible effect on individual image sharpness but subsequent images might appear to jump from one to the next. This pointing instability is NOT corrected in the images but is recorded in the FITS WCS CRVAL1 and CRVAL2 keywords, in as far as that information is available in the spacecraft metadata (the as-flown SPICE kernels). Other useful keywords are the EUXCEN and EUYCEN that contain the center of the Sun in pixel units. Only for FSI L2 files, we have updated this information based on a solar limb fitting technique (see eui_prep). FITS viewing applications like jHelioviewer read the CRVAL1,2 and correct the image alignment on the fly. Data users, writing their own software, who want to improve the pointing stability in image sequences, should read the CRVAL1,2 deviations and correct the images accordingly. Note that in some image sequences, the solar rotation is tracked and thus the CRVAL1,2 and EUXCEN and EUYCEN are continuously changing. In other cases, the keyword information might not be accurate enough. Cross-correlation techniques between the images are then recommended.
Basic IDL reading script
SSWIDL users should use “mreadfits_tilecomp” with the option “/use_shared_lib”. eui_readfits.pro is an example routine that encapsulates this.
An image catalogue is provided. It takes the form of an SQLite database (metadata.db) containing the metadata of all L1 and L2 FITS files, with each image file corresponding a database row and each FITS keyword corresponding to a database column. An extra column ‘level’ is included to distinguish L1 from L2 files. This image catalogue can be downloaded to a local computer for SQL queries across all images. SQLITE is preinstalled on Apple computers and is available for all platforms. Here is a short usage demo on a Mac:
at the terminal prompt, go to the directory in which the metadata.db is stored
enter “sqlite 3 metadata.db” and a prompt sqlite> will appear
at the prompt, type “.schema” to see the schema of the database
type “SELECT * FROM fits_file LIMIT 2;” to see a couple of rows
type “.header on” and retype the previous command to see also the header names
type “SELECT filename FROM fits_file WHERE telescop = ‘SOLO/EUI/FSI’ and xposure < 0.5 and ledstate= ‘all off’ and doorpos = 34 LIMIT 100;” to see all the FSI images with an exposure them less than 0.5s
type “.separator COL”," and retype the previous command to have a comma-separateed version version
type “.output myfile.txt” and retype your query to have the output in an external file
.quit to exit
HRIEUV temporal noise
The temporal noise on repeated HRIEUV pixel values is typically dominated by the sensor read noise and the photon shot noise according to the formula:
s2 = r2 + I * t * a/n
with:
s: uncertainty on measured value
r: readout noise, 2 DN
I: the measured intensity (DN/s)
t: exposure time (s)
a: photon to DN conversion factor (6.85 DN/photon)
n: sample size (number of pixels over which the intensity is averaged)
See Gissot et al 2023 (in preparation) for more detail.
List of changes
List of changes as compared to EUI Data Release 5.0
Data from 2020-03-03 till 2023-01-10 has been included.
Updated the descriptor part of the filename, it now ends with ‘-image-dark’, ‘-image-led’, ‘image-occulter’, ‘image-short’ for darks, LEDs, occulter images, and short exposure solar images respectively. All other descriptors still end with ‘-image’. The
The IMGTYPE keyword value ‘solar image, short’ has been introduced for short exposure solar images. These short exposure have an exposure time ~50 times shorter than regular exposures in order to minimize over-exposing solar flares, and are typically taken a few seconds before a regular exposure. The short exposures images are taken systematically for FSI since July 2022, and for HRI EUV there are a few images sequences containing short and regular exposures from 2022-10-17 onward.
The FILTER keyword for HRI EUV images with the filter wheel in the open position is now ‘open’ instead of ‘none’, and similarly, the HRI EUV descriptor ‘eui-hrieuvnon-..’ has been replaced with ‘eui-hrieuvopn-..’.
L1 calibration data from before first light (2020-05-12) was missing in release 5 and has been added in release 6.
Updated values of TARGET keyword, and removed this optional keyword when no target could be determined.
The values of the SOOPNAME and SOOPTYPE keywords are now set to ‘none’ when the observation does not belong to a SOOP.
L2+ Introduced DCRVAL1 and DCRVAL2 keywords for limb-fitted FSI images that show the applied delta offset to the original CRVAL values coming from L1 (from as-flown kernels).
L2+ Introduced DCRVAL1 and DCRVAL2 keywords for short exposure FSI images by linearly interpolating between nearby DCRVAL values from FSI limb-fitting results for regular FSI synoptic exposures using the same filter. These DCRVAL values show the applied delta offset to the original CRVAL values coming from L1.
L2+ Introduced DCRVAL1 and DCRVAL2 keywords for HRI images by linearly interpolating between nearby DCRVAL values from FSI limb-fitting results for regular FSI synoptic exposures using the 174 filter only. These DCRVAL values show the applied delta offset to the original CRVAL values coming from L1.
L2+ HRI EUV and FSI combined gain images are scaled to high gain DN numbers. The pixel values are divided by the commanded global high gain value (* 256/GAINHG).
L2+ Improved column shift correction for HRI LYA to better deal with low-signal solar images.
L2+ Fixed NaN values in RICE compressed L2 images.
L2+ For RICE compressed L2 images, the image data is scaled as 16-bit integer data using the BZERO and BSCALE keywords.
L2+ HRI EUV and FSI low gain images are excluded. All ‘both’ gain images are excluded.
L2+ Improved initial Sun center estimate for limb-fitting.
L3 Improved displaying of FSI occulter images and added HV_INNER keyword to allow JHelioviewer to cut out the occulted part of the image.
List of changes as compared to EUI Data Release 4.0
Data from 2020-03-03 till 2022-04-13 has been included. This release will continue to be appended with later incoming data, until the next full release (Release 6) is issued.
The pointing information - the difference between the satellite and instrument boresights - has been updated in the in the EUI instrument kernels (SPICE kernels), based on measurements performed on February 12, 2022 data. This also includes a 0.1 degree difference in roll-angle for the HRI EUV telescope as compared to the previous releases (from -0.35 to -0.25 degree). The updated pointing information is applied to all files (both observation times both before and after February 2022). The only FITS files for which this pointing information is further updated (from L1 to L2 files), are FSI 174 and 304 L2 files for which the limb-fitting method converged.
All L1 and L2 FITS files are internally compressed (tile-compression, RICE). For L1 files this procedure is lossless. For the L2 files the compression introduces a (small) quantization error, as the floating point data have been scaled between the minimum and maximum value in the data with 16-bit resolution. SSWIDL users should use “mreadfits_tilecomp” with the option “/use_shared_lib”.
Limb-fitting performance was significantly improved for all FSI images, resulting in improved pointing accuracy in FSI 174 and 304 L2 files.
The calibration software (euiprep) has been updated at various locations without altering the functionality. Minor changes in the L2 files are possible.
NaN values in L2 files no longer show up as bright lines in L3 files, but as dark lines instead.
For GAINCOMB = ‘both’ images, the DOORPOS value that travelled down with the image has been corrected using housekeeping data. This fixes incorrect DOORINT and IMGTYPE for the affected images.
For GAINCOMB = ‘both’ images, when making L3 files only the LG channel is used (the HG channel is ignored).
The dark frame and flat field correction is now also applied for recoded images.
List of changes as compared to EUI Data Release 3.0
Data from 2020-03-03 till 2021-09-15 has been included
As compared to release 3, also additional data from the period 2021-02-20 till 2021-04-28 has been included. This was backup data that was received on ground after release 3 was created.
Limb-fitting performance was improved for FSI 304 images resulting in improved pointing accuracy in FSI 304 L2 files.
A bug occurred in Release 3 (flatfield not matching criteria) which implied that some HRI-EUV images on 2020-05-30 were erroneously removed. These files have been restored in Release 4.
The calibration software (euiprep) has been updated at various locations without altering the functionality. Minor changes in the L2 files are possible.
List of changes as compared to EUI Data Release 2.0
Data from 2020-03-03 till 2021-04-28 has been included
FITS keyword IMGTYPE = ‘occulted image’ for FSI images with the internal door in occulting position (and external door open).
For combined gain images, the FITS keyword COMP_RAT now gives the compression ratio with respect to a 15-bit raw image of the same dimensions (was 12-bit for all images).
Improved matching of FITS keywords OBS_MODE, OBS_TYPE, OBS_ID to observations defined in SOOP kitchen.
The FITS keywords SOLAR_P0, SOLAR_PE contained incorrect values in previous releases, this has been fixed.
To improve the accuracy of the FITS keywords GAINHG, GAINLG, OFFSETHG, OFFSETLG, their values are now extracted from parameter table dump reports instead of from housekeeping data.
The last 32 columns of the FSI detector have been systematically removed from L2 and L3 images as these columns are not functioning correctly.
The calibration software (euiprep) has been updated at various locations without altering the functionality. Minor changes in the L2 files are possible.
List of changes as compared to EUI Data Release 1
Data from 2020-11-01 till 2021-02-18 has been included
L3 data is included
Added FILCPOS keyword. This is the commanded filter wheel position (FILTPOS is the actual filter wheel position determined from service 5 messages)
Fixed unknown internal door status (DOORPOS=255), DOORINT is now determined from service 5 messages.
Fixed OBS_MODE and related keywords. Now correctly taking into account the modules defined in SOOP kitchen: EUV, LYA, FSI, COMMON_MODULE (so it excludes matches with flushes).
Fixed OBS_MODE and related keywords. If images are outside of observation period as defined in SOOP kitchen, then closest observation period in time is found (as sometimes we are imaging slightly before an observation period, while those images belong to that same campaign).
Fixed values of HEEX_OBS,HEEY_OBS,HEEZ_OBS, and GSEX_OBS,GSEY_OBS,GSEZ_OBS after mistake was corrected in SPICE kernels.
Fixed FSI plate-scale in L1 (CDELT1 was incorrectly set to 4.43614 in previous release, it is now 4.4353411)
The calibration software (euiprep) has been continuously updated at various locations without altering the functionality. Minor changes in the L2 files are possible. For this release, the git SHA (first 8 characters) is 64ce9853 (see also FITS header VERS_CAL keyword).
Updated FILTER keyword for FSI files; the values Magnesium_304_n26, Zirconium_174_n25, Magnesium_304_n4, and Zirconium_174_n13 are now used to identify the four available filters. (blocking positions are still identified by the value blocked, and any other position has the keyword value undetermined.
Known remaining issues
Spacecraft pointing instabilities (thermoelastic, others) might affect the pointing metadata. It is recommended to use the pointing metadata only as a rough approximation. For FSI L2 images, the pointing metadata was updated using a solar limb-fitting procedure, and should typically be more accurate than 0.5 pixel for both FSI 174 and FSI 304 images, unless the solar limb is not clearly visible in the image (either the limb is saturated or under-exposed).
FSI images suffer from a dark vertical banding off-limb. The dark vertical bands appear above and below the brightest areas on disk such as the limb and active regions. These vertical bands are only visible when strongly enhancing the off-limb brightness.
FSI gain map correction is applied incorrectly, resulting in a small error (< 1 DN) that can be seen when strongly enhancing the off-limb brightness.
HRIEUV combined gain images only have a column correction applied, not a gain correction. Furthermore, there is also a small LG and HG mismatch in the combined gain images.
HRILYA suffers from a ‘remnant image effect’ which shows up as a haze of the brightness distribution of the scenery in the past 30min. This remnant image effect is most clearly seen in the first of a series of limb images.
HRILYA suffers from a degraded performance (resolution and contrast) when close to perihelion.
CMOS sensor settings vary a lot throughout the Commissioning Phase and might affect image quality.
The calibration of the detector temperature sensor is to be improved. The temperature keywords (TEMPINT, TEMP1DET, TEMP2DET) can deviate.
For HRILYA, LYCMCP and LYCSCR FITS keyword values are inaccurate for low values.
HRILYA data taken on 2021 March 22 between 10:00:00 and 12:00:00 are incorrectly calibrated in L2/L3 (there is a sub-pixel misalignment between the lights and the applied darks/flats)
The values of the CHECKSUM and DATASUM keywords in the RICE compressed L1 FITS files generated before January 11, 2023 are incorrect because of a bug in Astropy. The bug has been fixed, and future releases, and L1 files generated on and after January 11, 2023 will have a correct CHECKSUM and DATASUM value.
Use DATAMEAN with caution, as it can be affected by detector artifacts (and/or artifacts introduced by the WICOM compression system). For example, FSI images that were binned onboard (NBIN1>1) can occasionally have columns at the edge of the image with (much) brighter pixel values.
L2 short exposures for HRI EUV are not included in Data Release 6. These will be included in Data Release 7.