Verification Analysis of the SIDC Forecast

Main page
Glossary
F10.7 Solar Flux Forecast
F10.7 Solar Flux Error Analysis
K-index on Full Scale (0-9)
K-index: Forecast of Geomagnetic Storm
Forecast of Flares
Forecast of M/X Flares

Glossary

General

SIDC forecast: forecast sent out daily by the Solar Influence Data Center (SIDC) in it's role of Regional Warning Center (RWC). The forecast is performed manually.
 
Short term climatology model: simple numerical model that takes the average value across the past 30 days.
 
Climatology model: average value across the full available period.
 
Persistence model: model that uses the observational value of yesterday as forecast of today's value.
 
Recurrence (with a time shift of 14 or 27 days): numerical model that uses the observational value of 14 days or 27 days ago. This assumes the observed value of one (or half) Carrington rotation ago reoccurs.
 
Corrected recurrence (with a time shift of 14 or 27 days): numerical model that applies the daily increment of one (or half) rotation ago to yesterday's observation.
 
Linear fit: linear regression model using observations of the past 4 days to forecast the values of today and the next days.
 
Absolute error: the absolute error is defined as the error in a non-relative sense.
 
Absolute value of the error: unsigned error. The absolute value of a negative error is positive. The absolute value of a positive error is the error itself.
 
Median: value that separates a sample in two halves. The median is defined by ranking the observations from the lowest to highest value. The middle value corresponds to the median. In case of an even number of observations, the median typically is defined as the mean of the two values in the middle. The median is also called the 50%-quantile.
 
Quantile: value that separates a sample in two parts, one part with lower values and the other part with higher values. A specific example are the quartiles.
 
Quartile: a quartile is one of three values that separates a ranked dataset in four equal parts. The first quartile (or 25%-quantile) separates the lowest 25% values from the higher values. The second quartile is the median, corresponding to the middle of the set. The third quartile (or 75%-quantile) is the value that separates the lower 75% values from the 25% highest values.
 

Categorical forecasts:

Contingency table:
Observed
YesNo
ForecastYes ab
Nocd
 
Event: Observation=Yes. Number of events: a+c
 
Hit: correctly forecasted event. Number of hits: a
 
Miss: event that was not forecasted. Number of misses: c
 
False alarm: forecast of an event, while no event was observed. Number of false alarms: b
 
Correct rejection: forecast of a non-event, while indeed no event was observed. Number of correct rejections: d
 
POD=Probability of Detection (or hit rate): ratio of the number of hits, divided by the number of events; a/(a+c)
 
PC=Proportion Correctness: ratio of total number of correct forecasts divided by the total number of forecasts; (a+d)/(a+b+c+d)
 
FAR=False Alarm Ratio: Is a verification measure equal to the ratio of the number of false alarms by the total number of event forecasts; b/(a+b)
 
SR=Success ratio: The success ratio is the complement of the false alarm ratio (FAR). Is the number of hits divided by the total number of event forecasts; a/(a+b)
 
HSS=Heidke Skill Score: skill score taking into account the number of correct random forecasts. HSS= (PC-E)/(1-E), with E=proportion of correct random forecasts, assuming forecasts and observations are independent and assuming the same proportion of forecasts of occurrence to non-occurrence. HSS has a range from -1 to 1, with 1 a perfect forecast, 0 as good as random and -1 the worst forecast
 
TSS=True Skill Statistic: verification measure of categorical forecast similar to the Heidke Skill Score (HSS). The TSS has nice characteristics for rare event forecasts. For example random and constant forecasts get a TSS of 0. For extremely rare events, TSS tends to converge to POD. TSS=(ad-bc)/((a+c)(b+d)). The TSS is also called the Peirce's skill score or the Hanssen and Kuipers' score.
 
Bias: the degree of correspondence between the mean forecast and the mean observation; as such it indicates whether observations are over- or underestimated. For categorical forecasts, bias is defined as the ratio of the number of forecasts of occurrence to the number of actual occurrences. B=(a+b)/(a+c)
 
CSI=Critical Success Index: CSI is a sample estimate of the conditional probability of a hit, given that the event of interest was either forecast, observed or both. It is regarded as a performance measure for forecasts of rare events, since the calculation is irrespective of the number of correct rejections. The CSI is an unreliable score, cause constant and random forecasts may result in different CSI values, depending on the proportion of forecast of occurrence to non-occurrence in the sample. CSI=a/(a+b+c)
 
GSS=Gilbert Skill Score: Is an alternative to CSI that allows for the number of hits obtained purely by chance. The hits due to chance expected for forecasts independent of observations is given by: ch=(a+b)*(a+c)/n and GSS=(a-ch)/(a+b+c-ch).
 

Continuous forecasts:

MSE=mean squared error
 
RMSE=root mean squared error
 
Skill score: performance measure with respect to a reference model. The skill score is defined as 1- MSE/MSEref with MSEref being the MSE of the reference model. For forecasts of the 10.7 cm flux, the persistence model is currently used as reference.
 



The research leading to these results has received funding from the European Commission's Seventh Framework Programme (FP7/2007-2013) under the grant agreement nr. 263506 [AFFECTS].
Our policy on the use of our websites can be found here.
Further verification analysis is in the pipeline.