SFI researchers (or award holders) are required to complete an Annual Report on the progress of their award. Since 2014, the majority of SFI annual reports are managed through its online grants and awards management system, SESAME. Each SFI researcher or Centre has a “Researcher Profile” in SESAME which they update with their outputs continuously throughout the year. These outputs, in turn, can be used to populate the researcher’s or centre’s Annual Report.
The Annual Report template for the majority of the awards that SFI makes contains a section on “Strategic Impact” in which the award holder is asked to prioritise at least five relevant Impact declarations from a list of options provided (see below).
SFI are using this self-assessment type approach, since it goes in some way towards quantifying the types of impact arising from the awards it makes. This approach, however, may be subject to bias and so in addition to providing outputs through the Researcher Profile in SESAME in support of their chosen impact declarations, the award holders are required to provide narrative/details justifying the options they have selected.
As part of the annual reporting guidelines, all of SFI award holders are provided with a list of 11 Impact ‘declarations’ or statements. At least one statement must be selected but award holders are encouraged to rank up to 5 statements, starting with the number 1 (being the most relevant). They are then asked to provide more details justifying the statements selected. The Impact statements help SFI to quantify the types of impacts coming from the various SFI Programmes. Each statement can be aligned with at least one of the “Types of Impact” discussed here. The Impact type is listed in parenthesis after each statement.
1. The research conducted through my award has enabled me to leverage international funding through industry/collaborative research [Economic and Commercial, International]
2. The research conducted through my award has resulted in the start or expansion of a company which has resulted in the creation of high value jobs [Economic and Commercial]
3. The research conducted through my award has attracted developing and nurturing businesses [Economic and Commercial]
4. The research conducted through my award has attracted international scientists and talented people [Human Capacity; International Engagement]
5. The research conducted through my award has resulted in a new policy being implemented and/or an improvement to the delivery of a public service [Public Policy and Services]
6. The research conducted through my award has enhanced the quality of life and health of Irish citizens [Health & Wellbeing, Societal Impact]
7. The research conducted through my award has improved the environment and/or the sustainable relationship between society, industry and the environment [Environmental Impact]
8. The research conducted through my award has increased the knowledge, appreciation and understanding of science, engineering and technology amongst the general public. The research conducted through my award has developed the country’s international reputation [Societal Impact, International Engagement]
9. The research conducted through my award has resulted in the creation of employment through directly influencing and inspiring the future workforce and/or the production of a highly educated and relevant workforce in demand by industry and academia [Human Capacity, Economic and Commercial]
10. The research conducted through my award has impacted in other areas not reflected in the choices provided, for example by enhancing the creative output of Irish citizens [Environmental, Professional Services, Societal]
11. The research conducted through my award has not yet realised any significant Impact
MID-TERM PROGRAMME PROGRESS REVIEW
Many of SFI’s awards, in particular awards of scale and Investigator-led awards, are subject to a mid-term programme progress review. International experts in the relevant discipline, including those with expertise in relevant areas of industry, commercialisation and translation, are required to evaluate the progress being made against the original Impact Statement as submitted in the funded proposal. There is specific guidance provided to direct the reviewers to ‘score’ with reference to indicators of impact, as defined by SFI, and not simply to ‘rate’ the outputs on the award, some of which may have little relevance to impact. Furthermore, all Research Centre awards are asked to provide targets for specific outputs which are considered in direct support of delivering Impact. These are examined internally by SFI staff, and used to gauge progress and report on their progress to the SFI Executive and Board, and our parent department.
Programme evaluation is a critical process towards ensuring that the mechanisms for providing funding are aligned with the relevant strategy du jour, with particular focus on delivering scientific excellence and impact. Specifically, data collated for the annual stocktake of SFI Research Outputs (formerly referred to as the SFI Census) and collected in the annual reporting process are used to support detailed programmatic evaluations. This ensures that the success of a particular programme in delivering against its original objectives and the collective metrics gathered in support of Key Performance Indicators (KPIs) as defined in SFI’s Agenda 2020, can be determined. Furthermore, the associated narrative in support of the metrics gathered (typically found in the annual report) can be used to provide case studies and additional context to this data. This is particularly important for scenarios where rationale in support of impacts arising from specific programmatic activities is required to be articulated to our parent department as justification for the investment made.