Please cite this report as: d’Haenens, L., Vissenberg, J., Martinez, D., Martinez, G., Garmendia, M., Larrañaga, N., Helsper, E., Edisherashvili, N., Maksniemi, E., Puusepp, M., Kasemets, M., Tomczyk, L., Kielar, I., Krzeczkowska, M., Irani, F., Hietajärvi, L., & Salmela-Aro, K. (2024). Report on the Development of the Evaluation Strategies Tailored to Media Literacy and Digital Skills Intervention Programmes. KU Leuven, Leuven: REMEDIS. Disclaimer REMEDIS is supported by Research Foundation – Flanders (FWO), Belgium; UK Research and Innovation (UKRI), United Kingdom; Estonian Research Council (ETAg), Estonia; Agencia Estatal de Investigación (AEI), Spain; Academy of Finland (AKA), Finland; and Narodowe Centrum Nauki (NCN), Poland under CHANSE ERA-NET Co-fund programme, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme, under Grant Agreement no 101004509. 1 Report on the development of the evaluation strategies tailored to media literacy and digital skills intervention programmes Work package 2 – Deliverable 2.2 2 3 Table of contents Executive summary .................................................................................................................................................. 5 1 2 Introduction ....................................................................................................................................................... 6 1.1 The REMEDIS project .......................................................................................................................... 6 1.2 Introduction to this report ................................................................................................................ 6 Development of the data collection strategy ..................................................................................... 10 2.1 Collaboration with the partner organisations in developing a data collection strategy 10 3 Overview of the intervention programmes: Objectives and stakeholder collaboration ......... 12 3.1 Belgium ........................................................................................................................................................... 12 3.2 Estonia ............................................................................................................................................................. 20 3.3 Finland ............................................................................................................................................................. 29 3.4 Poland ............................................................................................................................................................. 32 3.5 Spain ................................................................................................................................................................ 38 3.6 United Kingdom .......................................................................................................................................... 46 4 Questionnaire development ........................................................................................................................... 55 4.1 Description of the different modules (A-D) and how the questions were set up ............. 55 4.2 Validation of the instrument .................................................................................................................. 56 4.3 Data collection scenarios ......................................................................................................................... 56 5 Conclusion ............................................................................................................................................................. 59 Acknowledgements ............................................................................................................................................... 62 References ................................................................................................................................................................. 63 Appendix 1: Recommendations shared with non-academic partners/intervention providers based on the Systematic Literature review (ESTONIA). ........................................................................... 64 4 Executive summary This report offers an analysis of the co-development process for intervention programmes, utilising the REMEDIS Canvas framework. The framework aids in planning, refining, and adapting interventions to align with varying contextual factors across different countries. The summary captures the essence of the process, frameworks used, stakeholder engagement, intervention design, implementation, monitoring, evaluation, and feedback mechanisms. Initiating with a detailed examination of the problems and contextual landscape, each intervention identified key challenges, stakeholder dynamics, and socio-cultural influences. Using the Canvas framework, objectives and strategies were clearly delineated to tackle specific contextual issues. The process heavily emphasised stakeholder engagement to incorporate diverse viewpoints into the intervention design. Workshops and consultations facilitated by the Canvas structure ensured productive discussions and consensus-building. Drawing on insights from problem analysis and stakeholder feedback, interventions were custom-designed. They remained adaptable to feedback and shifts in context throughout their execution.nThe Canvas framework was pivotal in setting performance indicators and evaluation metrics, enabling effective monitoring. Established feedback loops ensured interventions were responsive to ongoing needs and adjustments. Feedback highlighted various organisational and national context issues such as resource constraints and cultural sensitivities, necessitating strategic stakeholder engagement and adaptive approaches to navigate these challenges. Challenges in data collection strategies were noted, particularly with constraints on session attendance and participant availability, influencing the comprehensiveness of data coverage. The focus was on qualitative insights to augment quantitative data, aiming for a holistic understanding of intervention impacts. The questionnaire developed assesses intervention effectiveness through four modules capturing a) socio-demographic data, b) digital skills, c) pre-intervention, and d) postintervention outcomes. Different scenarios outlined the application of these modules under varying conditions to accommodate resource availability and ensure continuity in data collection. The quantitative analysis (to be presented in future REMEDIS reports D3.1 and D3.2) will present descriptive and inferential statistical findings, while the qualitative analysis will adopt a thematic approach to interpret participant experiences and perceptions. Both approaches aim to provide a comprehensive understanding of the intervention impacts, discussing the findings transparently and acknowledging any biases. The REMEDIS Canvas has proven to be an effective framework for the co-development of tailored interventions, addressing diverse needs and contexts. Despite some limitations and challenges, the structured approach enabled meaningful insights and informed decisionmaking for future intervention planning and improvement. The detailed reports to follow will examine further the specifics of each evaluated programme, ensuring a thorough understanding of impacts and improvements across the board. 5 1 Introduction 1.1 The REMEDIS project The REMEDIS (Rethinking Media Literacy and Digital Skills) project is funded by the European Union’s CHANSE (Collaboration of Humanities and Social Sciences in Europe) programme. The consortium involves 7 academic partners from 6 countries, along with 14 non-academic cooperation partners. REMEDIS seeks to develop evidence-based approaches to develop and evaluate initiatives that foster media literacy and digital skills (ML&DS) to understand what the impacts of ML&DS interventions in different life domains are in terms of positive outcomes. REMEDIS adopts an innovative research strategy that first aims to identify and quantify the most salient driving factors for ML&DS from a lifelong perspective and to synthesise the existing evidence concerning the perceived effectiveness of current interventions fostering ML&DS. REMEDIS will pay special attention to target groups, including disadvantaged youths (NEETs or Not in Education, Employment, or Training), the unemployed, refugees, people with lower SES, carers of NEETs, and (future) teachers. To achieve its aim, the REMEDIS project has four research objectives. 1. To improve existing theoretical knowledge about the actual outcomes of interventions. 2. To improve and enhance existing ML&DS intervention strategies based on existing and emerging evidence. 3. To adopt advanced methods and to develop and validate instruments for evaluating intervention strategies. 4. To produce evidence-based policy recommendations and develop a user-friendly, customisable evaluation toolkit. This report contributes to achieving the first objective of REMEDIS by developing an evidence base synthesis using a systematic review of the drivers and outcomes of ML&DS interventions and of characteristics of potentially effective ML&DS intervention programmes that lead to positive outcomes. 1.2 Introduction to this report In this report, we outline our tailored approach for each intervention programme, aiming for cohesive implementation and integrated data analysis as detailed in report D3.2. Data collection for this intervention spanned from October 2023 to June 2024, involving participants representing diverse demographics. Our team implemented targeted outreach strategies, resulting in commendable engagement levels within the selected population. We used a flexible array of methods customised to suit the nuances of each context. Online forms facilitated seamless interaction and accessibility for participants in certain regions, while face-to-face interviews were conducted where digital access was limited. Modules were carefully chosen based on contextual relevance and feasibility, ensuring a streamlined process without compromising data integrity. Additionally, adjustments to questionnaire items were made to enhance clarity and mitigate potential respondent ambiguity, thereby fostering more accurate data interpretation. 6 These refinements were guided by a pragmatic approach aimed at maximising the intervention's impact and relevance within each unique context. By aligning methodology with the specific objectives and contextual characteristics of the intervention, we established a framework conducive to deriving meaningful insights and actionable outcomes. Table 1. Interventions included in REMEDIS Country Intervention Partner organisation (s) Description of intervention Gezinsbond, Child Focus Individual courses targeting parents, on different topics relating to digital parenting, such as cyberbullying, social media, gaming, and online privacy of children. Gezinsbond Series of three workshops aimed at teaching basic digital skills to vulnerable parents. Star Cloud LLC An introductory 2.5-hour webinar for teachers who use Opiq: i.e. interactive e-books in teaching to increase their specific digital skills and confidence in using the e-tools of the environment. Education and Youth Board of Estonia Three 2.5-hour teacher webinars, supported by homework and activities, allow teachers to learn about students' personalised selfassessment methods. The webinars help improve teachers' digital skills by guiding them in using digital platforms and tools. The Finnish Society on Media Education (FSME) Interventions were conducted for vocational school students on school days, focusing on digital multi-literacy as part of everyday life. Belgium Veilig Online Belgium Alle Ouders Digitaal Vaardig Estonia Opiq Webinar Estonia ÕPIRAAM course Finland Strengthening multiliteracy Finland Familiarising oneself with the digital University of the 3rd Age in Jyväskylä, CoE AgeCare project 7 Pilot course (1 day, 5.5h) developed to support elderly people’s digital service use and services of the wellbeing services county information security aspects on an everyday level. Poland The openminded and creative teacher of the future Polish Society for Educational Technology and Media, IH Bielsko Biała A training programme (at least 5 meetings x 2h + individual work) to strengthen basic digital competences and teachers' digital competence in the creation of open educational materials OER (video tutorials). Federation of Universities of the Third Age in Poland, Polish Radio Media Education Centre, OpenUJ, Krakowski Instytut Rozwoju Edukacji - Krakow Institute for the Development of Education The training programme is delivered in an online course format (supported by the OpenUJ platform) covering nine thematic modules related to effective digital inclusion. The programme is aimed at carers, trainers and future geragogical staff minimising digital exclusion among older adults. Foundation All About Science. University of the Third Age in Trzebinia, University of the Third Age in Chrzanów, Senior citizens' club in Zabierzów The programme is delivered in the form of two offline meetings (2x1.5h) in which intensive activities (including PBL, case analysis) are carried out to strengthen digital resilience media competence in manipulation and disinformation. The intervention targets a group with low media and digital competences or those at risk of digital exclusion. Poland Enhancing digital skills and teaching for carers of older adults Poland Strengthening media competence of people at risk of digital exclusion Spain Cibermanagers por la Igualdad PantallasAmigas 8 Programme (total: 8 weeks, 35hstudents: 5 weeks, 14h) aimed at teachers and secondary school students, focused on preventing violence against women in the digital context and promoting gender cyber-coexistence. Based on students’ agency (service, project and problem-based learning) to reinforce critical thinking skills, empathy, emotion management, and digital knowledge. Spain Desfake Verificat Educational project to develop students' (aged 14 to 15) ability to critically evaluate the information they receive through social networks by improving their ability to recognize reliable sources and identify rigorous information from that which is not. Citizens Online One-to-one sessions to provide online support for learners to develop digital skills and knowledge. Code Your Future Three-week in-person digital skills course on how to use smartphones in everyday life. UK Digital Champions UK Digital Literacy Programme UK The Good Things Learn My Way Foundation Consists of 11 courses on Technical, information navigation, Communication & interaction skills. 9 2 Development of the data collection strategy 2.1 Collaboration with the partner organisations in developing a data collection strategy In this report, we provide an in-depth analysis of the co-development process for each intervention programme evaluated, focusing on how the REMEDIS Canvas framework facilitated our approach. This REMEDIS Canvas served as a guiding tool, enabling us to plan, refine, and adapt our interventions in response to diverse contextual factors. Below, we outline our use of the Canvas framework and highlight any modifications made to enhance intervention effectiveness. Additionally, we discuss feedback received regarding organisational and national context issues that may impact implementation. Furthermore, we address limitations encountered during the formulation of data collection strategies, including constraints on session attendance and participant availability. Co-development Process and Canvas Use: Problem Definition and Context Analysis For each intervention, we began by defining the problem and conducting a comprehensive analysis of the contextual landscape. This involved identifying key challenges, stakeholder dynamics, and socio-cultural nuances influencing intervention outcomes. Using the Canvas, we outlined clear objectives and strategies tailored to address specific contextual characteristics. Stakeholder Engagement and Co-creation Stakeholder engagement was central to our co-development process, ensuring diverse perspectives were incorporated into intervention design. Through a collaborative workshop and consultations, we actively involved stakeholders at every stage, soliciting feedback and co-creating solutions. The Canvas provided a structured framework for stakeholder engagement, facilitating productive discussions and consensus-building. Intervention Design and Implementation Drawing on insights garnered from problem analysis and stakeholder engagement, we formulated intervention designs tailored to meet identified needs. Throughout implementation, we remained flexible, adapting strategies based on real-time feedback and contextual shifts. Monitoring and Evaluation Monitoring and evaluation were integral to our co-development approach, enabling us to assess intervention effectiveness and make data-driven adjustments. The Canvas facilitated the establishment of key performance indicators and evaluation metrics, streamlining the monitoring process. Feedback loops were established to solicit input from stakeholders, ensuring interventions remained responsive to evolving needs. 10 Feedback on organisational and national context Issues: Throughout the co-development process, we encountered feedback highlighting various organisational and national context issues that could potentially hinder implementation. These included resource constraints and cultural sensitivities. Addressing these challenges required proactive engagement with stakeholders and strategic navigation of complex institutional dynamics. We encountered limitations in formulating data collection strategies for some interventions. Constraints on the number of intervention sessions that could be attended and participant availability posed challenges in achieving comprehensive data coverage. Additionally, we prioritised qualitative insights to supplement quantitative data, ensuring a holistic understanding of intervention impact. Our co-development process, guided by the REMEDIS Canvas framework, facilitated the design and implementation of tailored interventions responsive to diverse contextual factors. While challenges and limitations were encountered, proactive engagement with stakeholders and strategic adaptation enabled us to overcome obstacles and derive meaningful insights for informed decision-making.In what follows, we provide an overview of each intervention programme evaluated in REMEDIS, detailing the co-development process and key components within the context of each REMEDIS country. Each description follows a similar format, outlining the purpose, stakeholders, access/resources, motivations/attitudes, media literacy and digital skills focus, achievement/engagement, satisfaction/outcomes, and organisational/national context issues specific to the intervention in the respective country. For each intervention programme evaluated in each REMEDIS focal country, the structure of the descriptions is as follows: Name of Intervention Programme - Country: - Purpose: A brief overview of the intervention's goals and objectives; Key stakeholders involved in the co-development and implementation process; Discussion on the availability and accessibility of resources necessary for intervention implementation; Insights into the motivations and attitudes of participants towards digital literacy and skill-building. - Media Literacy and Digital Skills Focus: Description of the specific focus areas within media literacy and digital skills addressed by the intervention. - Intervention Outputs and Outcomes: Discussion on the achieved outcomes and level of participant engagement throughout the intervention; Overview of participant feedback and the observed outcomes resulting from the intervention. - Organisational/National Context Issues: Identification and discussion of challenges related to organisational and national contexts that may have impacted intervention implementation. - Intervention enhancement/recommendations provided. 11 3 Overview of the intervention programmes: Objectives and stakeholder collaboration 3.1 Belgium 3.1.1.1 Veilig online: Intervention description and purpose Veilig Online ("Safe Online") is a critical initiative launched by the Gezinsbond, a Flemish organisation dedicated to supporting families, in collaboration with Child Focus, a foundation committed to the safety and welfare of children, including issues related to missing children, online safety, and prevention of sexual abuse and exploitation. The primary goal of Veilig Online is to equip parents with the essential knowledge and skills needed to effectively guide and protect their children's internet usage. For over fifteen years, Gezinsbond and Child Focus have been at the forefront of digital parenting education, leveraging their extensive experience to help parents navigate the complexities of the digital world. The workshops specifically target parents of children and youths aged 8 to 16, a critical period for developing safe and responsible online behaviours. Veilig Online workshops are organised in partnership with local organisations such as schools, libraries, and other stakeholders, demonstrating a communitybased approach to education and awareness. These local partners play a vital role, providing a venue for the workshops and handling logistical aspects such as marketing, communication, and registration. Meanwhile, Gezinsbond and Child Focus contribute by deploying trained instructors who deliver these informative and engaging workshops, ensuring parents receive practical and actionable advice to support their children’s safe internet use. This collaborative effort not only fosters a safe digital environment for children but also strengthens the community's overall commitment to protecting its youngest members online. Veilig Online targets a demographic often overlooked in the digital world. The audience of these courses is predominantly female, with 75% participating in "Veilig Online". These participants are parents with vulnerabilities such as a migration background, language barriers, low literacy, or low education, many of whom are unemployed or work part-time. The Veilig Online workshops are designed to be accessible and affordable, with a nominal fee of just a few euros, thanks to subsidies that help keep costs low for participants. However, the primary barrier to participation is not financial but temporal. Each workshop lasts two hours: this time commitment can be particularly challenging for parents juggling multiple responsibilities or those with demanding work schedules. The challenge of accessibility is compounded when trying to engage parents from economically vulnerable backgrounds. These parents often face additional hurdles, such as irregular working hours, limited access to transportation, or lack of childcare, making attendance even more difficult. One of the major challenges highlighted during the workshop was the frequent cancellation of the sessions due to a lack of participants. This issue underscores the need to reconsider the approach to recruitment and engagement in these critical programmes. Despite these challenges, the motivations for attending the Veilig Online workshops are quite strong. Parents are primarily driven by a desire to better support and protect their children in an increasingly digital world. They recognise the importance of being informed and proactive 12 about their children’s online safety, including understanding the risks and learning strategies to mitigate them. There is also a significant interest among parents in staying updated with the digital landscape, which is continually evolving. They are keen to bridge any knowledge gaps that may exist between themselves and their digitally native children. This eagerness is often fueled by a combination of concern for their children's well-being and a personal desire to be involved in their children's lives as comprehensively as possible. However, attitudes can vary, with some parents feeling overwhelmed by the complexity of digital parenting or skeptical about their ability to control or influence their children's online behaviours. Addressing these attitudes through the workshops, by providing clear, manageable strategies and fostering a supportive community environment, can help increase participation and engagement, making the sessions more effective and beneficial for all attendees. 3.1.1.2 Media literacy and digital skills targeted The Veilig Online initiative is structured around a comprehensive series of workshops, each focusing on critical aspects of digital literacy and safety. This suite of workshops includes six distinct topics tailored to address the diverse needs of digital parenting: Digital toddlers: This workshop educates parents on how to introduce digital devices to toddlers safely. It covers age-appropriate technologies, setting boundaries for screen time, and ensuring content suitability. Social media: Participants learn about the various platforms their children might use and the risks associated with social media, such as privacy breaches and exposure to inappropriate content. It also discusses strategies for monitoring and guiding children's social media activities constructively. Privacy: This session focuses on teaching parents how to protect their and their children’s personal information online. It includes understanding privacy settings, recognizing secure websites, and knowing the implications of data sharing. Gaming: The workshop covers the positive and negative aspects of online gaming, such as potential for skill development and risks like addiction and exposure to online predators. It also provides tips on how to choose age-appropriate games and encourage healthy gaming habits. Cyberbullying: Parents learn how to identify signs of cyberbullying and understand its impact. The workshop offers advice on how to talk to children about cyberbullying, strategies to prevent it, and steps to take if their child is affected. Online relationships and sexuality: This topic addresses how the internet can influence children's views on relationships and sexuality. It discusses how to educate children about maintaining healthy online interactions and the importance of consent and privacy. Each workshop is designed to be interactive, combining various teaching methods such as discussions, knowledge exchange, and interactive quizzes to engage participants fully. While the primary focus is on sharing knowledge rather than developing technical skills, the 13 workshops leverage expert testimonials and real-life scenarios to enhance learning outcomes. The format is conducive to a group size of up to 25 participants, ensuring a robust exchange of ideas without overwhelming the facilitator or the attendees. Organisations like schools and libraries play a crucial role in facilitating these workshops. They provide the necessary infrastructure, such as a suitable venue, and handle the promotion and registration processes, ensuring that the workshops reach a broad audience. The Veilig Online team, in turn, supplies well-prepared content and skilled facilitators to deliver these crucial sessions, emphasising the empowerment of parents through increased awareness and understanding of digital environments. This strategic partnership allows Veilig Online to effectively disseminate vital digital literacy skills among parents, equipping them to better guide their children in the digital age. 3.1.1.3 Intervention outputs and outcomes The Veilig Online initiative is designed to enhance the digital literacy of parents, enabling them to guide and support their children in navigating the complexities of the online world safely and responsibly. The workshops focus on imparting knowledge that helps parents feel more assured in their digital parenting roles. Here's an expanded look at the outputs and outcomes of these workshops. The primary outputs of the Veilig Online workshops include: Comprehensive workshop delivery: Each session covers one of six key topics: digital toddlers, social media, privacy, gaming, cyberbullying, and online relationships and sexuality. These workshops are facilitated by trained professionals and designed to be interactive and engaging, using discussions, expert testimonials, and quizzes. Educational materials and resources: Participants receive various resources and materials that can be used as references post-workshop, aiding in the continued application of the knowledge gained during the sessions. Evaluation forms: At the end of each workshop, participants are asked to complete an evaluation form. This form assesses their satisfaction with the content and delivery, as well as the applicability of the knowledge imparted. The outcomes of the Veilig Online workshops are geared towards achieving long-term benefits for parents: Enhanced digital literacy: Parents gain an understanding of critical digital literacy dimensions such as online safety, privacy management, and the social and psychological impacts of digital usage. This knowledge helps them to better support their children in developing healthy and safe online habits. Increased confidence in digital parenting: As parents learn more about the digital environment and the specific challenges and opportunities it presents, they feel more confident in their ability to guide their children. This confidence stems from a deeper understanding of the platforms and tools their children are using and the risks associated with them. 14 Critical thinking and decision-making skills: The workshops encourage parents to think critically about digital content and technology use. This aspect is crucial for helping parents make informed decisions about the digital media their children consume and the activities they engage in online. Community and network building: By participating in these workshops, parents connect with other like-minded individuals facing similar challenges. This networking can lead to the formation of supportive communities that share resources, advice, and experiences, further enriching the learning environment. Feedback for continuous improvement: The data collected from the evaluation forms is used to continuously refine and improve the workshops. This feedback loop ensures that the workshops remain relevant, effective, and responsive to the needs of the participants. By focusing on digital knowledge and critical aspects of digital literacy rather than technical/operational skills, Veilig Online ensures that parents are not only educated but empowered to make wise decisions regarding their children’s digital interactions. This strategic focus contributes to a safer online environment for children and helps build a foundation of responsible digital citizenship within the community. 3.1.1.4 Organisational/national context issues The Veilig Online workshops are situated within a complex organisational and national context that reflects broader debates about digital media use by children and young people in Flanders and other European countries. Here is an expanded examination of how these issues shape and influence the intervention. Organisations like Child Focus and Gezinsbond play a critical role in shaping the discourse around children’s digital media use. Given their longstanding commitment to supporting families, these organisations advocate for a balanced approach to digital parenting that moves beyond the simplistic solutions of bans or unrestricted access. Instead, they emphasise: Education over restriction: Their position is that teaching parents and children about the benefits and risks associated with digital media is more beneficial than outright bans. This educational focus helps parents make informed decisions based on a deep understanding of the digital landscape. Promoting digital opportunities while addressing risks: The workshops are designed to highlight not only the potential dangers of screen time and social media but also the positive opportunities these technologies can provide. This dual focus ensures that the narrative is balanced and constructive. Empowerment through knowledge: By providing digital knowledge and tools, the workshops empower parents to develop their own strategies for managing their children’s digital media use. This approach respects the unique dynamics of each family and avoids the pitfalls of a one-size-fits-all solution. 15 The broader debate in Flanders and other parts of Europe often revolves around the impact of screen time on children’s development and the potential harms of early and unsupervised social media use. In this climate, the approach taken by Veilig Online is particularly relevant: Responding to public concerns: The workshops address public and political concerns by integrating the latest research and expert opinions into their content. This ensures that the information is not only current but also scientifically grounded. Influence on policy: By actively participating in the discourse, organisations like Child Focus and Gezinsbond can influence broader policy discussions about children's digital media use. Their evidence-based approach advocates for policies that are informed by empirical data rather than knee-jerk reactions to societal fears. Adaptation to changing technologies and norms: The digital landscape is continually evolving, with new platforms and technologies emerging regularly. The workshops must, therefore, remain adaptable and responsive to these changes to stay relevant and effective. Collaboration across borders: The issues of digital literacy and children’s media use are not confined to national boundaries. By engaging with similar organisations across Europe, Veilig Online can share best practices, learn from others’ experiences, and advocate for coherent European policies regarding digital media education. By addressing these organisational and national context issues, the Veilig Online workshops contribute to a nuanced understanding of digital parenting. This approach not only aligns with the needs and values of modern families but also supports the development of informed, responsible digital citizens. 3.1.1.5 Intervention enhancement/recommendations provided The Veilig Online intervention's efficacy can be enhanced significantly by refining its format, increasing collaboration with local communities, and improving evaluation techniques. Each of these areas can be specifically tailored to better meet the needs of the participants and maximise the impact of the training. Here is an examination of (future) enhancements and recommendations: Multiple training sessions: The decision to shift from a single session format to multiple training moments could be instrumental in deepening the learning outcomes for participants. Research supports the idea that repeated exposure to information and skills development over time leads to more substantial and lasting learning effects. Sequential learning: Break down complex digital literacy topics into smaller, manageable parts delivered over several sessions. This spacing allows participants to absorb and reflect on the information fully. 16 Progressive skill development: Start with foundational concepts in earlier sessions and gradually introduce more advanced topics, which helps maintain participant engagement and ensures a building-block approach to learning. Co-creation and local integration: Enhancing the relevance and accessibility of the Veilig Online workshops through collaboration with local entities such as schools, libraries, and community centres can lead to more tailored and impactful interventions. 3.1.2.1 Alle Ouders Digitaal Vaardig: Intervention description and purpose The "Alle ouders digitaal vaardig" (AODV) intervention programme developed by Gezinsbond is designed specifically for parents in Flanders, Belgium, who are identified as having low digital skills. The programme is committed to enhancing the digital skills of parents, enabling them to use these technologies effectively in their everyday lives. This includes navigating online platforms, understanding digital security, and using digital tools to manage family logistics. It also aims to empower parents to seek and use support when facing digital challenges. This involves educating them on how to identify and approach the right resources or organisations for help and encouraging them to build the confidence necessary to ask for assistance. Ensuring that participants have access to the necessary resources is crucial for the success of the AODV programme, which provides participants with access to internet connectivity during the workshops. This not only facilitates practical learning but also ensures that lack of access at home is not a barrier. AODV supplies a range of learning materials, including tutorials, guides, and online resources, tailored to the learning pace and style of adults with low initial digital skills and often low literacy levels in general. Gezinsbond collaborates with local libraries, community centres, and educational institutions to host training sessions and provide ongoing support, making resources more accessible to a broader audience. Many participants are motivated by the desire to become more independent in managing digital aspects of their daily lives, such as online banking, shopping, or communicating with schools. Parents are often driven by the wish to engage more fully in their children’s education and social activities, which are increasingly mediated through digital platforms. The programme addresses common fears and reservations about technology, such as concerns about privacy, fear of making mistakes, and feeling overwhelmed by the fast pace of technological change. AODV fosters a supportive community atmosphere where parents can share experiences and learn from each other, reducing the stigma or embarrassment they may feel about their skill levels. By focusing on these comprehensive aspects, the AODV programme not only equips parents with essential digital skills but also enhances their confidence and motivation to use these skills actively. This holistic approach ensures that the benefits of the programme extend beyond individual participants, potentially impacting their families and wider communities. 3.1.2.2 Media literacy and digital skills targeted The "Alle ouders digitaal vaardig" (AODV) programme is thoughtfully structured into a series of three workshops, spread over three weeks, with each session lasting two hours. Designed 17 to accommodate small groups, each workshop hosts between five and eight participants, fostering an intimate and supportive learning environment that encourages active participation and personalised attention. Workshop 1: Introduction and Assessment The first workshop serves as the foundational session where the facilitator and participants get acquainted. The primary goal during this initial meeting is to understand each participant's digital proficiency and their specific learning objectives. Through engaging in simple exercises, the facilitator assesses the participants' current digital skills levels. This assessment allows the facilitator to tailor the following sessions more effectively to meet the group’s needs. This session sets the tone for the workshops, establishing a comfortable environment where participants can openly discuss their digital challenges and aspirations. Workshop 2: Building Core Skills In the second workshop, the focus shifts to imparting essential concepts of media literacy and foundational digital skills. This session is crucial for equipping participants with the knowledge needed to navigate the digital world more effectively. The facilitator introduces topics such as internet safety, understanding digital content, and the basics of using digital tools and platforms. This workshop aims to build a solid base of knowledge that empowers participants to interact with digital media confidently and responsibly. Workshop 3: Customisation and Continuing Support The final workshop is highly customised to address the specific needs that emerged from earlier sessions. This session is participant-driven, with the facilitator working closely with attendees to delve deeper into particular areas of interest or difficulty identified through the course. It also focuses on practical application, where participants practice the skills they’ve learned and explore scenarios that they might encounter in their everyday digital interactions. Additionally, this workshop provides crucial guidance on where participants can find further help and resources once the workshops conclude, ensuring they feel supported in continuing their digital literacy journey beyond the classroom. This structured approach not only facilitates a gradual and thorough learning process but also ensures that each participant can progress at a pace that suits their individual needs. By the end of the three weeks, participants are not only more knowledgeable but also more confident in their ability to use digital tools effectively and safely in their daily lives. 3.1.2.3 Intervention outputs and outcomes The Veilig Online workshops implement a structured approach to evaluating their effectiveness, employing both quick and comprehensive tools to capture participant feedback and measure the workshops' outcomes. This systematic evaluation process is integral to understanding the impact of the intervention on the participants and to further refine the workshop content and delivery. 18 At the end of the first and second workshops, the organisation has incorporated a brief evaluation period, lasting approximately 2-3 minutes. During these moments, Mentimeter, an interactive presentation software, is used to quickly gather real-time feedback from participants. This tool allows attendees to respond to questions via their smartphones or tablets, providing immediate insights into their understanding and satisfaction with the workshop content up to that point. The use of Mentimeter is particularly effective for gauging immediate reactions and for its ease of use, which encourages higher participation rates among attendees. The final workshop includes a more detailed evaluation session, which extends for about 7 minutes. For this thorough assessment, Google Forms is employed to collect more in-depth feedback from participants. This platform allows for the creation of detailed surveys that can include a range of question types, from multiple-choice to open-ended responses, enabling the organisation to gather data on various aspects of the workshop experience. This data is crucial for assessing the overall impact of the workshop series and identifying areas for improvement. These evaluation moments are strategically placed to ensure that the organisation can monitor the effectiveness of the workshop content and the facilitation methods throughout the series. The feedback collected through both Mentimeter and Google Forms is analysed to make any necessary adjustments to the workshops, enhancing their relevance and effectiveness for future iterations of new, similar workshops. This ongoing evaluation process not only helps in fine-tuning the intervention but also ensures that the workshops continue to meet the evolving needs and expectations of the participants, ultimately leading to a more informed and digitally competent parent community. 3.1.2.4 Organisational/national context issues The Alle ouders digitaal vaardig (AODV) intervention was a noteworthy project, albeit a temporary one, that was primarily funded by the Flemish government. This funding arrangement meant that the project had a constrained operational timeline, running only from October to December 2023. Within this brief period, AODV faced numerous challenges, including many cancellations which impacted the continuity and potential outreach of the program. The temporary nature of AODV underlined a critical issue within the organisational and national context—sustainability of initiatives aimed at improving digital literacy among parents with low digital skills. The limited duration and funding of such projects raise concerns about their long-term impact and the ability of the participants to continue improving their skills without ongoing support. Furthermore, being a government-funded initiative, AODV was subject to fluctuations in policy and funding priorities. This dependency means that without a consistent commitment from the government or the acquisition of alternative funding sources, the continuation of such beneficial programmes remains uncertain. The organisational framework of AODV, managed by Gezinsbond with the support of the Flemish government, highlighted the importance of collaboration between non-profit organisations and governmental bodies in addressing digital literacy. However, it also brought to light the need for a more sustainable approach that does not rely solely on temporary 19 projects. For future initiatives, securing stable funding, perhaps through partnerships with private sectors or long-term government grants, is vital. Moreover, the national context of increasing digitisation and the push towards integrating digital competence of parents makes programmes like AODV essential. The Flemish government's support for such initiatives is crucial not only for their initiation but also for their sustained operation, ensuring that all community members can benefit from the digital era. 3.1.2.5 Intervention enhancement/recommendations provided To optimise the effectiveness of the Veilig Online workshops, it is essential to engage comprehensively with the communities served, starting with conducting needs assessments in collaboration with local partners. These assessments provide vital insights into specific community concerns and gaps in digital literacy, allowing for the customisation of workshop content to address the most pressing needs directly. Furthermore, enlisting local champions— respected and well-known figures within the community—to promote and deliver the workshops can significantly increase trust and encourage higher attendance. Regularly soliciting feedback from community members and stakeholders is another critical strategy. This continuous input helps refine the content and delivery methods of the workshops to better suit local preferences and conditions, ensuring that the program remains relevant and impactful. The evaluation and adaptation of the workshops are also crucial for their success. Implementing ongoing evaluation processes at various points during the intervention, not just at its conclusion, helps measure knowledge acquisition and shifts in attitudes effectively. These evaluations include both pre- and post-session assessments. Moreover, using visual aids and infographics can greatly enhance understanding and overcome language barriers, making complex information more accessible and engaging. Adaptive learning models are integral to the programme's agility. The data gathered from continuous evaluations should be used to make immediate adjustments to the workshop structure, content, or delivery methods. This adaptability ensures that the intervention remains effective and responsive to the evolving needs of participants, thus maintaining its relevance and maximising its impact in fostering digital literacy. 3.2 Estonia This section provides an overview of a webinar series facilitated by the Education and Youth Board (HARNO), a division of the Ministry of Education and Research. It outlines the webinars' key goals, participation incentives, necessary resources, outcomes, and associated challenges. The concluding part discusses enhancements made to improve the intervention's effectiveness. HARNO plays a crucial role in implementing Estonian education and youth policy, spanning from primary to higher education. They promote high-quality, modern, and accessible education in Estonia, helping individuals to craft personalised professional development paths. 3.2.1.1 ÕPIRAAM webinar series for teachers: Intervention description and purpose ÕPIRAAM, or the Teaching and Learning Framework, is a practical tool designed to enhance educational effectiveness. It guides teachers in crafting secure, efficient, and innovative 20 learning experiences across five key areas: effective learning and motivation, mental and physical health, digital competencies, and compliant instructional resources. The framework helps teachers address contemporary educational challenges, balancing student well-being with academic progress. Additionally, the intervention integrates online testing with ÕPIRAAM to improve digital learning experiences through well-designed assessments. The intervention includes six structured components: Contact seminar: An introductory session that provides a foundational understanding of the topics. Independent work 1: Participants explore the ÕPIRAAM learning framework on their own. Independent work 2: Trial of self-monitoring tests with students to evaluate effectiveness. Intermediate seminar: A session for reflecting on progress and discussing challenges encountered. Independent work 3: Participants analyse test results to enhance data-driven decisionmaking. Contact seminar for feedback: A final meeting to exchange insights and gather feedback for ongoing refinement. The intervention is designed to equip educators with the skills and resources necessary to maximise the benefits of digital self-assessment for learners. It trains teachers to use digital tools and platforms that enable students to perform self-assessments and reflect on their educational progress. The focus extends beyond simply administering digital self-assessment tests; it also includes aligning these assessments with educational objectives. This approach fosters a personalised, informed learning experience, ultimately improving learning outcomes. At the time of intervention, participants must be engaged in teaching as the course includes practical assignments that must be implemented in a classroom setting. Additionally, they must have access to a computer with internet connectivity, a Google account, and the Zoom application for active engagement. The intervention does not supply additional devices or services, as teachers typically have these resources available at their schools. There are no prerequisites or participation fees required to join the course. Even though the individual teachers' motivation to participate in the professional development activities administered by HARNO varies, it is centred around several key factors: ● ● ● Skill enhancement: With the evolving landscape of education technology and methodologies, teachers are motivated to update their skills to deliver better quality education. Networking opportunities: Participation in HARNO's activities provides teachers valuable networking opportunities, allowing them to connect with peers, share experiences, and collaborate on innovative teaching methods. Incentives offered: HARNO offers free access to the intervention. Additionally, participants are incentivised with certificates of participation, validating their enhanced skills and knowledge. 21 ● ● ● ● Interest in specific topics: HARNO tries to offer activities that align with teachers’ specific interests or areas they wish to improve, whether it is digital literacy, subjectspecific teaching methods, or student engagement strategies. Collaborative learning environment: The intervention fosters a collaborative learning environment. Teachers value the opportunity to learn from each other's experiences and perspectives. Easily accessible: the intervention is free and web-based, so it’s easy to take part in. Career advancement: Many Estonian teachers see professional development as a pathway to career advancement and enhancing their teaching credentials. Participation in the ÕPIRAAM intervention and enhancement of digital competences may contribute to the teacher’s individual e-portfolio en route to earning an occupational qualification certificate for teaching per the requirements of the national occupational qualification standard. Additionally, certain negative motivators can prompt teachers to participate in professional development, particularly those focusing on digital skills and media literacy. These motivators include: Awareness of inadequate technology skills: Recognising personal gaps in digital skills drives teachers to seek professional development to boost their competence and confidence in their roles. Fear of falling behind: Rapid technological advancements may create a sense of urgency, pushing teachers to update their skills to keep up with peers and meet institutional expectations. Professional expectations: The increasing importance of digital literacy in education compels teachers to adopt technology-enhanced teaching methods to align with modern educational standards. Overall, while the motivations may vary among Estonian teachers, the combination of career and qualification advancement opportunities, skill enhancement, networking, and the incentives provided by HARNO is a compelling factor for teachers to participate actively in the given intervention. 3.2.1.2 Media literacy and digital skills targeted The HARNO Intervention consists of the following elements aligned with DigCompEDU (DigCompEdu, 2017), which is the framework describing what it means for educators to be digitally competent: ● Technical and operational skills ○ Teachers learn to use various online self-assessment platforms such as Curipod, Quiz, Kahoot, Conker, Gibbly. ○ Teachers learn to use AI tools in preparing tests (Curipod, Quizz, ChatGPT, Copilot) ● Information navigation and processing ○ Working through the methodology of Õpiraam (reading academic articles) 22 ○ Developing skills to efficiently locate, evaluate, and utilise digital information for learning ● Content creation and production ○ Designing and implementing lessons by incorporating new knowledge ○ Creating quizzes and self-assessment tests using digital tools such as Kahoot and Quizlet ● Communication and interaction techniques ○ Analysing practical experiences for homework ○ Creating presentations to share insights with peers ○ Using Zoom for virtual meetings (annotation, breakout rooms, polls) ○ Engaging in online group chats ○ Collaborating through Google Docs to share files, and provide feedback 3.2.1.3 Intervention outputs and outcomes The HARNO intervention, while focusing on enabling teachers to efficiently integrate in their instructional process students’ digital self-assessment methods/tools, at the same time, fosters teachers ML and DS competence. The given intervention benefits teachers and (indirectly) their students in the following way: ● Increased confidence in digital skills: ○ Teachers feel more assured using a variety of digital tools. ○ Teachers adeptly create digital content, in particular, learner self-assessment ● ● ● ● tests and quizzes. Enhanced communication and collaboration through digital media: ○ Teachers engage in effective information exchange via video communication platforms such as Zoom. ○ Teachers provide feedback efficiently through digital shared documents such as Google Docs. Feedback and personalisation facilitated by digital resources: ○ Enhanced feedback loops for continuous professional development. ○ Teachers are equipped with skills to select and integrate diverse digital learning materials, such as interactive tools for self-assessment tests, enabling students to engage in reflection and self-development. Innovative digital learning experiences: ○ Teachers are equipped with the skills to create personalised and innovative digital learning experiences for their students ○ Technologically enhanced classroom climate is established, leading to greater learner engagement. Mental health and well-being: ○ Teachers feel more empowered, positively affecting their self-confidence and reducing their technology-use-related anxiety. ○ Supportive and trusting relationships among teachers and their students are established. 23 ● Professional growth and networking: ○ New professional connections with colleagues from various schools are built digitally, fostering collaboration and sharing best practices. ○ Teachers acquire new knowledge and skills, enabling them to use pedagogically informed digital tools more efficiently for learners’ self-assessment. ○ Teachers practice technologically-enhanced ways of collaboration with their colleagues. HARNO benefits from improved educational practices and teacher development resulting from the intervention. The University of Tartu provides expertise, resources, and academic guidance to support this initiative. Below is the list of outcomes of the intervention: ● Enhanced teaching practices: Teachers improve their instructional methods, ● ● ● ● ● bolstering effectiveness. Digital competence: Teachers gain confidence and proficiency in using digital tools. Collaboration: Digital platforms foster seamless communication and collaboration among educators. Personalised learning: Integration of digital resources enables tailored learning experiences for students. Innovation: Teachers embrace innovative digital teaching methods. Professional growth: Reduced anxiety and networking opportunities support continuous learning and knowledge sharing among educators. 3.2.1.4 Organisational/national context issues Within the REMEDIS intervention enhancement initiative, many of the recommendations derived from the ML and DS literature review (Vissenberg et al., 2023) had already been implemented by HARNO. As a result, there was minimal need for co-development in this area. The intervention provider, however, did consider the recommendations (see Appendix 1) very valuable for less experienced educators/trainers as a checklist to make sure their courses or interventions on media literacy and digital skills are in line with research. Additionally, in general, there is a reluctance among teachers in Estonia to participate in intervention evaluations. This hesitancy stems from their already overloaded schedules, time constraints, and a perceived lack of direct benefits from the evaluation. Consequently, such limited engagement leads to insufficient interest and reduced participation in the intervention. The limited quantitative data available after the intervention studies hinders thorough data analysis and a comprehensive evaluation of the interventions. The intervention provider also noted that the instrument used for evaluation in REMEDIS was not in line with the specific ÕPIRAAM intervention in their opinion. Intervention providers’ personal appeals and arguments in favour of providing valuable data for research (while keeping participation voluntary and all responses anonymous) were, thus, key to establishing a trusting relationship with the participants and an adequate response rate. However, there was no negative feedback regarding the instrument or data collection from the participants. Moreover, intervention providers hesitate to permit extensive researcher observations, fearing they may disrupt the instructional process and group dynamics. Additionally, both providers 24 and participants are rather reluctant to engage in lengthy interviews due to time constraints, or simply a lack of motivation. Consequently, these factors limit the collection of both qualitative observational and interview data, impacting the depth and comprehensiveness of the evaluation. 3.2.1.5 Intervention enhancements/recommendations provided This intervention has undergone several foundational improvements, initially suggested in our systematic evidence review, being integrated over time. An impactful enhancement by the ÕPIRAAM provider was the introduction of peer feedback sessions during their webinars, enhancing individual work and fostering collaboration. This feature was so beneficial that it was added as an assignment in the course, receiving positive feedback from participants who valued the peer insights and collaboration during their final presentations. To increase engagement and measure proficiency with digital tools, the provider used interactive Zoom polls, effectively assessing participants' skills and promoting active involvement. Additionally, to ensure thorough evaluation and address any recurring issues, we recommended the inclusion of external experts using established evaluation methods for a comprehensive assessment of the intervention's effectiveness. 3.2.2.1 Star Cloud LLC: Opiq webinar intervention description and purpose This document outlines the webinar conducted by Star Cloud LLC in Estonia, which primarily focuses on the pedagogical use of digital resources, with a secondary emphasis on Media Literacy and Digital Skills. The content covers the webinar's main objectives, participation incentives, necessary resources, achieved outcomes, and challenges encountered during the intervention. The concluding section details specific enhancements that were implemented to improve the results of the intervention. Star Cloud LLC is the developer of Opiq software & services. Opiq is a cloud-based learning environment that contains fully digitised high quality textbooks from leading Estonian educational publishers with interactive assignments, selftesting, and other assessment possibilities, including both immediate automatic feedback and teacher feedback. Opiq materials are (Estonian) curriculum-aligned resources that are kept upto-date. The given digital materials help make the school instructional process innovative and interactive. Opiq textbooks were also widely used in Estonian schools during COVID-19 pandemic for e-lessons. The online resources adhere to the latest cloud technologies and security protocols in alignment with the privacy standards set forth by the European General Data Protection Regulation (GDPR). Star Cloud LLC also offers professional development activities that support teachers in adopting and integrating Opiq digital resources in their teaching in an efficient manner. Opiq is operating in six countries including Finland and Lithuania. Star Cloud is a partner for educational content providers, schools and the public sector since 2014. Opiq webinars are scheduled regularly to introduce new functionalities within the Opiq environment, helping teachers enhance their instructional techniques. Prior to the autumn session, Starcloud LLC conducts a survey to determine which features teachers currently use and how they wish to expand their skills using these tools. This feedback informs the structure of the webinars. The intervention consists of a 1.5- to 2-hour interactive webinar, offered on three different dates, where the Opiq teacher interface was showcased to in-service teachers. 25 These participants are predominantly new to the Opiq platform or interested in adopting it for their schools. The providers tailor the webinar content based on information collected from the participants ahead of time. To ensure the webinar addresses specific needs, participants are encouraged to submit questions beforehand, allowing for a focused discussion on topics requiring more extensive coverage. The intervention is designed to empower in-service teachers with the skills to effectively use digital instructional and assessment resources. Through the interactive webinar, teachers are guided in seamlessly integrating digital materials, textbooks, and self-assessment tests into their teaching practice. Using Opiq materials not only provides teachers with secure and highquality digital resources, which positively affects their instructional quality but also streamlines their lesson preparation process. The Opiq platform further supports teachers in conducting ongoing formative assessments and making data-driven pedagogical decisions. Through digital tracking, teachers can closely monitor student progress, provide timely feedback, and gain insights into individual and class performance, facilitating a more personalised learning experience. For students, active use of digital resources eliminates the need to carry heavy textbooks, promoting a more physically comfortable learning environment. Consequently, learners benefit from a technologically advanced and pedagogically enriched educational experience, improving their physical wellbeing and engagement in the instructional process. Participants need Internet access, a computer, and the Zoom app to attend the online webinar. As part of the intervention, no additional devices are provided, as teachers typically have these resources at school. There are no prerequisites or participation fees required for enrollment. However, since the Opiq platform itself is not a free resource, not all participants have access before or after the intervention. This results in restricted accessibility. Participating in the Opiq webinar/intervention offers several motivational benefits for educators: ● ● ● ● Individualised feedback: Participants are encouraged to ask specific questions about using Opiq materials in their teaching practices, ensuring tailored guidance and support. Access to readily available quality and safe digital resources: The intervention offers high-quality, secure digital resources like interactive workbooks, learning materials, and auto-graded tests, enhancing teaching and learning while reducing assessment time for teachers and students. Increased efficiency of instructional practice: Teachers can monitor individual student progress through robust usage statistics, fostering a more personalised and dynamic classroom learning environment. This data-driven approach positively impacts classroom dynamics and instructional effectiveness. Free access to the webinar: Participants can join the webinar at no cost, making the intervention accessible to all interested educators without financial barriers. 26 3.2.2.2 Media literacy and digital skills targeted The webinar supports and encourages educators to practise media literacy and digital skills. The main components covered include: ● ● ● Technical and operational: teachers are encouraged to actively employ technology to access and use Opiq materials. Information navigation: teachers are given access to readily available quality and safe digital resources, which they can navigate to identify relevant instructional resources. Digital communication: the Opiq platform allows seamless digital communication and material sharing (e.g., by allowing connections to communication platforms such as Stuudium and E-kool). Additionally, teachers further improve their digital communication competence by participating in the Oqpiq webinar through Zoom. 3.2.2.3 Intervention outputs and outcomes The Oqiq webinar is designed to bring about a range of positive developments for both educators and students. The webinar aims to contribute to the overall educational growth, and improved instructional process, communication and networking, the mental health, and overall well-being of the teachers. Key Outcomes for educators: Innovative digital learning: Teachers gain skills to create personalised digital learning experiences, select diverse digital resources, and employ various teaching methods supported by digital training. Data-driven decision-making: Educators learn to monitor student progress through detailed digital usage statistics. Feedback and personalisation: The Oqiq platform offers customised digital content recommendations to address individual student needs. Enhanced communication and collaboration: Teachers enhance interaction through video platforms like Zoom and share resources via Stuudium and E-kool. The webinar also fosters professional networking and collaborative exchanges of best practices. Well-being: The webinar boosts teacher confidence and competence, reducing anxiety and stress, and enhancing overall job satisfaction. These impactful outcomes collectively contribute to a positive and transformative educational experience for both educators and students: 1. Educational growth: Contributing to overall educational advancement and improvement in instructional processes. 2. Communication and networking: Enhancing communication channels and fostering networking opportunities among educators. 3. Mental health and well-being: Promoting teacher well-being by boosting confidence, reducing anxiety, and managing workload stress. 27 4. Innovation in teaching: Encouraging innovative teaching methods and empowering educators with diverse digital resources. 5. Student engagement and academic outcomes: Enriching student learning experiences and increasing engagement, leading to improved academic performance. 3.2.2.4 Organisational/national context issues Many of the recommendations for intervention enhancements that originated from the ML and DS literature review results could not be implemented due to lack of time (intervention duration 1.5-hour stand-alone webinar) and financial resources. Consequently, the codevelopment part was not optimal. Due to work overload, lack of time, and no direct benefits perceived with regard to the proposed intervention evaluation, teachers are reluctant to engage in an extensive intervention outcome/satisfaction survey process. This resulted in limited participation and scarcity of data to conduct rigorous analysis. The intervention providers also noted that the instrument used for evaluation in REMEDIS was too broad for the Opiq intervention in their opinion, and especially the questions concerning media literacy were foreign to them. However, there was no negative feedback regarding the instrument or data collection from the participants. 3.2.2.5 Intervention enhancements/recommendations provided While the intervention providers acknowledged the value of our recommendations, they opted not to implement specific changes for the three webinars in question. They appreciated the concept of allowing teachers to experience and practice from a student's perspective but felt constrained by the required resources, both financial and technical, to execute this idea. Another suggestion to use breakout rooms for group discussions was considered but ultimately deemed incompatible with their workshop design, as they felt they needed to observe and guide each group, which posed human resource challenges. With a total of 132 participants across the three webinars (fewer than usual), they expressed a preference for smaller group sizes in future sessions, potentially enabling the use of breakout rooms in Zoom. Interestingly, some of our recommendations found application in face-to-face courses conducted in schools and at the Education festival in Estonia. Here, mathematics teachers had the opportunity to experience the student's view of the platform and actively engage in solving tasks, garnering positive feedback. Inspired by our suggestions, the providers also plan to incorporate word clouds and Kahoot in their future sessions. Additionally, to ensure a thorough evaluation of the activity's effectiveness and to prevent recurring shortcomings, we recommended involving external experts. These experts, using reliable and recognized evaluation methods, could provide an objective assessment of the intervention's impact. Overall, while substantial changes were not implemented, the providers, with years of experience in conducting these webinars, noticed enhanced collaboration and better work distribution during the sessions after our discussions on potential improvements. 28 3.3 Finland 3.3.1.1 The pilot course: Intervention description and purpose This intervention was initiated, planned, prepared, and carried out by the Finnish Society of Media Education (FSME), which partnered with Finnish vocational schools where the intervention took place. Some regional libraries helped recruit the schools. Public libraries and vocational school teachers benefit from the intervention materials that will be useful for them in the future. Initially, it was planned that the intervention would cover five topics: information literacy, visual literacy, participation, digital security, and online interaction. Teachers at vocational schools were asked in advance to consider the topics they would prefer from this list. The most preferred topics were information literacy and visual literacy, as well as interaction on social media. Therefore interventions were primarily centred around these topics. Information literacy teaching includes skills necessary to assess the reliability of sources to recognise different motivations communicators may have and to recognise expert knowledge and expertise. The focus of visual literacy was primarily on memes and different online images. Interventions addressed topics also related to social media interaction, including cyberbullying. The intervention aimed to strengthen young people's versatile and flexible multi-literacy skills, which include the interpretation and knowledge of visual information and audiovisual texts, as well as critical reading skills. The project also emphasised the importance of multi-literacy as part of everyday life, working life, and functioning in society. The interventions were carried out in vocational school classrooms with ordinary equipment, including a blackboard and a projector for PowerPoint presentations. There were no computers or other devices than personal mobile phones used by participants. Students used their own mobile phones for a couple of tasks. There were no fees associated with participation, nor were students given any special rewards or incentives for participation. Students participated in the intervention while attending school. In some cases, students did not know that the intervention would take place before they entered the classroom, but some groups were better informed. At the beginning of the intervention, teachers usually introduced the FSME representative to the class and explained that he would conduct the workshop. Student motivation for the intervention was not thoroughly discussed prior to or during the sessions. Observations by an FSME representative revealed varied levels of engagement among students. Some were actively involved, while others appeared disengaged, likened to "sleeping at the back of the class." Motivation fluctuated with the relevance of the content; students were more engaged with topics like memes, which closely relate to their daily media interactions, and less interested when the content felt distant from their personal experiences. 3.3.1.2 Media literacy and digital skills targeted Interventions addressed the following types of digital skills: ● ● ● Information navigation and processing Communication and interaction Content creation and production 29 Teaching of technical, operational skills and programming related skills was not included in this intervention. 3.3.1.3 Intervention outcomes and outputs ● ● ● ● Information navigation & processing: increased students' ability to assess the reliability of different sources online, to recognize expert knowledge and expertise, and to recognize different motivations communicators may have in online environments. Developed critical thinking towards memes and different online images. Informal relationships: Increased awareness of different elements of interaction in social media, including cyberbullying. Work: developed students’ ability to recognize their skills and how to communicate them in an online environment or via media outlets. Education: development of teaching materials that could be used in vocational schools by vocational school teachers or other professionals working with vocational school students. 3.3.1.4 Organisational/national context issues The interventions on media literacy, visual literacy, and online interaction, were conducted for vocational school students between November 16th and 27th, 2023. A concise two-hour session was attended by a small group of 58 participants, designed to fit conveniently into the school schedule. The brief workshop format not only accommodated participant availability but also streamlined the evaluation process and feedback collection, ensuring it was manageable within the allocated time. 3.3.1.5 Intervention enhancements/recommendations provided This intervention aimed at promoting multi-literacy in daily life involved a steering group with researchers from UH and FSM and vocational school teachers. Participation was optional, but incorporating the workshop into the curriculum could boost student motivation and engagement. The two-hour session included interactive activities to improve learning outcomes and skill development. To evaluate effectiveness, tools like evaluation forms and feedback surveys were used to collect insights on participants' experiences, satisfaction, and suggestions for improvement. For future extended workshops, ensuring data anonymisation is critical to address security concerns, particularly concerning the use of digital media by students. Enhanced data security can lead to higher response rates, more accurate feedback, and greater overall effectiveness of the intervention. 3.3.2.1 Digital well-being services skills pilot course for elderly people: Intervention description and purpose This section outlines a pilot course on digital well-being service skills for the elderly in Central Finland, organised by the University of the 3rd Age in Jyväskylä (U3A, open university). The course was part of a research and development project titled "Significance of digital support for older adults using digital technologies and services in the wellbeing services county of Central Finland," led by The Centre of Excellence of Ageing and Care (CoE AgeCare) at the 30 University of Jyväskylä. Collaboration included non-academic partners such as the Wellbeing services county of Central Finland, the Jyväskylä City elderly council, and The Finnish Association for the Welfare of Older Adults. The course was held on September 25 and 26, 2023, at the University of the 3rd Age in Jyväskylä (U3A), focusing on helping elderly individuals navigate key digital wellbeing services and understand information security. Each session, led by an experienced U3A teacher, ran for approximately 5.5 hours and repeated the same content on both days. This intervention specifically targeted elderly participants in Central Finland, addressing the new needs arising from the reorganisation of social and health services in the area. In 2023, the course was offered free due to the involvement of the University of Jyväskylä's CoE AgeCare project, which also prompted a new focus and increased research activities. Pre-registration was required, and participants were encouraged to bring their own digital devices, although U3A provided computers, internet access, note-taking tools, and support from elderly peer-tutors. The course was well-received, attracting nearly full attendance with 13 participants each day out of a maximum of 14. Most attendees were motivated by the desire to gain confidence in using digital health services and stay current with digital advancements. Many learned of the course through personal referrals from past participants. 3.3.2.2 Media literacy and digital skills targeted The course, tailored specifically for the elderly in Central Finland, focused on equipping participants with essential skills for using digital health services effectively in their daily lives. It covered key services, such as seniorchat, seniorisivut, hyvaks.fi, omaks.fi, and hyvis.fi. Additionally, the curriculum included important aspects of cybersecurity, teaching secure electronic authentication and how to adjust device security settings. This comprehensive approach aimed to enhance the digital competence and safety of elderly users navigating online health resources. 3.3.2.3 Intervention outputs and outcomes The course was designed to significantly enhance the daily lives of the elderly, focusing on enabling them to access necessary healthcare services efficiently and securely online. It introduced participants to specific health support tools and websites, such as a chat tool tailored for senior citizens. Additionally, the curriculum emphasised the importance of cybersecurity and privacy, teaching participants how to safely use these digital tools, ensuring their personal information remains protected while they navigate health services. The expected outcomes of the intervention are as follows: ● ● Information navigation & processing: The ability of elderly people to use digital services (websites and tools) and understand the cybersecurity aspects of using them. Health: Increasing the well-being of the elderly as they learn to find and use help from digital health services safely and securely. 31 ● Education: Developing the course content and teaching materials for future courses and forming recommendations for developing similar courses for elderly people in general. 3.3.2.4 Organisational/national context issues With the introduction of new welfare areas in Finland in 2023, which shifted the responsibility of social and health services from municipalities to these areas, there was a recognised need for a pilot course tailored to assist elderly people in using digital services. This course was organised as part of a development project and was systematically evaluated using various methods, including interviews with peer-tutors and participants, class supervision, and the REMEDIS surveys. Designed to accommodate a small group (14 participants per day), the course saw a total of 26 elderly attendees. This small cohort size, however, limited the potential for large-scale quantitative data collection. Additionally, the participants showed a preference for focusing on the course material rather than participating in research activities. This was further complicated by the wide variance in digital skill levels among the participants, affecting their engagement with the research components of the project. 3.3.2.5 Intervention enhancements/recommendations provided The development of the course was a collaborative effort involving CoE AgeCare project researchers, U3A course teachers, and key non-academic partners such as the Well-being services county of Central Finland, the Jyväskylä City elderly council, and The Finnish Association for the Welfare of Older Adults. A steering group formed from these expert organisations played a critical role in shaping the course content. Starting from scratch, the course planning addressed the newly introduced digital services by the Well-being services county, marking the first time such a course was offered. Based on positive feedback from previous courses, a section on digital security was incorporated, and other well-received elements from past digital courses for the elderly were also included. This approach ensured the course was both comprehensive and tailored to meet the specific needs of its elderly participants. 3.4 Poland 3.4.1.1 The Open-minded and Creative Teacher of the Future: Intervention description and purpose This intervention aims to enhance digital and media literacy among pre-service teachers by focusing on the creation of open educational resources (OER). It will shift their approach from passive to active use of new media, thereby improving their ability to develop modern ICTbased educational solutions. The project targets future teachers, identified as needing specialised support based on prior assessments of their digital and media skills. This aligns 32 with the Polish team's focus on teacher education and addresses the need to modernise education in response to post-pandemic challenges and experiences from the e-learning crisis. This activity, in collaboration with external partners, focuses on enhancing teaching and learning support through modern solutions. IH Bielsko-Biała, represented by Christopher Walker, aids in validating the theoretical framework and research reports' methodological and linguistic aspects. The Polish Society for Educational Technology and Media, with Natalia Walter participating, supports the intervention's implementation and the effectiveness evaluation. The intervention offered a free course to pedagogy students at the Jagiellonian University’s Institute of Pedagogy. It was held in a recently upgraded computer lab (including software updates and additional memory) to ensure a comfortable learning environment. The course used free university software, edtech tools, and video tutorial creation software like Flashback Express and OBS, aiming to keep costs minimal for both organisers and participants. The motivation for participating in the course stemmed from a needs analysis that highlighted a demand among prospective teachers to enhance their digital competences, specifically in creating open educational resources (OER). This interest is driven by the rapid digitisation of Polish schools and increased expectations on teachers. The course also aimed to modernise the curriculum at the Institute of Pedagogy to better engage students, acknowledging their diverse backgrounds. Motivation levels were continuously assessed through observations, direct communication, and evaluations of the OERs produced. 3.4.1.2 Media literacy and digital skills targeted The course designed to boost media literacy and digital skills was structured into three stages: Introduction to ICT: Focused on creating, editing, saving, and transferring data across different digital formats, adhering to the ECDL standard and using the office suite available. OER Use and production: Led by a media pedagogy specialist, this stage introduced free software like OBS and Flashback for creating, editing, archiving, and sharing OERs. Trainees also accessed YouTube tutorials on making video tutorials and communicating OERs effectively. Dissemination: The OERs developed by trainees were disseminated by the Polish REMEDIS project team. This approach is grounded in Fedeli's 2020 model, emphasising the enhancement of teacher digital competence through video-based OERs, with a focus on problem-based learning complemented by initial instructions. 3.4.1.3 Intervention outputs and outcomes The evaluation of achievement and commitment in the course occurred in several stages: Pre-test: Trainees initially took a universal pre-test as part of the REMEDIS project to assess their starting competencies. 33 Identifying benefits: During the second stage, trainees explored potential benefits from developing OER, enhancing their understanding and motivation. Post-test and engagement: In the final stage, participants completed a post-test to evaluate the individual benefits they gained from the course. Engagement was consistently monitored by a media pedagogy specialist, who also ensured the quality of the OERs produced. The course satisfaction was linked to self-reflection on trainees' readiness for teaching in a digitally evolving educational environment. Prospective teachers received feedback on their assignments and saw their high-quality OERs uploaded online and actively promoted. This hands-on problem-solving experience, often a first for many, significantly contributed to their satisfaction, particularly through the creation of valuable educational materials. 3.4.1.4 Organisational/national context issues The intervention is a key component in modernising teacher education for future generations (Tomczyk & Fedeli, 2022), emphasising the critical role of digital and media literacies as fundamental skills in the teaching profession. Their importance has become particularly pronounced in the post-pandemic landscape. This initiative aligns with the expectations of the digital-native student generation, meeting both educational challenges and societal demands for modernised, digitised education. As such, it is integral to higher education development strategies. Furthermore, the intervention facilitated the integration of new educational content into mandatory courses, allowing for practical validation of this content and its pedagogical approach. 3.4.2.1 Enhancing Digital Skills and Teaching for Carers of Older Adults: Intervention description and purpose The project was executed in partnership with several entities, including the University of the Third Age, which provided honorary patronage and helped reach the target audience. The OPEN UJ platform, part of the Jagiellonian University Teaching Support Center, made the training accessible online, requiring only account creation. Additional collaborators include the Krakow Institute for the Development of Education, which offers teacher training; the Media Education Center at Polish Radio Kielce, which supports media education goals; Super Senior Magazine, Poland’s first magazine for the senior community; and the Cluster Life Science Kraków Foundation, which facilitates a network in the scientific and business sectors. Participation in the course was free, requiring only account creation on the Open UJ or KIRE Moodle platforms. Focused on participant needs, the e-learning course was hosted on userfriendly platforms with robust technical support. The course design was guided by insights from REMEDIS analyses, ensuring it was effectively tailored to meet learners' requirements. The primary motivation for participating in the course stemmed from a lack of understanding about the digital divide and a desire to enhance skills in teaching seniors. Additionally, some participants viewed the course as a step in their career development, indicating a strong external motivation. To support this, a promotional campaign was launched in collaboration 34 with partners such as KIRE, UTW, CEM, LifeScience Klaster, and Super Senior Magazine. The initiative highlighted the need for institutions committed to digital inclusion to have professionally trained staff capable of planning, implementing, and assessing educational programs based on contemporary pedagogical approaches in geragogy, social gerontology, and andragogy. 3.4.2.2 Media literacy and digital skills targeted The course, designed for caregivers and trainers, encompasses content that extends beyond improving media literacy and digital skills. It aims to educate participants about the digital divide and the importance of inclusion. The curriculum is tailored to meet the specific needs of the target audience, providing insights into adult learning, challenges faced by older adults in learning, and their educational preferences. The course is structured into nine comprehensive parts: Digital Exclusion Digital Competences Specific Educational Needs of Seniors Supporting Older Adults' Learning Process Motivating Seniors Role of a Perfect Geragogue Guidelines for Educating Seniors in Digital Skills Educational Objectives and Content Internet Safety for Seniors 3.4.2.3 Intervention outputs and outcomes Platform statistics and lesson summaries reflect strong engagement. Participants in the REMEDIS project completed both a pre-test and a post-test, which included questions about their opinions of the course; responses were overwhelmingly positive. Participants suggested improvements and expressed satisfaction with the course content, requesting more short and instructional films. 3.4.2.4 Organisational/national context issues As global populations age, it is projected that by 2030, one in six people worldwide will be over 60. This demographic shift coincides with difficulties for older adults in integrating into the digital world. Amid rapid technological advances, ensuring digital inclusion for seniors becomes crucial. In Europe, Denmark, Sweden, Ireland, Luxembourg, Finland, the Netherlands, and Belgium excel in intensive internet use. However, countries like Bulgaria, Croatia, Greece, Italy, and Poland fall behind in digital engagement. For example, in Poland, only 25.6% of those aged 65-74 use voice or video calls online, 28.6% send and receive emails, 21.3% engage on social networking sites, and 22% make online purchases (GUS, 2023). Despite the opportunities presented by new media, Polish seniors struggle to enhance their quality of life through information and communication technologies. 35 The enhancement of digital skills among seniors is facilitated through intermediaries such as caregivers and trainers who collaborate with Universities of the Third Age and senior clubs. An e-learning course specifically designed for these intermediaries aims to support this effort. 3.4.3.1 Strengthening media competence of people at risk of digital exclusion: Intervention description and purpose The primary goal of the intervention is to enhance understanding of public media operations, promote critical analysis of media messages, and strengthen defenses against disinformation. Interactive workshops provide a secure environment where participants can discuss their online experiences, learn to identify fake news, distinguish between truth and falsehood, and acquire both basic and advanced information searching techniques. They will also learn to verify sources using appropriate tools. Participants will develop critical thinking skills and recognise emotional responses to disinformation, which will help them consciously counteract false content in the future. They will understand how emotions and personal experiences influence their perception of information, thus affecting their vulnerability to disinformation. The workshop incorporates experiential learning elements from the Kolb cycle. This initiative is a collaborative effort in the Malopolska region, spearheaded by the All About Science Foundation in Kraków, which brings extensive experience in organising educational events for diverse age groups. The foundation has provided honorary patronage and support in engaging the target audience. Additional support comes from the Universities of the Third Age in Trzebinia, Chrzanów, and Zabierzów, which offer free access to necessary facilities and equipment. Recruitment assistance is provided by the Chrzanów Municipal Public Library and the Trzebinia Scout Troop. The project leverages the resources of institutions affiliated with the Jagiellonian University, using local organisations specialising in non-formal education to provide spaces for in-person workshops. Following the BYOD (Bring Your Own Device) approach, as discussed by Santos (2020) and Aggarwal (2018), trainees will use their own smartphones to perform media literacy exercises that enhance their understanding of manipulation in new media. This includes exercises to classify sources as credible or misleading. BYOD is particularly useful in areas lacking computer access. Additionally, access to various media literacy websites, such as demagog.org.pl and Fajnie, że Wiesz, is crucial. These resources are integral to the training programme, offering essential support for reinforcing media literacy skills during the workshops. Participant motivation for the training is fostered through various strategies. Collaborating institutions like Universities of the Third Age and senior citizens' clubs inform and engage 36 seniors about the course. Given the rarity of such educational activities, this approach is expected to significantly motivate participants. Throughout the course, trainees learn the importance of strong media competencies and how these skills defend against misinformation and manipulation. Their motivation will be reinforced from the start by showcasing the comprehensive skills and knowledge they will acquire. Finally, at the course's end, participants receive an evaluation of their enhanced Digital Literacy and Media Literacy (DL&ML) skills, encouraging a positive shift in their digital information consumption habits. 3.4.3.2 Media literacy and digital skills targeted The aim of the training programme (intervention) is to strengthen media literacy in terms of resilience to manipulation and disinformation in the new media space. The training programme will include skills relating to: distinguishing fake from real information, strengthening the ability to search for information sources, knowing how to characterise information identified as fake news, knowing how to check the veracity of information on websites dedicated to this process, or improving knowledge of the positioning mechanisms of online content, including the clickbait phenomenon. 3.4.3.3 Intervention outputs and outcomes Engagement levels were evident during the workshops, which included integrated and activating exercises like discussions, individual and group tasks, and games. Participants also completed pre-tests and post-tests as part of the REMEDIS project. The effectiveness and satisfaction from the intervention are measured using these tests and through individual interviews. Conducted face-to-face, the intervention allows for immediate feedback and swift adjustments. The workshops aim to equip participants from disadvantaged groups with essential knowledge for protection against manipulation and disinformation in new media. Ultimately, the real enhancement of media skills is expected to significantly increase participant satisfaction. 3.4.3.4 Organisational/national context issues The Polish Government's website describes fake news as intentionally misleading, shocking, and controversial, noting the increasingly blurred line between minor errors and deliberate falsehoods. This highlights the importance of addressing fake news today. The Polish REMEDIS team's intervention responds to this pressing issue, targeting key vulnerable groups such as seniors, NEETs (those not in education, employment, or training), and immigrants, underscoring the project's relevance and urgency. 37 3.5 Spain 3.5.1.1 Cibermanagers por la Igualdad: Intervention description and purpose The "Cybermanagers for Equality" programme, coordinated by PantallasAmigas and funded by the Spanish Ministry of Equality, aims to combat gender-based violence in digital contexts. The programme fosters positive coexistence and promotes egalitarian behaviour among students. Implemented in various secondary schools across Spain, especially those facing high complexity due to socio-economic challenges, diverse student backgrounds, and specific educational needs, the programme prioritises schools with the greatest needs. Teachers are trained on digital coexistence risks and myths (e.g., sexting, aesthetic pressure, romantic love) to equip them to educate students. These students then create content and conduct training for younger peers and their families. The programme details the main objectives, participation motivations, necessary resources, involved agents, and outcomes. Challenges and recent improvements to enhance programme effectiveness are also discussed in subsequent sections. PantallasAmigas, a non-governmental organisation active in Spain since 2004, focuses on protecting and promoting the rights of children and adolescents in digital contexts. Its action is grounded in five pillars: educational communication, education in digital life skills, innovation, engagement in various educational aspects, and the promotion of universal values. Key activities include developing awareness campaigns, creating educational materials, providing consultancy to public bodies and businesses, and offering training for minors, teachers, families, and professionals. One of its key initiatives, the Cybermanagers programme launched in 2010, pioneered the use of peer learning and service learning in Spanish schools to foster digital citizenship and cyber coexistence. The Cybermanagers for Equality programme, an extension of this strategy, combines digital and face-to-face educational practices to promote egalitarian behaviour and positive coexistence in schools. Supported by the University of the Basque Country (UPV/EHU), which offers evidence-based improvements, the programme equips teachers with innovative training to address cyber violence. It empowers students to create their own content and disseminate their knowledge to peers and families, enhancing their digital literacy and societal engagement. Cybermanagers for Equality is a dynamic eight-week training programme for secondary school students (ages 13 to 16) that integrates service learning, project-based learning, and emotional design to foster coexistence and equality. It aligns with the Spain Digital Plan 2026 by enhancing key digital competencies through initial teacher training. Programme Structure: Project Presentation (Week 1): PantallasAmigas introduces the programme’s objectives, methodology, and benefits to the school's management team, who then present it to interested teachers for implementation during school hours. 38 Teacher Training (Weeks 1-4): Weeks 1-3: Teachers voluntarily choose and train autonomously in six tailored modules via a digital platform, dedicating 10 hours total. PantallasAmigas provides support for module selection and activity planning. Week 4: A 4-6 hour workshop, led by PantallasAmigas experts, helps teachers finalise their training, address doubts, and prepare methodologies. Student Training (Weeks 4-7): Weeks 4-5: Teachers pass on their knowledge to students, who volunteer for further involvement. Weeks 6-7: Volunteer students develop their own content under teacher guidance, preparing to educate younger peers and families. Action Plan (Week 8): Implementation and Evaluation: Volunteer students act as trainers, disseminating their content to the educational community. Teachers evaluate the programme's impact and suggest improvements to PantallasAmigas, aiming for sustainable programme continuation. This structured approach ensures that both teachers and students actively participate and contribute to a culture of digital equality and respectful coexistence within the school environment. The primary goal of the Cybermanagers for Equality programme is to address and prevent gender-based cyber violence affecting minors in Spain, particularly those from vulnerable backgrounds like migrants or those with low socioeconomic status. Implemented in secondary schools, which are seen as crucial venues for identifying and addressing such violence, the programme focuses on promoting equality and positive coexistence online. Annually renewed, this programme strives for sustainability and independence across schools, encouraging active student involvement. It transforms students into peer trainers and influencers using service learning, project-based learning, and emotional design. Essential to the programme is the prior training of teachers in digital competencies and conflict resolution, tailored to student needs. This training enables teachers to effectively mediate and communicate digitally, respecting and responding to the specific needs of their students. The Cybermanagers for Equality programme is offered free of charge, thanks to funding from the Spanish Ministry of Equality for its inaugural edition. This no-cost availability requires schools to commit to fostering positive digital coexistence, involving teacher training and active student participation. Teachers volunteer for this programme, undertaking expertguided, autonomous training focused on digital competence and resolving digital conflicts among students and families. Similarly, students volunteer as cybermanagers, engaging in training and content creation, and commit to educating their peers and families. 39 Both groups—teachers and students—are expected to have internet access and digital devices to use instructional materials and participate in online training sessions. It is also essential for teachers to use digital devices during sessions to demonstrate real-life online scenarios rather than just theoretical knowledge. Moreover, an annual commitment is required from schools, teachers, students, and families to attend cybermanager meetings to exchange experiences and insights with other educational communities across Spain. This connectivity is vital for maintaining effective communication and collaboration among all participating cybermanager schools. The motivation to participate in the Cybermanagers for Equality programme varies among different stakeholders, driven by concerns over the gap in adult understanding of minors' recreational internet use, particularly on social networks. Local Ggovernments: Initiate participation through education, equality, and social affairs departments, enabling municipal schools to join the programme. Schools and teachers: Schools join independently or through teacher advocacy. Teachers are motivated by the chance to enhance their digital skills and apply them to educate students on equality, respect, and positive coexistence. Active learning: Teachers appreciate the programme's active learning methodologies that allow a co-educational approach to address digital coexistence issues affecting students' learning. Access to resources: Easy access to training materials, professional advice, and support is crucial for teachers, helping them integrate the programme into their busy schedules and apply its methodologies effectively. Adaptability and sustainability: The programme's design to adapt to specific school needs in its first year and aim for autonomous operation in subsequent years motivates schools and teachers to engage initially and sustain participation. Non-academic validation: For students acting as cybermanagers, a key motivation is the opportunity to voice their concerns and share their knowledge on daily life issues not typically covered in formal education. They value the role of influencing and aiding their peers, enhancing their commitment to the program. Each of these motivations underscores the overarching goal of Cybermanagers for Equality: to bridge the digital knowledge gap and foster an environment of informed and respectful digital interaction. 3.5.1.2 Media literacy and digital skills targeted The Cybermanagers for Equality programme aligns with both the new Digital Competence Framework for Teachers in Spain and the European DigCompEdu framework. It also adheres to the educational competencies outlined in Spain's LOMLOE Educational Law, supporting the goals of the Spain Digital Plan 2026. The programme specifically enhances digital 40 competencies vital for the personal development of young people, focusing on the following areas: Technical and operational skills: Students enhance their abilities using digital platforms like Kahoot, Quiz, Canva, etc., to develop and share content with peers and families. Teachers learn to navigate new digital platforms to access training resources effectively. Navigation and information processing: Both teachers and students boost their skills in identifying, evaluating, and utilizing information, particularly information that influences gender equality and promotes positive digital coexistence and privacy rights. Creation and production of content: Central to the programme, students engage in project-based, service learning, and problembased learning to create digital content. This content is then shared with their peers and families, fostering knowledge transmission. Teachers facilitate and guide content creation, enhancing their digital skills through both formal training and collaborative learning with students. Communication and interaction techniques: Students are trained on effective digital communication methods (e.g., social networks, emails, chats) to foster positive coexistence and egalitarian behavior. They also learn to craft and present digital content effectively. Teachers and students use educational platforms for file sharing and communication, enhancing interaction within the educational setting. These focused training areas are designed to meet specific digital competencies that are critical in today's digital landscape, particularly those that address the intersection of technology with social issues like gender equality and privacy. 3.5.1.3 Intervention outputs and outcomes The Cybermanagers for Equality programme is designed to foster positive coexistence and address gender-based violence in digital spaces by developing digital competence among students and teachers. This initiative brings multiple benefits across the educational community: Increased digital confidence: Teachers and students enhance their digital skills, aligning with the objectives of the Spain Digital Plan 2026. Risk management: Participants develop strategies to identify and address potential risk behaviours related to gender inequality in digital contexts. Autonomous critical thinking: Students produce their own content, guided by teachers but encouraged to maintain creative independence. 41 Enhanced communication skills: Students learn to tailor their communications effectively to diverse audiences, including peers, families, and other educational communities. Promotion of well-being: The programme supports a positive and respectful coexistence within the school environment, benefiting teachers, students, and families involved. Educational integration: Many schools incorporate the Cybermanagers programme into their educational plans post-implementation, ensuring its long-term sustainability. Protection of minors’ rights: The programme helps participants understand and advocate for minors' digital rights, fostering respect and awareness. Active methodologies: The initiative emphasises collaborative and community service methodologies, enhancing traditional education with the active participation of students. School autonomy: The programme’s design allows schools to tailor content and methods to their specific needs, supporting sustainable implementation. Networking and collaboration: Participation in the Cybermanagers network offers opportunities for sharing experiences and building connections with educators from other schools. These components collectively aim to create a safer, more inclusive digital environment for all members of the school community. 3.5.1.4 Organisational/national context issues The Cybermanagers programme, with a proven track record since 2010 in over a hundred Spanish schools, has incorporated many of REMEDIS's recommended improvements based on literature reviews (Visenberg et al., 2023; Martínez et al., 2023). During the co-development phase, special emphasis was placed on creating an evaluation system to assess the program's impact on student well-being. Despite its established success, gauged by increasing school adoption and positive feedback, developing and implementing a tailored quantitative evaluation tool for cybermanager students, especially those from vulnerable backgrounds, presented challenges. Issues included students' difficulties with interpreting complex questions and the logistical challenges for teachers, who had to assist in completing these surveys during their limited available time, adding strain due to coordination demands across different centres. 3.5.1.5 Intervention enhancements/recommendations provided The Cybermanagers programme faces significant challenges in establishing a robust evaluation framework to systematically assess its broader impacts beyond digital skill enhancement and media literacy. Although the provider has gathered valuable observational data, there is resistance from school centres, primarily due to workload concerns, against allowing researchers to participate in these sessions. Additionally, there is a lack of quantitative tools to measure the programme's impact on students' digital well-being. Schools and teachers, possibly wary of being evaluated, show reluctance to engage in or facilitate the completion of questionnaires by students. This hesitancy complicates efforts to quantitatively assess the program's effectiveness. Moreover, there is no established method to systematically 42 verify whether the program achieves sustained implementation in school centers as intended, leaving a gap in evidence-based assurance of its effectiveness. Given these issues, it is recommended to develop systematic qualitative and quantitative data collection and evaluation tools. These tools are essential not only for measuring the programme's impact but also for demonstrating to school centres and participants that evaluation is a critical and beneficial aspect of the project. Furthermore, the involvement of experts and evidence-based interventions is crucial for enhancing the effectiveness of this programme, ensuring its continued improvement and success. 3.5.2.1 Desfake for education centres: Intervention description and purpose The Desfake project, managed by Verificat, focuses on combating misinformation through educational interventions aimed at enhancing critical thinking and information verification skills among students, particularly in Catalonia’s high-complexity schools. These schools often serve vulnerable populations, such as migrants and students from low socio-economic backgrounds in areas with high unemployment. Verificat, known for its fact-checking and data journalism, especially on political and scientific topics, initiated Desfake in response to the escalating misinformation problems that worsened during the COVID pandemic. Realising the need for a long-term solution, Verificat expanded into educational programmes to foster media literacy. Desfake operates through three primary models: - Direct student training by Desfake professionals, with some teacher involvement for sustainability. - "Teachers Desfake," where teachers receive 15-30 hours of training in digital skills and media literacy, often funded by educational centres or local councils, with certification that enhances their professional growth. - "Desfake Schools," the most common approach, focuses on whole-school engagement, funded by regional governments or local councils. It involves training 3-5 teachers per school who then integrate the learning into the school’s broader culture. Funding for Desfake often comes from local or regional governments, such as a public tender by the Catalonian Department of Education for digital competence training. This programme not only targets student development but also emphasises the role of teacher training centres, despite their limited budgets, in extending the program’s reach and impact. The main stakeholders in the Desfake programme include: Teachers: They are pivotal, receiving training in media literacy and digital skills (ML&DS) and are responsible for imparting this knowledge to their students. 43 Students: As active participants, students not only receive training but also engage in developing their own content, applying their new skills. Families: Recognised as crucial recipients of ML&DS education, engaging families effectively through school centres remains a challenge. Other key stakeholders involved in promoting and funding the programme are: Educational entities: Teacher Training Centres, town councils, and education departments of Autonomous Communities often support the initiative financially and programmatically. Third sector entities: Foundations and organisations focused on enhancing informational media literacy and fact-checking skills among educators and other community agents also play a significant role. These stakeholders collectively contribute to the programme’s reach and effectiveness in fostering educational change and enhancing digital literacy. The Desfake programme, driven by Verificat and Escuela21, focuses on enhancing Media Literacy and Digital Skills (ML&DS) through project-based workshops where secondary school students actively participate and apply their skills by tackling verification challenges. The programme collaborates with teachers to co-develop tailored interventions for each school, considering their unique needs and contexts. This approach has been piloted in 10 Catalan schools with plans for broader implementation next academic year. The programme trains teachers extensively, with a recent shift from 70 to 30 hours of focused training on digital transformation, emphasising practical application in school environments. The training is supported by an online platform like Moodle, providing continuous access to resources and online monitoring, requiring basic digital infrastructure which schools typically possess. Funding comes from the Catalan Department of Education and aims to co-finance schools that lack budgets for such programmes. Motivation for participation stems from a recognised need among educators to enhance their and their students’ information media literacy, critical for navigating the challenges posed by the digital information landscape. Teachers are also incentivised by career development opportunities, as training credits contribute towards public service advancements. The overarching goal is to empower teachers and, by extension, their students and families, to critically engage with digital content and combat misinformation effectively. 3.5.2.2 Media literacy and digital skills targeted The Desfake intervention is structured around key elements from the DigCompEdu framework, focusing on enhancing digital competencies in educators, particularly: 44 Information Navigation and Processing: Central to the programme, this involves teaching students to critically assess and manage information. Content Creation and Production: Students learn to produce and manage digital content, integrating Media Literacy skills to navigate digital media effectively. Communication and Interaction Techniques: The programme emphasises digital communication skills, including the appropriate use of social media, crafting formal emails for public agencies, and distinguishing between formal and informal communication contexts. Overall, Desfake prioritises developing a critical understanding and communication of information across various digital platforms and formats. The intervention also teaches students the nuances of communication codes on social networks and how to adapt their messaging for different relational contexts, whether formal or informal. 3.5.2.3 Intervention outcomes and outputs The Desfake intervention equips teachers to enhance their own and their students' media literacy and digital skills (ML&DS), focusing particularly on improving information navigation and processing capabilities. The intervention offers several benefits: Verification skills: Students improve their ability to verify information. Post-intervention evaluations require them to assess the veracity of various cases, demonstrating any enhancements in their information evaluation skills. Critical thinking: The programme significantly boosts students' critical thinking abilities, enabling them to critically analyse information across various contexts, not just on digital platforms. Socio-emotional education: It integrates socio-emotional learning by helping students recognize their own beliefs and judgments. This fosters a critical dialogue among peers in the classroom, influencing a shift in attitude towards more thoughtful engagement with content. Transversal relevance: Media and Information Literacy are woven through various school subjects and situations, highlighting its importance to the entire educational community. Focus on vulnerable students: Desfake prioritises work in highly complex schools, addressing the unique challenges faced by vulnerable students. It encourages these students to understand and articulate their identities in a manner that avoids stigmatisation and judgment. Empathy and Socratic method: The intervention encourages teachers to adopt a Socratic approach, promoting empathy and a deeper understanding of different perspectives, essential for fostering an inclusive and reflective classroom environment. 3.5.2.4 Organisational/national context issues After the pilot phase of the Desfake project, the goal is to foster autonomy among participating schools, enabling them to continue the programme independently in the following year. While Desfake will likely offer ongoing support and advice, a key challenge for the provider is determining the optimal point at which a school is ready to self-sustainably 45 manage the implementation of the programme. This involves assessing the school's ability to continue applying the learned principles and methodologies without direct oversight. 3.5.2.5 Intervention enhancements/recommendations provided Even though the provider has already started to design and to implement different evaluation methods, they would need to develop an appropriate measurement instrument in order to assess the process of pre and post intervention, data collection in a systematic way. In this sense, the provider addresses that the collaboration with REMEDIS- could be very useful for this purpose as far as the evaluation methods are evidence-based. Evaluating and measuring the effectiveness of the Desfake intervention presents a significant challenge for the provider. To this end, they have developed and implemented quantitative pre and post questionnaires designed to gauge students' prior knowledge and their ability to discern the trustworthiness of various information sources and digital content. After completing Capsule 2 of the training, teachers distribute these questionnaires to assess students' understanding based on specific questions. A second questionnaire follows to measure the learning outcomes. These results are anonymously used by Verificat to gauge the educational impact of the content. Despite the utility of these questionnaires in measuring students' development of critical digital skills, the provider, Verificat, supported by Escuela21, is concerned about the lack of a comprehensive evaluation tool that can assess broader impacts on participants’ wellbeing, such as identity, inclusion, and digital knowledge. In addition to quantitative methods, Verificat has undertaken significant qualitative work, including observations and focus groups with students and teachers across various Desfake centres. However, there is a recognised need to develop a more robust measurement instrument to systematically collect and analyse pre and post-intervention data. The collaboration with REMEDIS could be instrumental in achieving this, as it would bring evidence-based evaluation methods to refine and enhance the assessment processes. 3.6 United Kingdom 3.6.1.1 Digital Champions - Citizens Online: Intervention description and purpose Citizens Online (CO) targets both long-term organisational and project-specific objectives to promote digital inclusion. Their core mission is to enhance people's digital skills, access, and confidence across various dimensions. Each project, such as Digital Gwynedd, is tailored to specific goals like job-seeking skills, set in collaboration with funders who may focus on particular outcomes. CO emphasises a lifelong learning model, preparing users to continually adapt and maintain their digital skills over time. They recognise the recurring need for support, drawing on past experiences where sustainability in digital engagement has been a challenge. The intervention design heavily relies on a personalised approach, highly valued by beneficiaries for its adaptability and ongoing support from digital champions. This flexibility is 46 crucial as the need for digital connectivity often emerges through an iterative discovery process. Stakeholders include learners, intermediaries, funders, and digital champions, with each group playing a distinct role. Intermediaries refer users to support services, while funders, such as local councils or NHS trusts, finance these interventions. Digital champions, often local volunteers familiar with related charitable work, are essential for on-the-ground implementation, receiving training from CO but not specific digital skills during their induction. Despite a slight dip in volunteer numbers, CO continues to expand its impact through a digital inclusion network, fostering collaborations with charities, councils, and the NHS to enhance referral processes and resource sharing. This network also hosts monthly sessions to exchange field experiences, reinforcing CO’s commitment to leveraging collective expertise for broader digital inclusion. In the North Wales project managed by Citizens Online (CO), participants must be of working age, a requirement not stipulated in other projects. Access to training or devices is gained through referrals, and trainees select the type of support needed, such as devices or digital skills training. Originally, devices were loaned but due to the risk of excluding individuals upon return, this practice was discontinued. Now, CO provides recycled devices that are functional and updatable. Despite the absence of specific data, some trainees transition into volunteers post-training. CO's projects rely on external funding, leading to some projects closing and discontinuing support. However, it has been noted that some digital champions continue assisting trainees independently after project conclusion. CO uses a Kanban dashboard to organise third-party training resources by topic, streamlining access for digital champions without the need to develop new content. While detailed motivations of trainees are not comprehensively tracked, experience shows that many seek help for specific online tasks like paying council tax, connecting with friends and family, or online shopping. Initially unaware of the broader benefits of internet use, trainees often discover new online possibilities through training, which is tailored to their emerging needs and perceived opportunities. Occasionally, CO creates specific videos and content for certain contracts and organisations, but generally, no regular content for digital literacy support is developed. Volunteers are encouraged to discuss any topics with trainees, focusing on building confidence and safety online to minimise risks and online harm. 3.6.1.2 Media literacy and digital skills targeted Citizens Online (CO) focuses on informal training, making it challenging to categorise the specific digital skills covered in each session. Rather than developing bespoke content for media literacy and digital skills (ML&DS), CO leverages existing resources and facilitates access for their trainers. Digital champions, lacking specific guidance, can seek support from the project manager as needed. CO expects these volunteers to confidently address trainees' queries, regardless of their initial familiarity with the topics. The training provided blends digital skills with media literacy to enhance participants' confidence and problem-solving 47 abilities. The adaptability of the training varies with the project and is sometimes customised to meet funders' requirements, while the regular training for CO volunteers follows a more standardised format. 3.6.1.3 Intervention outputs and outcomes Based on questionnaires applied to the Digital Gwynedd project in 2022, meaningful changes in participants’ confidence are observed in the aspects consulted. Those are, using communication tools, avoiding online scams, using Internet to find information, and using Internet to shop, bank or access services (see Figures 1 to 4). Figures 1-4: Self-reported confidence levels before and after support 48 One key expected outcome for Citizens Online (CO) is enhancing information navigation and processing skills, supporting their commitment to lifelong learning and digital inclusion. This involves empowering communities to use digital tools for learning and connecting with family and friends. For specific projects, improving employment prospects and engagement is often an essential goal set by funders, though these outcomes may not be immediately quantifiable. Regarding evaluation improvements, there are opportunities to enhance how learning is captured and used. Currently, lessons are often learned informally, but there is a need to formalise data collection processes to better inform organisational practices for future 49 projects. Additionally, developing methods to understand why participants drop out could provide valuable insights into improving engagement and retention strategies. 3.6.1.4 Organisational/national context issues The organisation has recently experienced increasing difficulties in recruiting volunteers (Digital Champions), in part, due to the cost of living crisis in the UK, which has negatively affected those organisations that depend on volunteers to run their interventions. Furthermore, given the nature of the short, one-to-one support of Digital Champions, collecting data to evaluate impact is challenging, provided that comprehensive questionnaires could not be applied with participants. 3.6.1.5 Intervention enhancements /recommendations provided As mentioned above, two potential enhancements could be implemented in the short term. On the one hand, there is difficulty in providing evidence of the project’s contribution to people getting jobs, provided that this can be regarded as a long-term goal that requires other skills to be fulfilled. Nonetheless, Citizens Online could measure and identify what skills have been developed as intermediate achievements that may support job seeking. In that sense, the REMEDIS project is willing to support the implementation of changes to their extant instruments in order to identify and measure the mid-term goals set. On the other hand, additional follow-up protocols can be elaborated to understand dropouts. Provided that CO does not follow a specific termed curriculum, the fact that people do not return frequently does not necessarily mean that the support provided was bad. Instead, their concerns could be addressed in fewer sessions and they gained the confidence to continue the learning process by themselves. Unfortunately, none of these possibilities can be supported at the moment because of a lack of data. 3.6.2.1 Code Your Future: Intervention description and purpose The primary goal of the course offered by CYF is to enhance adults' confidence in using online platforms, focusing on everyday digital activities within the UK. While many adults possess basic online skills, the course aims to boost their confidence and proficiency. Face-to-face training is a core strategy, helping overcome connectivity barriers and reducing digital exclusion. Key challenges include defining clearer objectives and collecting data to assess the program's impact. Future plans involve extending long-term support through connections with other digital skill platforms, even non-partners, to provide continuous learning opportunities. The course also seeks alignment with European standards, particularly the Digital Competence Framework (DigComp), to enhance its credibility and secure further funding. 50 Stakeholders are primarily funders, delivery partners, and referral organisations, with little interference in content delivery. Primary funding has been provided by Santander for three years, with CYF now seeking new funding sources. Collaboration with organisations like the IRC helps in securing attendees and venues, while companies like Capgemini contribute venues and volunteers. Beneficiaries are encouraged to use their own devices for assignments, with plans to facilitate device upgrades through partnerships. A completion certificate is offered, though completion rates are low due to the intimidating nature of homework requirements. Financial support for transport is available, but the reimbursement process can be challenging. Trainees primarily enrol to enhance their digital independence, stay updated with digital trends, and prepare for online job applications. The course is particularly proactive with refugees, focusing on practical skills for job hunting and community integration. Cybersecurity, especially in banking, is a key interest area for attendees, while Internet addiction currently remains a lesser concern. 3.6.2.2 Media literacy and digital skills targeted In broad terms, digital skills covered in the course are foundational in the following categories: • Technical and operational. • Information navigation and processing. • Communication and interaction. There is no content focused on media literacy in the course. However, some related topics are briefly discussed—for example, social media and examples of how anyone could post anything or misinformation and fake news. It must be noted that all this content is aimed at people at a foundational level and the course is three sessions long. 3.6.2.3 Intervention outputs and outcomes When it comes to the specific engagements expected in the short term, CYF expects participants to improve their Info navigation processing behaviour. For example, enhancing their web browsing and using different search engines and keywords. Translation was particularly valuable for a specific target group of migrants whose native language is not English. In their financial activities, they are expected to better understand online banking and feel more confident using banking apps. In terms of improvement opportunities, data collection and analysis have not been a priority during the course implementation, so this would be important in order to understand how to improve the current delivery. The digital skills course by CYF focuses on boosting participants' confidence in navigating online environments. Recognising the challenges in measuring long-term outcomes from a three-week intervention, CYF aims for participants to overcome isolation by enhancing their 51 digital skills. This goal encompasses broad digital inclusion, enabling participants to engage not only in social media but in all activities where internet use is beneficial. 3.6.2.4 Organisational/national context issues CYF initially launched the digital literacy course in order to facilitate users at low digital skills levels to engage in the main training offer, which is related to software development. However, the offer of further training is on hold and the current demand for the digital literacy course is not related to software development. The funding for the programme is about to end this year, so this is likely to become an additional difficulty not experienced until now, and the timing might not be beneficial since there is a perception that many organisations are competing for reduced funds on digital literacy. 3.6.2.5 Intervention enhancements/recommendations provided CYF has expressed the need to create impact measurement instruments in order to define continuous improvement strategies and communicate their achievements to stakeholders, especially potential funders. Furthermore, they are interested in aligning the DLP curriculum with the DigComp framework, so future expansions are drawn upon general skills rather than context-specific apps or services. Bearing that in mind, the REMEDIS project has recommended that CYF review and redesign the content, curriculum and evaluation instruments to measure the impact on specific skills and how such skills are contributing to enhancing engagement. The REMEDIS team advises the process. 3.6.3.1 Learn My Way (Good Things Foundation): Intervention description and purpose The Good Things Foundation (GTF) is dedicated to reducing digital exclusion in the UK by providing devices, data, and digital skills training through its Device Bank, Data Bank, and the Learn My Way learning platform. These resources are distributed to members of the Digital Inclusion Network, which includes various organisations like the UK Government's Community Renewal Fund, combined authorities, care charities, and councils that manage library access points. The primary recipients of devices and data are local organisations that serve broader purposes beyond digital inclusion, integrating these tools as supplementary to their main services. Only a few organisations focus exclusively on digital skills training. Generally, users access the Learn My Way platform incidentally while seeking other services. GTF supports these local hubs through a panel of ten ambassadors who facilitate product development and disseminate key messages. They also oversee the National Digital Inclusion Network, hosting events to gather live feedback from hubs on their service utilisation. There are no prerequisites for using the online learning platform or receiving devices and data, which are typically provided without conditions. However, local hubs assess additional needs and manage registrations. In some cases, hubs proactively identify and approach those most in need of devices. No incentives are directly offered for using GTF services or completing 52 courses. Occasionally, devices are loaned and given to users after training completion. Most users engage with hubs due to urgent non-digital needs, such as housing or employment support, making GTF's offerings a vital supplementary service. 3.6.3.2 Media literacy and digital skills targeted The offer on digital skills is related to the Learn My Way platform. There are currently 11 courses, as shown in Figure 5. Figure 5. Learn My Way course list snapshot Technical and operational, Information navigation and processing, and Communication and interaction skills are found in the courses provided. Most of them focus on the foundational levels of these skills and are related to specific outcomes, such as finances, leisure, and health. In addition, some content creation skills are also covered at a basic level. As it can be assumed in the course titles, the courses are guided by the intended outcomes rather than the skills. 3.6.3.3 Intervention output and outcomes The expected achievements can be inferred from the course titles and descriptions. They can be related as follows: ● ● ● ● ● ● Info navigation and processing: Starting to use the Internet, using email, and safety and security online. Education: Working with office programmes. Work: Employment and work, working with office programmes. Health: managing your health online. Information relations: Staying in touch. Finance: Managing your money online. The work of GTF outcomes and evaluation is based on six pillars: Digitally Able, Equal, Safe, Happier, Healthier and Better Off. They are related to the activities and outputs in Figure 6. 53 Figure 6. The Good Things Foundation impact and outcomes framework. (Source: The Good Things Foundation The REMEDIS framework identifies several core pillars connected to specific outcomes in the Good Things Foundation's (GTF) initiatives: Happy: Linked to combating loneliness and isolation by fostering informal relationships. Digitally Able, Equal, and Safe: Associated with enhancing skills in information navigation and processing. Better Off: Connected to improvements in education, work, and financial stability. To evaluate these outcomes, GTF uses simple binary questions in their questionnaires, aiming to boost the typically low feedback rates (around 5% for both baseline and post-intervention responses). The goal of this data collection is to gather practical evidence of the projects' impacts rather than academic rigor. Preliminary results indicate that half of the platform users report feeling happier, healthier, and better off. The primary focus is on assessing short-term impacts due to the challenges of measuring longterm objectives like employment or educational enrolment, which require more extensive interventions. Integrating the data obtained into ongoing project improvement remains a challenge, especially since demographic data collection is robust on Learn My Way but more limited for the device and data banks. 3.6.3.4 Organisational/national context issues The Good Things Foundation (GTF) is a large organisation that groups three programmes on access and digital skills. Nonetheless, they do not provide direct services to the beneficiaries targeted in the REMEDIS project, but this is done via their Digital Inclusion hubs. These are the organisations that act locally, using the resources provided by GTF, sometimes along with other support not related to digital skills training. 54 3.6.3.5 Intervention enhancements/recommendations provided The REMEDIS project has offered to participate in the discussion rounds to be carried out in incoming months with the digital inclusion hubs in order to understand how the current offer could be enhanced based on their outcomes framework. In addition, the GTF collects a high volume of data from different sources, but such data have not been entirely assessed in light of providing input to improve their services. In that sense, the REMEDIS project has suggested to characterise all data collected and establish how such data informs the theoretical framework provided. This could lead to define protocols of data management, identify what data are missing and what other data already collected might not be relevant for decisionmaking processes. 4 Questionnaire development 4.1 Description of the different modules (A-D) and how the questions were set up This section on questionnaire development outlines the structure and content of the survey designed to assess the effectiveness of our interventions. It details four modules (A-D) that collect data from various angles, ranging from socio-demographic information to digital skills and post-intervention outcomes. The questionnaire draws on successful elements from previous projects, ensuring relevance and accuracy in data gathering. A – Sociodemographic information This module gathers basic background information about participants, including: Age Gender Education Occupation Self-confidence (some items may be omitted depending on the intervention’s focus; education categories are tailored to each country’s framework; self-confidence items are optional and context-dependent). B – Digital lived environment This module assesses participants' access to technology and their digital competencies across seven areas: Device access Technical and operational skills Information navigation and processing skills Communication and interaction skills Content creation skills Media Literacy Digital knowledge 55 Questions range from device usage to evaluating statements about their digital skills and media literacy. C – Uses and outcomes (Pre-intervention) Part C refers to participants' internet use and the outcomes they perceive, focusing on: Learning Well-being Occupation (optional) Finance Social relations Each dimension features two questions: one on internet activities and satisfaction with the results, and another on negative experiences such as online scams. D – Uses and outcomes (Post-intervention) Mirroring Part C, this module is used post-intervention to assess changes in internet use and ability to avoid negative outcomes. It consists of 12 items, with 8 fixed and 4 optional based on the relevance to the intervention and the participants' context. By systematically collecting data through these modules, the questionnaire aims to provide comprehensive insights into the interventions' impacts on participants' digital engagement and overall life context. 4.2 Validation of the instrument Parts A and B have been externally validated. They have been used previously in the “youth Digital Skills Indicator (ySKILLS)” project, except for the media literacy question, which was not validated and it was the result of a synthesis of several media literacy scales explored in the literature. Questions in Parts C and D have been validated in the project “Networked Effects of Digital Inequalities”, but the wording was changed and the scales were adapted to the purpose of this project. Additional validation was made to the whole instrument by carrying out cognitive interviews with beneficiaries of ML&DS interventions in the UK. It must be noted that this process was simultaneous to the application of the questionnaire in other interventions, so the final version of the instrument was not applied in all cases. 4.3 Data collection scenarios Best case scenario This is the ideal scenario. It will consist of the pre-test with parts A, B and C. That is, it should be applied just before the intervention starts. Afterwards, a post-intervention instrument is to be used once the intervention is complete. This instrument is initially composed of part D but may also need to include part A when preintervention responses cannot be matched with post-intervention entries. 56 The follow-up instrument, which is similar to the pre-intervention one, should be applied after a month in order to identify how skills, literacy, use, and perceived outcomes have changed over time. Similarly, part A (questions 1,2,3) should be added when responses cannot be matched to specific individuals. Desirable scenario This is the plan for most interventions. It will have parts A, B and C for the pre-test and the post-test alike. The post-test would be applied with at least a month in between the intervention and the post intervention questionnaire. The post intervention questionnaire could be completed just before a second round in the intervention (for example, the second training course) Expected version Some interventions that may face resource limitations to apply the complete post-intervention instrument may opt to use a shorter version. It will consist of the pre-test with parts A, B and C and part D just after the intervention is complete to prevent dropout, especially if the partners are likely to lose contact with participants afterwards. This instrument is composed of part D but may also need to include part A (questions 1,2,3) when pre-intervention responses cannot be matched with post-intervention entries part B can be included if there is a desire to measure impact on skill levels before and after the intervention. 57 Minimum version One-off interventions and those with very limited time available for the evaluation could make use of a reduced application by omitting parts C in the pre-test and B in the post-test, respectively. Part A ought to be applied in both times if matching is not ensured. In other words, the pre-test instrument would comprise parts A and B only, whereas the post-test would be part D only and administered immediately after the intervention. With some exceptions where participants need more assistance, the application of either instrument is expected to take no more than 10 minutes in the latter cases, and the reduced post-test is aimed to take 5 minutes maximum, so dropout rates due to questionnaire fatigue can be minimised. 4.4 Development of the analysis strategy The quantitative data analysis strategy primarily focuses on statistical evaluation of responses gathered from the questionnaire modules. This will involve descriptive statistics to outline general trends and characteristics, as well as inferential statistics to test hypotheses related to digital skills acquisition, use outcomes, and socio-demographic impacts. Given the small sample sizes which are common in targeted interventions, there are inherent limitations that must be acknowledged. Small samples can reduce the statistical power of the analysis, which means detecting true effects or differences becomes less likely. This limitation also restricts the complexity of models that can be reliably used, as overfitting becomes a significant risk with more variables than data points. To mitigate these issues, analysis will focus on robust methods such as bootstrapping for estimating confidence intervals and effect sizes, which are less sensitive to sample size. 58 Furthermore, small samples can provide opportunities for detailed case studies or pilot testing of innovative practices, which could lead to valuable insights and guide future research or larger-scale data collection strategies. The qualitative data analysis will involve thematic analysis to identify and interpret patterns within the data that emerge from open-ended questionnaire responses, interviews, or focus groups. This approach will allow for a deep understanding of the participants' experiences, perceptions, and the subjective impacts of the digital skills interventions. Data will be systematically coded to categorise themes related to key topics such as barriers to digital access, perceived benefits of digital literacy, and personal narratives of change and challenges. The analysis will also consider the context and complexities of responses to understand the nuances behind participants’ interactions with digital technologies. Given the qualitative nature of this data, analysis will not just aim to quantify responses but to provide a narrative that reflects the varied experiences of participants. This narrative can offer insights into the effectiveness of interventions, highlight areas for improvement, and suggest mechanisms through which digital skills training impacts participants. In both quantitative and qualitative analyses, the strategy will ensure that findings are reported transparently, discussing the implications of the data comprehensively while acknowledging any biases or methodological constraints. This dual approach ensures a holistic understanding of the impact of the interventions, providing a robust foundation for making informed decisions about future directions and improvements. The forthcoming REMEDIS reports (D3.1 and D3.2) will address the analyses of each intervention in the various countries and on the integration of comparable questions in the pre- and post-intervention stages. 5 Conclusion In conclusion, this comprehensive report has elaborated on the intricate processes and methodologies employed in the co-development of intervention programmes across various REMEDIS focal countries, utilising the innovative REMEDIS Canvas framework. This framework has served as a cornerstone for the structured planning, refinement, and adaptive execution of each intervention, tailored to the nuanced contextual challenges of diverse geographical and cultural landscapes. Throughout the project, the initial phase of problem definition and context analysis was critical. By understanding the specific challenges and dynamics of each setting, the interventions were strategically aligned with the needs and socio-cultural nuances of the target populations. The REMEDIS Canvas facilitated a systematic approach to defining clear objectives and crafting strategies that are responsive to these detailed analyses. 59 Central to the success of these interventions was the engagement of stakeholders. This report highlights how the inclusive approach of co-creation and continuous feedback through collaborative workshops and consultations fostered a sense of ownership and commitment among stakeholders. The Canvas framework supported these efforts by providing a structured pathway for discussion, ensuring that every perspective was considered in the consensusbuilding process. The design and implementation of the interventions were informed by a robust engagement process and a thorough analysis of contextual factors. Adaptability was a key feature of the implementation phase, allowing programme strategies to be refined in response to real-time feedback and evolving circumstances. This flexible approach ensured that the interventions remained relevant and effective throughout their deployment. Monitoring and evaluation were embedded as integral components of the intervention process, underpinned by the systematic guidance of the REMEDIS Canvas. This setup not only allowed for the ongoing assessment of intervention effectiveness but also facilitated the necessary adjustments to enhance programme outcomes continuously. Feedback loops played a crucial role in this phase, enabling the capture of detailed insights from all stakeholders involved, which in turn helped refine the interventions further. However, the execution of these programmes was not without challenges. Feedback during the co-development stages frequently pointed to various organisational and national context issues, such as resource limitations and cultural sensitivities. These challenges required a proactive and strategic response, ensuring that interventions could be successfully implemented despite these hurdles. The formulation of data collection strategies also encountered significant limitations, particularly with regard to session attendance and participant availability. This affected the breadth of data coverage, prompting a greater reliance on qualitative insights to complement the quantitative data, thereby ensuring a more comprehensive understanding of the intervention impacts. The questionnaire development was a meticulous process, designed to gather relevant data across various domains—from socio-demographic information to digital skills and intervention outcomes. The structured application of this questionnaire in different scenarios—ranging from ideal to minimal—highlighted the adaptability of our data collection approach, accommodating different resource availabilities and intervention settings. In synthesising both quantitative and qualitative data, our analysis (in REMEDIS reports D3.1 and D3.2) will provide a dual perspective on the effectiveness of the interventions. While quantitative data offers a statistical evaluation of outcomes, qualitative insights reveal the nuanced experiences and perceptions of participants, providing a deeper understanding of the interventions' impacts. 60 This report not only underscores the efficacy of the REMEDIS Canvas in navigating the complexities of intervention development and implementation but also sets a precedent for future projects. It offers critical insights and strategic recommendations that will undoubtedly enrich the design and execution of similar initiatives moving forward. As we continue to refine our methods and strategies, the lessons learned from this extensive analysis will guide our efforts to enhance the effectiveness and reach of intervention programmes tailored to meet the diverse needs of communities worldwide. 61 Acknowledgements The authors are grateful for the generous collaboration of their colleagues in REMEDIS, and for the critical suggestions that helped improve this report. 62 References Joint Research Centre. (n.d.). DigCompEdu framework. European Commission. Retrieved from https://joint-research-centre.ec.europa.eu/digcompedu/digcompedu-framework_en. Vissenberg, J., Puusepp, M., Edisherashvili, N., Tomczyk, L., Opozda-Suder, S., Sepielak, D., Hietajärvi, L., Maksniemi, E., Pedaste, M., & d'Haenens, L. (2023). Report on the Results of a Systematic Review of the Individual and Social Differentiating Factors and Outcomes of Media Literacy and Digital Skills. KU Leuven: REMEDIS. https://doi.org/10.5281/zenodo.10356744 GUS (2023). Rozpoznawanie nieprawdziwych informacji—Baza wiedzy—Portal Gov.pl. Baza wiedzy. Retrieved April 27, 2024, from https://www.gov.pl/web/baza-wiedzy/rozpoznawanienieprawdziwych-informacji Tomczyk, Ł., Fedeli, L. (2022). Introduction—On the Need for Research on the Digital Literacy of Current and Future Teachers. In: Tomczyk, Ł., Fedeli, L. (eds) Digital Literacy for Teachers. Lecture Notes in Educational Technology. Springer, Singapore. https://doi.org/10.1007/978981-19-1738-7_1 Tomczyk, Ł., Mascia, M. L., & Guillen-Gamez, F. D. (2023). Video Tutorials in Teacher Education: Benefits, Difficulties, and Key Knowledge and Skills. Education Sciences, 13(9), 951. https://doi.org/10.3390/educsci13090951 Fedeli, L. (2020). Participatory video and digital citizenship: a case-study within an instructional technology course for social educators. Journal of E-Learning and Knowledge Society, 16(2), 8794. https://doi.org/10.20368/1971-8829/1135211 Santos, I. M. (2020). BYOD in the Classroom, Opportunities, Issues, and Policies. In Encyclopedia of Education and Information Technologies (pp. 267-272). Cham: Springer International Publishing. Aggarwal, D. (2018). Using the technology acceptance model to understand the use of bring your own device (BYOD) to Classroom. Journal on Today's Ideas-Tomorrow's Technologies, 6(2), 83-91. 63 Appendix 1: Recommendations shared with non-academic partners/intervention providers based on the Systematic Literature review (ESTONIA). Participant Engagement and Needs Assessment v Plan Engaging Activities: 1. Activate participants by designing dynamic and immersive learning experiences that encourage active engagement. 2. Identify the individual needs of participants and, whenever feasible, structure tasks to facilitate collaboration among participants who share similar needs, promoting effective discussion and joint participation in activities. 3. Consider participants' digital and media literacy skills when planning activities, utilizing a brief questionnaire for assessment. v Collecting Participant Information: 4. Carefully consider which characteristics of the participants are crucial for achieving the activity's objectives most effectively. Additionally, strategize on how to gather this essential information about participants. 5. Collect data about participants to understand them better and tailor learning opportunities to their zone of proximal development, ensuring topics and activities are neither too simple nor too complex. 6. Create homogeneous participant groups by carefully selecting binding commonalities. Devise strategies for effectively collecting the required information to inform group formation. v Group Dynamics and Trust Building: 7. Offer participants opportunities for idea exchange, collaborative work, and peer learning, both during sessions and between sessions when additional support is necessary. 8. Divide participants into smaller groups where mutual trust can be cultivated. In such intimate settings with shared interests, individuals are more inclined to openly share thoughts and learn from one another. Activity Planning and Evaluation v Duration and Intensity of Intervention: 64 9. Determine the appropriate duration and intensity of the intervention to achieve desired changes effectively. v External Evaluation and Expert Involvement: 10. To assess the effectiveness of the intervention and prevent recurring failures, engage trustworthy and impartial external experts in the evaluation process. These experts should utilize reliable and recognized evaluation methods. 11. Employ both pre- and post-assessments, along with observations, to comprehensively evaluate the effectiveness of the intervention and measure its impact on participants. Facilitator and Training Considerations v Expert Facilitators: 12. Engage experts as facilitators to lead the intervention, ensuring they possess essential expertise in content knowledge, pedagogical skills, and theoretical understanding. Facilitators should undergo training as necessary to optimize their effectiveness in guiding intervention activities. v Guidance in Activity Planning: 13. Throughout the training, provide guidance to participants in structuring their activities using appropriate theoretical frameworks, such as ICAP, the digital competence model, or self-determination theory Practical Implementation and Support v Independent Practice Opportunities: 14. Create opportunities for participants to practice independently outside of formal learning sessions. v Consideration of Time Constraints: 15. When designing activities, take into account the availability of participants in terms of scheduling (dates, duration of activities) and consider any other commitments they may have. v Sufficient Practical Time and Personal Guidance: 16. Allocate enough time for participants to practically engage in activities and receive personal guidance. Goal Setting and Autonomy v Setting Personal Goals: 65 17. Encourage participants to set personal goals at the beginning of the training to enhance engagement and motivation. v Perceived Autonomy of Participants: 18. Ensure participants perceive autonomy in their learning process to avoid fatigue and boredom. 19. Think through that each activity would be meaningful and engaging for the participants. Technical Accessibility v Addressing Technical Obstacles: 20. Ensure participants have no technical obstacles related to planned activities, including access to tools and software. 66