Organisation for Economic Co-operation and Development

EDU/WKP(2019)10

Unclassified English text only

7 May 2019
DIRECTORATE FOR EDUCATION AND SKILLS

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

OECD Education Working Paper No. 202

By Adrien Lorenceau, Camille Marec and Tarek Mostafa (OECD)

This working paper has been authorised by Andreas Schleicher, Director of the Directorate for
Education and Skills, OECD.

Adrien Lorenceau (adrien.lorenceau@gmail.com), Camille Marec (camille.marec@oecd.org) and

Tarek Mostafa (tarek.mostafa@oecd.org)

JT03447189

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the
delimitation of international frontiers and boundaries and to the name of any territory, city or area.
2 │ EDU/WKP(2019)10

OECD EDUCATION WORKING PAPERS SERIES

OECD Working Papers should not be reported as representing the official views of the
OECD or of its member countries. The opinions expressed and arguments employed herein
are those of the author(s).
Working Papers describe preliminary results or research in progress by the author(s) and
are published to stimulate discussion on a broad range of issues on which the OECD works.
Comments on Working Papers are welcome, and may be sent to the Directorate for
Education and Skills, OECD, 2 rue André-Pascal, 75775 Paris Cedex 16, France.
This document, as well as any data and map included herein, are without prejudice to the
status of or sovereignty over any territory, to the delimitation of international frontiers and
boundaries and to the name of any territory, city or area.
Note by Turkey: The information in this document with reference to “Cyprus” relates to
the southern part of the Island. There is no single authority representing both Turkish and
Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern
Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the
United Nations, Turkey shall preserve its position concerning the “Cyprus issue”.
Note by all the European Union Member States of the OECD and the European Union: The
Republic of Cyprus is recognised by all members of the United Nations with the exception
of Turkey. The information in this document relates to the area under the effective control
of the Government of the Republic of Cyprus.
The statistical data for Israel are supplied by and under the responsibility of the relevant
Israeli authorities. The use of such data by the OECD is without prejudice to the status of
the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms
of international law.
Lithuania became a Member of the OECD on 5 July 2018. However, consistent with other
publications based on PISA 2015 data, Lithuania is not included in the OECD average.
You can copy, download or print OECD content for your own use, and you can include
excerpts from OECD publications, databases and multimedia products in your own
documents, presentations, blogs, websites and teaching materials, provided that suitable
acknowledgement of OECD as source and copyright owner is given. All requests for public
or commercial use and translation rights should be submitted to rights@oecd.org.Comment
on the series is welcome, and should be sent to edu.contact@oecd.org.
This working paper has been authorised by Andreas Schleicher, Director of the Directorate
for Education and Skills, OECD.

-------------------------------------------------------------------------
www.oecd.org/edu/workingpapers

--------------------------------------------------------------------------
© OECD 2019

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │3

Acknowledgements

The authors would like to thank Andreas Schleicher, Yuri Belfali, Miyako Ikeda and the
country representatives for valuable feedback on earlier drafts of this paper. This paper was
prepared by Adrien Lorenceau, Camille Marec and Tarek Mostafa from the OECD
Directorate for Education and Skills, and edited by Marilyn Achiron. The work on
developing the PISA 2021 ICT framework and for upgrading the corresponding ICT
questionnaire was co-funded by the European Commission and the OECD.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
4 │ EDU/WKP(2019)10

Abstract

This paper explains the rationale for updating the OECD Programme for International
Student Assessment (PISA) 2021 questionnaire on Information and Communication
Technology (ICT) and shows how it covers policy topics of current relevance. After
presenting key findings based on previous ICT-related PISA data, the paper provides a
summary of the PISA 2021 ICT framework guiding the development of the questionnaire.
The paper then describes the process followed by the OECD/PISA secretariat for the
development of the PISA 2021 ICT questionnaire items. The paper concludes by drawing
some lessons that would inform future development of this instrument.

Résumé

Cet article décrit les motivations de l’actualisation du questionnaire du Programme

international de l’OCDE pour le suivi des acquis des élèves (PISA) 2021 sur les
Technologies de l’Information et de la Communication (TIC) et explique en quoi celui-ci
couvre des sujets d’actualité en termes de politiques publiques. Après avoir présenté des
résultats clé du PISA portant sur les TIC, cet article propose un résumé du cadre théorique
qui a guidé la construction du questionnaire. Il décrit ensuite le processus suivi par le
secrétariat de l’OCDE/PISA pour développer les différents items. L’article se conclut par
des recommandations pour le développement de futurs questionnaires.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │5

Table of contents

Acknowledgements ................................................................................................................................ 3
Abstract .................................................................................................................................................. 4
Résumé ................................................................................................................................................... 4
1. Introduction ....................................................................................................................................... 6
1.1. Outputs of the project.................................................................................................................... 7
1.2. The process ................................................................................................................................... 7
2. Lessons from previous PISA cycles................................................................................................ 10
2.1. A rapid but unequal expansion of ICTs ...................................................................................... 10
2.2. A fast-changing ICT environment: From digital resources to digital practices .......................... 16
2.3. ICT use and students’ cognitive outcomes ................................................................................. 23
3. Summary of the PISA 2021 ICT Framework ............................................................................... 25
3.1. Overall objective and conceptual approach ................................................................................ 25
3.2. Access to quality ICTs ................................................................................................................ 28
3.3. Using ICTs in the classroom ....................................................................................................... 28
3.4. ICT use outside of the classroom ................................................................................................ 31
3.5. Digital competencies: Attitudes and dispositions ....................................................................... 33
4. Upgrading the ICT questionnaire items for PISA 2021 ............................................................... 37
4.1. Access to ICTs ............................................................................................................................ 37
4.2. Use of ICTs ................................................................................................................................. 40
4.3. Attitudes and disposition towards ICTs ...................................................................................... 42
5. Lessons for the future development of ICT questions in PISA ................................................... 45
References ............................................................................................................................................ 47

Figures
Figure 2.1. Access to the Internet (Panel A) and a computer available for schoolwork (Panel B) at
home (2003-15) ............................................................................................................................. 11
Figure 2.2. Access to the Internet at school (2009-15) .......................................................................... 12
Figure 2.3. Students with an Internet connection at home, by socio-economic status .......................... 14
Figure 2.4. Differences between advantaged and disadvantaged students in time spent using the
Internet........................................................................................................................................... 16
Figure 2.5. Change between 2009 and 2015 in the availability of desktop computers at home ........... 18
Figure 2.6. Change between 2012 and 2015 in the availability of tablet computers at home ............... 18
Figure 2.7. Access to educational software at home (2003-15) ............................................................ 20
Figure 2.8. Average life satisfaction, by time spent on the Internet outside of school during
weekend days ................................................................................................................................ 22
Figure 2.9. Students’ skills in reading, by index of ICT use at school .................................................. 24
Figure 3.1. PISA 2021 ICT Framework ................................................................................................ 27
Figure 3.2. Detailing ICT use in school ................................................................................................ 31
Figure 3.3. ICT use outside of the classroom ........................................................................................ 32

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
6 │ EDU/WKP(2019)10

1. Introduction

Information and communication technologies (ICTs) play an increasingly important role in

virtually all aspects of our daily lives. Not only is technology profoundly transforming
people’s work and professional lives, but it is also altering how people interact,
communicate, retrieve and share information, and even how governments provide public
services to citizens. ICTs also significantly affect multiple facets of education. They can
provide new opportunities for students to learn outside of school, and can change teachers’
pedagogical approaches and the learning experience of students in school.
In 2000, the OECD Programme for International Student Assessment (PISA) distributed,
for the first time, a questionnaire with the aim of documenting students’ familiarity with
and use of ICT. Since then the ICT familiarity questionnaire has been offered to countries
as an optional part of the PISA assessment. The broad participation in the ICT questionnaire
– with 50 countries and economies conducting the survey in 2018 – reflects the growing
importance of ICT and digitalisation in the education policy debate. With the role of ICT
in education figuring high on the agenda of many PISA-participating countries, the
OECD/PISA secretariat – with European Commission funding – launched the development
of an ICT conceptual framework, which serves as the basis for upgrading the ICT
questionnaire in PISA 2021.
This paper describes the rationale and process that the OECD/PISA secretariat followed in
completing this project. The process involved summarising lessons learned from the ICT
questionnaires distributed since PISA 2000, reviewing related literature, convening experts
to discuss the scope of the PISA ICT questionnaire, developing the ICT conceptual
framework, updating the existing ICT questionnaire items, introducing new ICT-related
items in the student, school, parent and teacher questionnaires, and testing the student-level
questionnaire items in cognitive labs.
The work was undertaken in response to the fact that certain aspects of the existing PISA
ICT questionnaire items have become obsolete. For instance, too much emphasis had been
placed on the availability of certain technologies and on the intensity of their use. Some of
these technologies no longer exist. To address those limitations, the questionnaire was
upgraded in a number of ways. First, the new questionnaire documents availability and use
simultaneously while focusing on generic digital devices and digital resources likely to
affect learning. Second, accessibility and quality of digital resources are now assessed by
asking students to describe whether the ICT resources available to them are fit for purpose.
Most important, prominence is given to topics of interest to education policy makers, such
as students’ awareness of the risks of using ICTs, their attitudes towards ICTs, their use of
ICTs for learning and for leisure in and out of school, their views on their teachers’ abilities
and willingness to use ICTs, and their views on the ICT environment at home and at school.
The new questionnaire also explores students’ self-efficacy and confidence when using
ICTs to perform various tasks.
The paper is organised as follows: Section 2 presents results from past PISA ICT
questionnaires and highlights the gaps and limitations of existing data and knowledge.
Section 3 summarises the conceptual framework guiding the development of the
PISA 2021 ICT questionnaire. Section 4 describes the new developments in the PISA 2021
ICT questionnaire. The concluding section presents lessons for the future development of
ICT questions in PISA.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │7

1.1. Outputs of the project

1.1.1. The PISA 2021 ICT framework

The main goal of the framework is to paint a comprehensive picture of the availability and
use of ICT resources by 15-year-old students and of students’ self-reported proficiency in
working with ICTs. The framework focuses on three areas:
Quality of ICT resources. Information on whether schools have computers connected to the
Internet is not sufficient to understand the relationship between ICT and student
performance, since most schools in OECD countries do have these resources. The major
differences between schools (and countries) revolve around issues of quality rather than
quantity of resources. For instance, Internet speed, whether the Internet is available in every
class and for every student, and how it is used to support learning, have all become more
important than the simple question of availability.
Detailed use of ICTs. Students use ICTs for various activities (e.g. doing research but also
chatting with friends and having fun), some of which, and under certain circumstances,
could be distracting. This raises the question of how students spend their time engaging
with ICTs, and whether such activities are monitored or structured in a way to foster
learning. The existing ICT questionnaire asks about the types of ICT-related activities that
students are engaging in. However, it does not provide much information on time spent on
different activities or whether the activities are monitored. Furthermore, teachers and
school principals can provide important information on detailed usage of ICT in schools.
This information includes how ICTs are used in teaching, the types and amount of
professional training and support that teachers receive to adapt ICT to their teaching, and
school-level policies regarding the use of ICTs in teaching and learning.
Students’ self-report on ICT proficiency. ICT literacy and the ability to use technology
effectively in today’s knowledge economy is a key 21st century skill. Today’s environment
is characterised by the ease of access to abundant information and by the rapid change in
technology. A successful student is someone who is able to use technology as a tool to
research, manage, analyse and communicate information while being flexible and able to
adopt new ICTs as they arise. Previously in PISA, students were asked basic questions
about their self-efficacy in the use of ICTs. The new ICT questionnaire aims to develop
these questions to better reflect students’ skills and knowledge of ICTs and their
effectiveness in using them in their daily life.

1.1.2. The PISA 2021 ICT questionnaire

The PISA 2021 ICT questionnaire was upgraded to cover topics of current relevance for
policy makers and to reflect the developments in the field. The lessons drawn from the
analysis of previous PISA cycles, and from an extensive review of academic literature and
policy reports on the subject, guided the selection of the areas to cover in the questionnaire.
Under the guidance of the ICT expert group, the revision of the PISA ICT familiarity
questionnaire was seen as an opportunity for a fresh start that will provide a forward-
looking model for the assessment of ICTs in education.

1.2. The process

In order to ensure that the ICT framework and questionnaire items covered all topics of
current policy relevance, the OECD/PISA secretariat followed a multistage process of
development and revisions. The process involved:

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
8 │ EDU/WKP(2019)10

1. Developing the ICT conceptual framework based on a review of relevant academic

and policy literature and on analyses of existing PISA ICT data.
2. Seeking guidance and advice from a group of experts on the topic. The experts
provided valuable input for the development of the conceptual framework and
identified key areas to cover in the questionnaires. They also reviewed and
commented all outputs throughout the process.
3. Once questionnaire items were selected, they were qualitatively tested in cognitive
labs organised and delivered by Statistics Canada.
4. Quality assurance was provided by the OECD/PISA secretariat. Analysts in the
OECD/PISA secretariat drafted the outputs and selected the questionnaire items.

1.2.1. The expert group

The expert group consisted of five external experts who provided advice on the
development of the ICT framework and the PISA 2021 questionnaire items. The group
members were:
 Jeppe Bundsgaard (Professor at University of Aarhus, Danish School of
Education – Denmark).
 Cindy Ong (Senior Specialist, Educational Technology Division, Ministry of
Education – Singapore).
 Michael Trucano (Senior Education and Technology Policy Specialist and Global
Lead for Innovation in Education, World Bank – United States).
 Patricia Wastiau (Principal Adviser for Research and Innovation, Schoolnet –
France/Belgium).
 Pat Yongpradit (Chief Academic Officer, Code.org – United States).
The expert group and the PISA team managing the project met on four occasions (two
virtual and two face-to-face meeting). The first virtual and first face-to-face meetings were
dedicated to discussing the scope and content of the ICT framework; the following two
focused on the development of the ICT questionnaire items.
For the ICT framework, members of the expert group were asked to identify key conceptual
areas regarding the use of ICTs to support learning. They also commented on the different
drafts of the framework and provided input when needed. Analysts in the OECD/PISA
secretariat drafted the framework.
For the ICT questionnaire, the expert group drew a list of key areas to cover, then provided
a set of possible questions to include in the questionnaire. Those questions were reviewed
internally by the PISA analysts and revisions were made after extensive discussion with
the experts. Finally, the chosen questionnaire items were sent to be tested in cognitive labs.

1.2.2. The cognitive labs

The first set of ICT questionnaire items developed by the OECD/PISA secretariat in
collaboration with the experts was qualitatively tested in cognitive labs organised and
implemented by Statistics Canada. Twenty-four one-on-one interviews were scheduled to
take place in Ottawa in both French and English. In the end, a total of fourteen interviews
were conducted: seven in each language.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │9

While ICT-related questions for teachers and principals were desk-reviewed, Statistics
Canada tested the student-level questionnaire with 15-year-old students from various
backgrounds and provided extensive comments on the wording and complexity of the
questions.
The objectives of the testing were:
 to obtain feedback from respondents on their overall impressions of and reactions to
the proposed content and questions
 to test the cognitive processes of respondents in answering the questions including:
a) an assessment of respondents’ understanding of the concepts, terminology,
questions and response categories; and b) an assessment of the availability of the
information requested
 to test respondents’ ability and willingness to answer the questions

1.2.3. Quality assurance

All drafts of the PISA 2021 ICT framework and of the corresponding questionnaire were
reviewed internally by members of the OECD/PISA secretariat. Comments and inputs were
also sought from other OECD colleagues and from colleagues at the European Commission
working on topics related to digital literacy and skills, digitalisation and the use of ICTs by
young people.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
10 │ EDU/WKP(2019)10

2. Lessons from previous PISA cycles

A critical feature of the ongoing digital revolution is that the landscape of digital
technologies and practices is evolving very fast. In a matter of a decade, the development
of online storage and computational capacities enabled people to stream movies, to save
and share their work online, and virtually replaced local storage devices (e.g. USB sticks
and hard drives). Conversely, PISA is relatively “sticky” when it comes to elaborating and
updating questionnaire items. Although PISA seeks to inform current education policy
issues, it is constrained in timing and flexibility by its simultaneous implementation in more
than 70 countries and economies. Moreover, one trade-off that PISA faces is the need to
follow trends over time on certain aspects of education policy while keeping the questions
up-to-date and relevant for current policy making.
This section highlights the shortcomings and limitations of the previous PISA ICT
questionnaire and the need to improve the questionnaire items to ensure relevance and
accuracy. It shows how the evolution of digital technologies and practices requires updating
the ICT questionnaire for PISA 2021. The data show that some questions covering
particular digital resources and practices are not relevant anymore, while other aspects are
not documented.

2.1. A rapid but unequal expansion of ICTs

2.1.1. A rapid expansion of ICTs

Most students in most countries have access to generic ICT resources

Whether at home or at school, students’ access to the Internet and to computers has
expanded rapidly enough to become ubiquitous in 2015 in most PISA-participating
countries. Among the 40 countries participating in 2003, the share of students with access
to the Internet at home increased by 35 percentage points, on average, from 57% in 2003
to 92% in 2015 (
Figure 2.1, Panel A). The bulk of the expansion took place between 2003 and 2009. By that
date, the share of students with access to the Internet was already close to 100% in many
OECD countries. Students’ access to a computer for schoolwork at home expanded less
over the same period (17 percentage points, on average), given that possession of a
computer was very common in 2003 with about 70% of students, on average, owning a
computer (
Figure 2.1, Panel B).
Students’ access to the Internet in school has also remarkably increased between 2009 and
2015. In 33 of the 47 countries that distributed the ICT familiarity questionnaire in 2015,
more than 90% of students had access to the Internet at school (Figure 2.2). On average,
91% of students had access to the Internet at school in 2015, a slightly higher percentage
than that in 2012 (89%, based on all countries with data in 2012) but much higher than the
percentage in 2009 (33%, based on all countries with data in 2009).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 11

Figure 2.1. Access to the Internet (Panel A) and a computer available for schoolwork
(Panel B) at home (2003-15)

1. B-S-J-G (China) refers to the four PISA-participating Chinese provinces of Beijing, Shanghai, Jiangsu and
Guangdong.
2. CABA (Argentina) refers to the adjudicated region of Ciudad Autónoma de Buenos Aires (CABA).
3. Note by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of
the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island.
Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is
found within the context of the United Nations, Turkey shall preserve its position concerning the “Cyprus
issue”.
Note by all the European Union Member States of the OECD and the European Union: The Republic of Cyprus
is recognised by all members of the United Nations with the exception of Turkey. The information in this
document relates to the area under the effective control of the Government of the Republic of Cyprus.
4. North Macedonia refers to the Republic of North Macedonia.
Notes: Only countries and economies with available data in 2015 are shown in the figure.
(Panel A) Countries and economies are ranked in ascending order of the percentage of students who had access
to the Internet at home in 2015. (Panel B) Countries and economies are ranked in ascending order of the
percentage of students who had a home computer available for schoolwork in 2015.
Source: OECD, PISA 2003, PISA 2006, PISA 2009, PISA 2012 and 2015 Databases
(http://www.oecd.org/pisa/data/ - accessed on 10 December 2018).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
12 │ EDU/WKP(2019)10

Figure 2.2. Access to the Internet at school (2009-15)

1. B-S-J-G (China) refers to the four PISA-participating Chinese provinces of Beijing, Shanghai, Jiangsu and
Guangdong.
Notes: Only countries and economies with available data in 2015 are shown in the figure.
Countries and economies are ranked in ascending order of the percentage of students who had access to the
Internet at school in 2015.
Source: OECD, PISA 2009, PISA 2012 and PISA 2015 Databases (http://www.oecd.org/pisa/data/ - accessed
on 10 December 2018).

2.1.2. Heterogeneity between and within countries in access to ICTs

ICT resources are not yet ubiquitous in emerging and developing countries
Internet and computers are not yet available for a large proportion of students in
developing and emerging countries. Of the 74 countries and economies that participated
in PISA 2015, the proportion of students with no access to the Internet at home was below
50% in 4 countries (Algeria, Indonesia, Peru and Viet Nam) (
Figure 2.1, Panel A) and ranged between 50% and 80% in 13 countries.
Similarly, in 2015 the share of students with access to a computer was below 50% in
Indonesia and Viet Nam and between 50% and 80% in 16 countries including, in ascending
order, Peru (55%), Mexico (57%), Japan (62%), Tunisia (65%) and Lebanon (77%) (
Figure 2.1, Panel B). The spread of computers and Internet access is still an ongoing
process in these countries as shown by the continuous increase in the proportion of
students’ accessing these resources. Those countries are progressively catching up with
wealthier OECD economies where students’ access to computers and the Internet is
widespread (
Figure 2.1).

Within-country differences in ICT access remain strong

The “first digital divide” remains a serious issue in many PISA-participating
countries. Across OECD countries that participated in PISA in 2015, 88% of

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 13

disadvantaged students (i.e. students who belong to the bottom quarter of the index of
economic, social and cultural status) had access to the Internet at home in comparison with
almost 100% of their advantaged counterparts (Figure 2.2). This divide is much greater
across OECD partner countries and economies, and remains a major policy concern in
Indonesia, Mexico, Peru and Viet Nam, where less than 15% of disadvantaged students had
access to the Internet at home. In contrast, as much as 99% of disadvantaged students in
Denmark, Finland, Iceland and Slovenia had access to the Internet at home, and the
availability of basic digital resources is now universal.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
14 │ EDU/WKP(2019)10

Figure 2.3. Students with an Internet connection at home, by socio-economic status

1. B-S-J-G (China) refers to the four PISA-participating Chinese provinces of Beijing, Shanghai, Jiangsu and
Guangdong.
2. CABA (Argentina) refers to the adjudicated region of Ciudad Autónoma de Buenos Aires.
3. See note 3 in Figure 2.1.
Source: OECD, PISA 2015 Database (https://www.oecd.org/pisa/data/2015database/ - accessed on
10 December 2018).

Those results suggest that documenting access to basic ICT resources and to the Internet
does not provide additional information for OECD countries since access and availability
are now universal. While the use of computers and the Internet by 15-year-old students was
not very common in 2003, by 2015 both computers and Internet access were widely
available. However, two considerations arise. First, the issue of access to basic ICT

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 15

resources remains important for PISA partner countries and economies and therefore
should be covered in future PISA ICT familiarity questionnaires. Second, for OECD
countries, the questionnaire should go beyond documenting the simple availability of basic
ICT resources. This could be achieved by focusing on more sophisticated resources, their
quality, the modalities of their use, and on students’ competencies in using them.

2.1.3. Students’ use of ICTs is increasing and is getting more diversified

Students spend more and more time online, both at home and at school. While most
students in OECD countries already had access to the Internet at home in 2012, the increase
in students’ Internet use between 2012 and 2015 was remarkable (Figure 2.4). Over these
three years, the time students spent online increased from 21 to 29 hours per week, on
average across the 27 countries that distributed the ICT questionnaire in 2012 and 2015
(OECD, 2018[1]). In Costa Rica – the country that experienced the largest increase –
students spent almost twice as much time online in 2015 (37 hours per week) than in 2012
(19 hours). While two-thirds of the increase in time spent using the Internet happened
outside of school (5.4 hours per week), on average across OECD countries, students also
spent 2.7 hours more per week online at school in 2015 than they did in 2012. This rapid
increase in the time spent online reflects the greater connectivity following the expansion
of access to the Internet. This trend is likely to continue in the near future and therefore
should be documented in the upcoming cycles of PISA data collection.
The modalities of Internet use are diverse. In countries and economies where access to
the Internet is almost universal, disadvantaged students spend more time online in
comparison with their advantaged counterparts (Figure 2.4). On average, disadvantaged
students spent 2.3 hours more online each week. Those in Chinese Taipei spent up to
7.7 hours more, those in Belgium 7.3 hours more, and disadvantaged students in Austria
and Spain spent 6.3 hours more. However, the traditional digital divide persists in countries
where access is still not widespread. Thus, advantaged students in Mexico spent 18.8 hours
more per week online than their disadvantaged peers. In Costa Rica, such students spent
17.2 hours more per week, and in Uruguay, 9 hours more per week. Hence, depending on
the situation the time students spend online takes on a different meaning. Documenting
precisely how students use ICTs is therefore of primary interest. Moreover, disadvantaged
students are more likely to have simple, repetitive skill based experiences while their
advantaged peers might have more experience with problem solving and creating digital
solutions.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
16 │ EDU/WKP(2019)10

Figure 2.4. Differences between advantaged and disadvantaged students in time spent using
the Internet

Notes: To obtain the weekly average, the response categories were recoded with the middle value
(e.g. “31-60 minutes per day” being coded as “45.5 minutes per day”) and then multiplied by 5 if they refer to
a school day and by 2 if they refer to a weekend day.
Countries and economies are ranked in descending order of the difference between advantaged and
disadvantaged students in the time spent using the Internet in 2015.
Source: OECD (2018) PISA in Focus 2018/83.

2.2. A fast-changing ICT environment: From digital resources to digital practices

In the past decade, the fast-changing digital and technological environment has modified
peoples’ digital practices and habits. The omnipresence of smartphones in one’s daily life
is a good example of how people’s habits have radically changed because of technological
progress. For example, people can now use their smartphones to listen to music, access the
Internet and make a phone call. These tasks were originally done separately using multiple
devices, but now they can be performed on a single device that has become essential
to own.
The rapid expansion and diversity of digital technologies challenges the ICT familiarity
questionnaire in different ways. It makes it difficult for the ICT questionnaire to cover the
full breadth and diversity of digital devices and practices. Moreover, as digital technologies
develop, some become ubiquitous and universal while others become obsolete. This would
require adapting and updating the questionnaire for each PISA cycle. In practice, different
generations of digital devices fulfill similar roles (e.g. desktop, laptop and tablet
computers), which complicates even further the documentation of their availability and the

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 17

evolution of their use. Moreover, PISA occurs every three years and questionnaires are
prepared well in advance of data collection. This means that technologies might change
within this timeframe, rendering some of the questionnaire items irrelevant. Therefore, ICT
questions should not be too specific and should not focus on a particular device or software
but rather on the role they fulfill.

2.2.1. Evolution of digital technologies

Relevance of ICT resources

The coverage of ICT resources has evolved over PISA cycles in an attempt to better
capture the changing digital landscape. Between 2000 and 2006, the ICT familiarity
questionnaires covered only students’ access to a computer and the Internet and their use
of video games, educational software and generic tools, such as text editors and spreadsheet
software. From PISA 2009 onward, coverage of the ICT questionnaire was extended to
include newly available digital devices, such as laptops, printers, memory sticks and
mp3 music players, and school-specific digital resources, such as school-provided storage
space, projectors and interactive whiteboards. From 2015, students were asked how
frequently they conduct specific tasks at and outside of school using digital devices. The
tasks ranged from playing multi-player video games, to writing e-mails and chatting online
to browsing the Internet for schoolwork.
Yet in spite of this evolution, the ICT familiarity questionnaire still documents digital
devices that will no longer be relevant in 2021. The PISA 2018 ICT familiarity
questionnaire documents students’ access to portable music players (e.g. MP3/MP4, iPod
or similar), a printer and a memory stick. These digital devices are deemed not relevant to
document students’ exposure to new or modern digital resources. Indeed, most students use
their smartphones to listen to music, and the possession of a memory stick is not necessarily
relevant to students’ educational experiences in 2021 since possession has become
universal and data storage is shifting to new platforms (e.g. cloud-based applications).
Some of these ICT resources were covered in the 2018 questionnaire in order to maintain
trends across PISA cycles. However, this is conceptually problematic because using a
memory stick in 2009 has a different meaning than doing so in 2018. Today, it would
indicate that a student is using an old technology in an era where everything is stored online.
Therefore, a decision was made to give precedence to updating the coverage of the
PISA 2021 ICT questionnaire rather than maintaining trends that are no longer useful.
Technological change over time adds a layer of complexity to the PISA ICT
questionnaire. The rapid replacement of desktop computers by laptop and other forms of
computers shows how the fast-changing ICT environment can affect ICT questionnaire
validity and relevance. Indeed, between 2009 and 2015, PISA data documents a 10% drop
(from about 87% to 77%) in the share of students having access to a desktop computer at
home (Figure 2.5). At first, this result could be interpreted as a decrease in access to ICT
resources. Yet, this decrease was more than compensated by the rise (from around 30% to
70%) in the share of students having access to a tablet computer at home between 2012 and
2015 (Figure 2.6). This evolution (i.e. common digital devices being replaced by new ones)
constitutes a serious challenge for investigating ICT access and usage. Indeed, while tablet
computers are not fundamentally different from desktop computers – at least in their
potential contribution to learning – they are still not exactly similar and involve a specific
cognitive experience.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
18 │ EDU/WKP(2019)10

Figure 2.5. Change between 2009 and 2015 in the availability of desktop computers at home

Note: Countries and economies are ranked in ascending order of the percentage of students who had access to
a desktop computer at home in 2015.
Source: OECD, PISA 2009 and PISA 2015 Databases (https://www.oecd.org/pisa/data/ - accessed on
10 December 2018).

Figure 2.6. Change between 2012 and 2015 in the availability of tablet computers at home

Note: Countries and economies are ranked in ascending order of the percentage of students who had access to
a tablet computer at home in 2015.
Source: OECD, PISA 2012 and PISA 2015 Databases (https://www.oecd.org/pisa/data/ - accessed on
10 December 2018).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 19

Moreover, the digital landscape is becoming even more complex due to the consolidation
of devices and uses. For instance, in the past decade, a plethora of new devices and services,
such as smartphones, connected watches, programmable vacuum cleaners and drones, have
emerged. Some, like smartphones, have consolidated multiple functions that were separate
some years ago. Not only do they include high-quality digital cameras and allow accessing
the Internet almost everywhere, they are also more powerful than older laptop computers.
The frontier between a laptop, a tablet and a smartphone is now blurry and is not necessarily
related to the use of each digital device. In this context, it becomes difficult and even
counterintuitive to focus on one particular type of digital device – especially given the
overlapping nature of these devices and their unpredictable evolution.
Therefore, continuously updating the PISA ICT questionnaire to include new ICT devices
seems unrealistic. Rather than measuring the degree of sophistication of students’ ICT
resources, the questionnaire should focus on the relationship between digital resources and
learning. Consequently, the PISA 2021 strategy focuses on the use of digital devices that
are expected to influence learning outcomes. The strategy is articulated around the function
of the device rather than its nature.

Covering software and online resources

As mentioned earlier, in a context of almost universal access to computers and the Internet,
documenting students’ access to ICT resources requires going beyond the mere availability
of digital devices to exploring how students use them. This suggests investigating the
availability and use of software and online resources in general, which proves to be
challenging for different reasons. One specific challenge in covering software lies in the
fact that students do not need to possess these resources per se. Having access to the Internet
is sufficient to ensure access to a wealth of online resources and software, many of which
are free or have a free equivalent.
Second, online resources and software are virtually infinite, and it is extremely difficult to
define and classify them in a way that is relevant to all PISA-participating countries and
economies while maintaining coherence over time. Following the same logic discussed
above, it seems realistic to document students’ access to online resources and software
specifically designed to support learning instead of mapping all types of software. Students’
access to educational software at home has been documented in the PISA ICT questionnaire
starting in 2003. Unlike computers and access to the Internet, educational software is not
widely available (see below) and as such, documenting its availability in future PISA cycles
is warranted. However, the range of digital learning resources covered in the questionnaire
should be extended to include new types of resources, including learning apps, games and
various online education tools, such as school platforms and learning management systems.
Again, the objective is not to extensively cover all types of digital learning resources but
rather to identify the main categories. The reason is that they differ widely in quality and
purpose, and because their potential effects on students’ cognitive outcomes depend on
how they are used in practice.
Students’ access to digital resources for learning purposes is not widespread in all
countries. On average across OECD countries, the proportion of students with access to
educational software at home increased from 43% in 2003 to 56% in 2006 and remained
stable since then until 2015 (Figure 2.7). The share of students with access to educational
software at home in 2003 varied greatly across countries, ranging from 12% in Japan, 19%
in Viet Nam, 36% in France and 40% in Finland to 75% in Estonia, 82% in Iceland and
91% in Denmark. Interestingly, even countries where more than 90% of students had access

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
20 │ EDU/WKP(2019)10

to the Internet and a computer at home (such as France and Finland) showed relatively low
access to educational software at home.

Figure 2.7. Access to educational software at home (2003-15)

1. B-S-J-G (China) refers to the four PISA-participating Chinese provinces of Beijing, Shanghai, Jiangsu and
Guangdong.
2. CABA (Argentina) refers to the adjudicated region of Ciudad Autónoma de Buenos Aires.
3. See note 3 in Figure 2.1.
4. North Macedonia refers to the Republic of North Macedonia.
Notes: Only countries and economies with available data in 2015 are shown in the figure.
Countries and economies are ranked in ascending order of the percentage of students who had access to
educational software at home in 2015.
Source: OECD, PISA 2003, 2006, 2009, 2012 and 2015 Databases (https://www.oecd.org/pisa/data/ - accessed
on 10 December 2018).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 21

2.2.2. Evolution of digital practices

Time spent online and risks related to ICT use

Digital practices and habits have evolved together with the spread of ICTs. As mentioned
earlier, one implication of the spread of Internet and smartphone use is a sharp increase in
the time students spend online. Between 2012 and 2015, the time students spent online
increased from 21 to 29 hours per week, on average in OECD countries (OECD, 2018[1]).
Given the importance of the intensity of students’ use of ICTs in the policy debate, it is
crucial for the ICT familiarity questionnaire to explore this upward trend. In fact, it will be
necessary to propose more response categories regarding the time spent online. Eventually,
the evolution towards a world of constant connection to the Internet will shed doubt on the
usefulness of the current classification of high vs. low Internet use. Thus, documenting
carefully when, where and why students are online might be necessary to provide a more
insightful account of time spent online.
The more time students spend online, the greater their exposure to online opportunities and
risks (Hooft Graafland, 2018[2]). Students can benefit from many opportunities online,
ranging from e-learning classes to seeking personal advice on gender, sex, health, identity
and other issues (Hooft Graafland, 2018[2]). Furthermore, the PISA 2015 ICT questionnaire
reveals that disadvantaged students spend more time online than their advantaged
counterparts (OECD, 2018[1]). This suggests that they might be more engaged in excessive
Internet use or “binge gaming” for example. This can have adverse effects on students’
well-being as intensive Internet users are less likely to be satisfied with their lives
(Figure 2.8). Other online risks include exposure to inappropriate content
(e.g. pornographic, violent or dangerous content) and online bullying. The ICT familiarity
questionnaire should cover more extensively both online risks and opportunities.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
22 │ EDU/WKP(2019)10

Figure 2.8. Average life satisfaction, by time spent on the Internet outside of school during
weekend days

1. B-S-J-G (China) refers to the four PISA-participating Chinese provinces of Beijing, Shanghai, Jiangsu and
Guangdong.
Notes: Categories of Internet users are based on students’ responses to questions about how much time they
spend online, outside of school, during a typical weekend day. Low Internet users: one hour or less; moderate
Internet users: 1 to 2 hours; high Internet users: 2 to 6 hours; intensive Internet users: more than 6 hours.
Statistically significant differences in life satisfaction between intensive Internet users and other Internet users
are shown next to the country/economy name.
Countries and economies are ranked in descending order of the average life satisfaction of intensive Internet
users.
Source: OECD, PISA Database (https://www.oecd.org/pisa/data/2015database/ - accessed on
10 December 2018).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 23

Mastery of basic technical and routine ICT tasks is now widespread

The dissemination of new digital technologies also affects how students use ICTs and
consequently how the ICT familiarity questionnaire should document it. For example, the
2018 ICT familiarity questionnaire assesses the frequency with which students “use
e-mails” and “chat on line”. Yet, writing e-mails is not common anymore among
15-year-old students and has been gradually replaced by a range of different
communication tools and modes (e.g. Messenger, Whatsapp, Snapchat, etc.). Moreover,
many activities that once were considered as technical digital tasks, such as downloading
and installing apps, are now part of students’ routine digital practices. This stresses the need
to focus the ICT familiarity questionnaire on current digital habits.

Attitudes and dispositions towards ICTs

For similar reasons, some questions regarding students’ attitudes and dispositions towards
ICTs are often outdated in the current version of the ICT questionnaire. Asking students
whether they “like using digital devices” or whether “the Internet is a great resource for
obtaining information” might have been relevant to document students’ interest in digital
devices when they were not widespread. Nowadays, these technologies are taken for
granted and are part of everyday activities.
Moreover, students’ self-efficacy as assessed in the 2018 questionnaire focuses mainly on
whether students feel comfortable using digital devices or can provide advice to their
friends. As such, this definition does not reflect the diversity nor the depth of the digital
competencies that students should possess in the 21st century. The 2021 ICT familiarity
questionnaire should reflect the growing importance of major digital competencies, such
as how to access and assess information, communicate effectively, create digital content or
solve digital problems. Some of these competencies overlap with broader areas of interest
for PISA, such as global competence and problem solving.

2.3. ICT use and students’ cognitive outcomes

Policy makers are increasingly concerned with how ICTs affect education. Most education
systems have launched ICT plans and policies to help schools and teachers integrate the
latest digital technologies into teaching while coping with potential disruptive effects
(Conrads et al., 2017[3]). Indeed, ICTs hold great promise for enhancing teaching,
supporting school management and developing students’ digital skills for the 21st century.
Yet, both international surveys and the recent policy evaluation literature provide mixed
evidence on the contribution of ICTs to education outcomes (Fraillon et al., 2014[4]; Escueta
et al., 2017[5]). The findings from PISA are not different. A more frequent and diverse use
of ICT in the classroom is not associated with higher performance in mathematics, reading
or science in general. In fact, PISA results show a bell-shaped relationship between ICT
use and students’ performance (OECD, 2015[6]). This unclear relationship between ICT use
and students’ cognitive outcomes warrants further investigation of how and why students
use ICTs in the classroom.

2.3.1. A bell-shaped relationship between ICT use and students’ cognitive

outcomes
The PISA 2009 ICT familiarity questionnaire documents how frequently students conduct
specific tasks with digital devices both at school and at home. Based on these data, an index
of ICT use was constructed. It was based on the frequency with which students chat online,

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
24 │ EDU/WKP(2019)10

use e-mails, browse the Internet for schoolwork, play simulations, and use learning apps
and websites. Findings from PISA 2012 and 2015 showed an overall negative correlation
between the index of ICT use at school and students’ performance in mathematics, science
and reading (OECD, 2015[6]; OECD, forthcoming[7]). Figure 2.9, based on PISA 2012
results, shows the overall relationship between this index and students’ performance in
reading. Students with the highest performance in both reading and digital reading use ICT
slightly less than the average OECD student does. The bell-shaped relationship suggests
that moderate use of digital devices at school may be better than no use at all, but ICT use
above the OECD average is associated with significantly lower results.

Figure 2.9. Students’ skills in reading, by index of ICT use at school

OECD average relationship, after accounting for the socio-economic status of students and schools

Notes: The lines represent the predicted values of the respective outcome variable, at varying levels of the index
of ICT use at school, for students with a value of zero on the PISA index of economic, social and cultural status,
in schools where the average value of that index is zero.
Source: OECD, PISA 2012 Database (https://www.oecd.org/pisa/data/pisa2012database-
downloadabledata.htm/ - accessed on 10 December 2018).

2.3.2. Those indecisive results require updating the questionnaire

Overall, the current ICT questionnaire does not provide sufficient information to interpret
this bell-shaped relationship. While using ICTs can have a direct positive impact on
performance, using it excessively may lead to unintended negative consequences.
For instance, students may lack attention or may engage in non-learning activities during
lessons. Spending too much time using ICTs may also reduce the amount of time students
spend on other useful learning activities. Moreover, low performers might also be adversely
affected by the intensive reliance on ICTs in school. These are, of course, hypothesised
relationships that require in-depth investigation. The PISA 2021 ICT familiarity
questionnaire aims at providing more data that should help answer some of these questions.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 25

3. Summary of the PISA 2021 ICT Framework

This section presents a summary of the conceptual framework guiding the development of
the PISA 2021 ICT questionnaire items. The framework provides a comprehensive strategy
to document how students access and use ICT resources in and outside of school for
learning and for leisure activities. It identifies how teachers, schools and education systems
integrate ICTs into pedagogical practices and learning environments and explores students’
experiences with ICTs. It also examines students’ attitudes towards ICTs, their self-efficacy
and their awareness of safety issues when using ICTs.

3.1. Overall objective and conceptual approach

3.1.1. Why develop a framework on the integration of ICTs in teaching and

learning?
ICTs affect individuals in many ways, including their education: how they learn, how they
are taught in school and what they learn. Consequently, it raises important policy questions
about students’ access to and use of ICTs at and outside of school. Despite the growing
body of literature exploring the relationship between students’ engagement with ICTs and
education outcomes, there is no consensus on the contributions ICTs make to students’
educational attainment or cognitive performance in general.
Do students have sufficient access to quality digital resources? How do teachers integrate
ICTs into teaching practices and what are the best practices using ICTs? What are the
effects of integrating ICTs on students’ cognitive performance and well-being?
ICTs play an increasingly important role in many facets of our daily lives. They transform
people’s work and professional life, but also how they socialise, communicate, and retrieve
and share information. Education is no exception and is affected in many ways by the
expansion of ICTs. ICTs can provide new opportunities for students to learn outside of
school, and can change teachers’ pedagogical approaches and students’ learning experience
in school. Moreover, education systems are increasingly embedding digital competencies
in their curricula.
ICTs can affect learning in three major ways:
 Students’ use and engagement with ICTs (both in and outside of school) can affect
their cognitive processes and their well-being, and eventually what they learn.
 Teachers are increasingly using ICTs for instruction, and administrative and
communication purposes, with numerous implications for classroom management,
instructional practices, pedagogical approaches and time use.
 Competence in using ICTs and digital literacy are being recognised as skills that
students need to acquire if they are to flourish in the digital age.
Yet, the ICT familiarity questionnaires in previous PISA cycles were developed in an
ad-hoc manner without a comprehensive ICT assessment framework to guide their
development. This resulted in a number of shortcomings. For example, questionnaires
covered mainly hardware and access to the Internet while software and digital learning
resources were covered to a lesser extent. The quality and accessibility of these resources
were not systematically documented; and, more important, the use of ICT resources was

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
26 │ EDU/WKP(2019)10

only partially documented, with limited coverage of teachers’ pedagogical practices related
to ICT. In addition, differences in questionnaires across PISA cycles offered limited
possibilities for examining ICT-related trends over time.

3.1.2. Objectives of the PISA 2021 ICT framework

This framework provides a comprehensive strategy to document how students access and
use ICT resources in and outside of school, and to identify how teachers, schools and
education systems integrate ICTs into pedagogical practices and learning environments.
The framework allows for an exploration of how system-level factors influence schools’
and students’ experiences with ICTs, how the availability and use of ICTs interact with
various teaching practices, and how these associations correlate with students’ performance
in mathematics, reading and science, and with other outcomes, such as students’ ICT skills
and well-being.
Importantly, while the framework is not exhaustive, it provides an in-depth discussion of
the diverse ways in which ICTs and education interact. This aims to provide a structure for
the development of the ICT questionnaire items in PISA 2021 but also in future cycles.
However, ICT questionnaires cannot cover such large ground and will only reflect partially
the framework, with the objective of shedding light on aspects of prime interest for policy
makers. As such, the focus of the ICT questionnaire is likely to evolve with the changing
policy priorities.
This ICT assessment framework covers the dimensions listed below while ensuring
consistency and comparability across countries:
 access to ICTs, which encompasses availability, accessibility and quality of ICT
resources with a special focus on (connected) technologies that can support learning
(e.g. digital learning resources, learning management systems, etc.)
 use of ICTs, which covers the types and modalities of ICT use by students in an
informal and possibly unsupervised environment for learning and leisure, and in a
supervised situation in the classroom, notably through teachers’ pedagogical
practices with ICTs;1 it also includes alternative uses of ICTs by teachers to support
teaching
 students’ ICT competencies, which include attitudes towards ICT use (for
learning and for leisure), self-efficacy measures in the use of ICTs, and
questionnaire-based assessments of ICT skills.

3.1.3. Overall approach

At the heart of the PISA 2021 ICT framework is the relationship between two major
dimensions of ICT – access and use – and students’ outcomes (cognitive performance, well-
being, and ICT-related attitudes and competencies). However, the framework also aims to
identify how these links depend on contextual factors and background characteristics, and

1
The framework distinguishes between student use of ICT resources during classroom lessons (and
therefore under the supervision of at least one teacher) and ICT use outside of the classroom, which
includes ICT use at home and ICT use outside of class but not at home (whether in a school computer
lab, a library, or at any other location except home). For the sake of simplicity, ICT use subsumes
all of the above situations in the remainder of this text, unless otherwise noted.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 27

on existing policies and practices related to ICTs. Figure 3.1 provides an overview of the
underlying logic used to elaborate the PISA 2021 ICT framework.
Students’ use of ICT resources is conditional upon the availability, accessibility and quality
of those resources. Similarly, the amount and type of ICT resources made available to
students and teachers depends on their use. Therefore, the framework documents whether
human and physical ICT resources are available and are of sufficient quality; but most
important it sheds light on how ICTs are used by students and teachers.
The framework also acknowledges the influence of contextual factors, policies and
practices on both access to and use of ICT resources, and on students’ outcomes. Contextual
factors include the general background characteristics of the education system, schools and
students’ households. In addition, specific ICT-related policies and practices could directly
influence access to and use of ICT resources. Such policies include, for example, the
availability of specific funding for ICT resources, principals’ attitudes towards ICT use as
an instructional tool, and guidelines and support for teachers in using ICTs in the classroom.

Figure 3.1. PISA 2021 ICT Framework

Source: OECD (2019), PISA 2021 ICT Framework.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
28 │ EDU/WKP(2019)10

3.2. Access to quality ICTs

Previous PISA cycles mainly documented the type of ICT resources available at home or
in school. This framework broadens the focus and proposes a systematic and consistent
approach to measuring 15-year-old students’ access to ICT resources. Availability,
accessibility and quality are documented.
 The availability of ICT resources documents the presence of a specific ICT
resource, which can be used either in class or during students’ free time.
 The accessibility of ICT resources describes the set of elements that characterises
the ease and flexibility with which ICT resources can be accessed. Therefore, it
refers to existing rules, norms, configurations and arrangements guiding access to
ICT resources both in and outside of school.
 The quality of ICT resources is a multi-faceted concept that refers primarily to the
functionality, technical capacity and capability of ICT resources. Quality measures
describe the extent to which ICTs function smoothly – without flaws, delays or
security issues – and are compatible with other ICT resources (hardware or
software). In addition, some aspects of availability and accessibility also contribute
to the definition of quality, such as the diversity of ICT resources and the quantity
available per student. Moreover, dimensions such as the relevance and usability of
ICT resources – notably in the context of their use by 15-year-olds for learning
purposes – are also important for defining quality. These correspond to the degree
to which the ICT resources are relevant to the curriculum, create interest among
students who can easily work with them, and can be used for a variety of purposes
under different education settings.

3.3. Using ICTs in the classroom

As pointed out in recent literature reviews, merely providing ICT resources is not enough
to ensure that they are used effectively to improve students’ cognitive achievements,
well-being and ICT competencies (Bulman and Fairlie, 2016[8]; Escueta et al., 2017[5]).
Although the positive impact of ICT use on student achievement remains subject to debate,
there is a consensus that the specific purpose, context and pedagogical practices
surrounding ICTs are central to their effect on students.

3.3.1. Learning with ICTs

Teachers’ use of ICTs for teaching

Teachers’ pedagogical practices and teaching strategies with ICTs largely determine the
extent to which their use in the classroom will result in improved cognitive achievement.
Research stresses the promising potential of computer-assisted learning to bolster student
achievement (Roschelle et al., 2016[9]; Pane et al., 2013[10]; Karam et al., 2016[11];
Campuzano et al., 2009[12]). Thus, using ICTs for teaching and learning in the classroom
does not minimise teachers’ role. On the contrary, as the primary actors for implementing
the curriculum and orchestrating learning activities, teachers are likely to be even more
central to learning with the adoption of ICTs. Indeed, the success of using ICTs for
educational purposes relies heavily on teachers’ abilities to select, create and manage
adequate digital resources in order to implement innovative and inclusive teaching
strategies in a specific context (Redecker, Punie and European Commission. Joint Research

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 29

Centre., 2017[13]). Integrating ICTs into teaching may encourage teachers to modify their
approaches to teaching itself, which, in turn, could affect students’ use of ICTs for learning.
In addition to implementing pedagogical practices, teachers could also use ICTs for
management purposes, such as planning teaching sessions, assessing students, and taking
part in communication and collaboration activities with colleagues, parents and the students
themselves.
After a preliminary period of investing time to become acquainted with new technologies,
ICTs might help teachers prepare their lessons, regardless of whether ICT-based activities
will be conducted during class or not. In fact, the preparation of teaching activities
constitutes the most frequent ICT-based activity conducted by teachers in EU countries,
with 30% to 45% of students taught by teachers who declare doing this every day, almost
every day, or at least once a week (European Commission, 2013[14]).

Teachers use of ICTs for management

Teachers also spend a non-negligible amount of time communicating and co-operating with
parents and students, in addition to collaborating with other teachers (OECD, 2014[15]).
These activities may enhance the school climate and improve classroom environments
(OECD, 2014[15]). Moreover, teachers’ beliefs about the nature of teaching and learning
determine their choice of which pedagogical practices to use in the classroom (OECD,
2014[15]).

Students’ use of ICTs for learning in the classroom

Using ICTs in the classroom is likely to affect instructional time, the curriculum to be
taught, and teaching and learning practices. These factors have been documented as
important predictors of student achievement (Scherff and Piazza, 2008[16]; Schmidt and
Maier, 2009[17]; OECD, 2017[18]). Analysing those relationships requires documenting the
frequency and modalities of students’ use of ICTs in addition to assessing teaching and
learning strategies.
Based on this, existing PISA constructs on teaching and learning practices can be
complemented with ICT-specific information. This includes data on classroom
arrangements when ICTs are used, and students’ opinions of teachers’ ICT competencies.
 Intensity and modalities of students’ use of ICTs: The integration of ICTs in
teaching and learning can affect instructional time in many ways. Teaching with
ICTs takes more time as it often requires changing the classroom layout and may
require frequently altering pedagogical practices (Trucano, 2005[19]). Moreover,
when using certain ICT tools, students’ attention could be drawn away from
learning; they might be tempted to use the ICT resources for leisure activities
(e.g. games, browsing the Internet, social media, etc.). Yet ICT-assisted instruction
may also increase the overall time students spend learning. Therefore, it is
important to document not only how frequently students use ICT for learning, but
also the modalities: length of time they use ICTs in each class, whether they use
them continuously or recurrently, in which classes, whether they bring their own
devices or not, whether they have to change rooms and other classroom dynamics
regarding ICT use.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
30 │ EDU/WKP(2019)10

 Learning experiences with ICTs:

o The PISA student questionnaire highlighted different dimensions in assessing
instructional quality: structure and classroom management, teacher support and
student learning practices (OECD, 2017[20]). Each of these dimensions was
found to be correlated with students’ cognitive achievements (OECD, 2017[20];
OECD, 2013[21]). Each can also be altered significantly by integrating ICTs in
the classroom (although probably not all to the same extent).
o Several features of ICT can affect the way teachers provide feedback to
students, personalise instruction, develop collaborative projects and rely on
group-work assignments. Indeed, findings from previous PISA cycles show
that “students using ICT in mathematics class” are more likely to describe their
teachers as frequently using structuring practices (e.g. setting clear goals,
asking questions to verify understanding), student-oriented practices
(e.g. giving different work to students who have difficulties or who can advance
faster, having students work in small groups), formative assessments
(e.g. giving feedback on strengths and weaknesses), and cognitive activation
(e.g. proposing exercises that require students to apply what they have learned
to new contexts and/or proposing exercises that can be solved in different ways)
(OECD, 2016[22]).
o Although students’ ICT use in school is positively correlated with effective
instructional strategies in PISA, it is not clear how students use ICTs for
learning and, in particular, whether ICTs are used in ways that are related to
quality instruction. Detailed documentation of whether and how frequently the
instructional processes described above actually involve ICT would help fill
this knowledge gap.

Enabling environments for teaching and learning with ICTs

Teachers’ capacity to use ICT resources for teaching and learning depends on several
contextual factors and practices, which could be referred to as the enabling environment
for ICT use in school. In addition to the availability and quality of ICT resources (described
in the previous section), enabling factors include contextual information on students’
background, school-level policies and practices regarding the governance of ICT use for
learning (notably incentives and support for teachers), and teachers’ attitudes towards and
competencies in using ICT for teaching. The enabling environment partly determines
whether and how teachers use ICT resources in the classroom. The adequacy of that
environment can be assessed by asking teachers to report the extent to which these factors
aid or impede ICT use for learning.

3.3.2. ICT-related practices and policies at the school level

Although numerous aspects related to ICT use in school are decided at the national level,
schools often retain some leeway in organising the integration of ICTs into teaching.
Schools, for instance, often define the policies related to the use of ICTs within school to
support teaching and learning. They also play a role in determining how ICTs are used for
communication and for sharing information with teachers, parents and students and in
implementing assessments. Schools also play a role in defining the type of incentives given
to teachers to encourage them to adopt ICTs in their practices.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 31

School principals’ and teachers’ attitudes also play an important role in enabling the
integration of ICTs into school as these influence their general level of engagement. Indeed,
in the OECD Teaching and Learning International Survey (TALIS), teachers identify
“teaching with ICT” and “using new technologies in the workplace” as the second and third
most important professional development needs (OECD, 2014[15]).

Figure 3.2. Detailing ICT use in school

Source: OECD (2019), PISA 2021 ICT Framework.

3.4. ICT use outside of the classroom

Over the past decade, the number of 15-year-olds with access to the Internet has grown
(OECD, 2015[6]). The amount of time spent on the Internet outside school also increased
by 40 minutes between 2012 and 2015 to reach two and a half hours (OECD, 2017[18]).
Consequently, policy makers are expressing greater interest in understanding how students
engage with ICTs outside of the classroom and how their use of ICTs is affecting their
well-being, cognitive outcomes and acquisition of ICT skills. One of the important
advantages of integrating ICTs into the education system is bridging the divide between
school and home, and allowing for more continuity between the two.

3.4.1. ICT use for learning

Since teaching and learning is not limited to formal instruction in the classroom, the
PISA 2021 questionnaire framework considers students’ after-school opportunities to learn
as an integral part of education (OECD, 2018[23]). ICTs can be a catalyst for learning outside
the classroom, notably through their potential effect on students’ engagement with learning
activities and by providing a powerful tracking and monitoring tool for teachers
and parents.
In addition, the development of a wealth of digital learning resources, providing students
with more and better learning opportunities, such as educational games, Massive Online

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
32 │ EDU/WKP(2019)10

Open Courses (MOOCs), and a variety of topic-specific media content, such as video and
audio podcasts, tutorials, etc., may spark students’ interest in using ICTs for learning
outside the classroom.

3.4.2. ICT use for leisure

Most of the time students spend using ICTs outside the classroom is dedicated to leisure
activities. The extent to which ICT use for leisure is potentially related to students’
cognitive performance, ICT skills and well-being depends on the frequency, the diversity
and the type of activities students engage in (van Deursen and Helsper, 2015[24]). ICT use
for leisure provides an opportunity for students to acquire ICT knowledge and skills but
could also be a source of distraction.
As mentioned earlier, students’ use of ICT for leisure also involves risks, and is a source
of concern among parents and policy makers (Hooft Graafland, 2018[2]). Inappropriate or
unsafe Internet use can expose students to harmful content or to cyber-bullying. Students
also face an enormous amount of information online that might help them develop online
reading skills, but can also have adverse effects if the students are not able to distinguish
fact from fake news and verify their sources. Additional risks, such as overuse of video
games and compulsive use of social media, can have serious physical, social, psychological
and cognitive consequences (OECD, 2017[18]; Smith et al., 2008[25]; Currie et al., 2012[26]).

Figure 3.3. ICT use outside of the classroom

Source: OECD (2019), PISA 2021 ICT Framework.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 33

3.5. Digital competencies: Attitudes and dispositions

The PISA 2021 ICT framework aims to assess the relationship between students’ access to
and use of ICTs and three distinct outcomes: students’ cognitive achievement, students’
well-being and students’ competencies in ICTs. These three outcomes are defined based on
the PISA frameworks on cognitive achievement in mathematics, science and reading, as
well as student well-being. Moreover, students’ ICT competencies are defined in a broad
sense to encompass digital literacy as a specific domain as well as students’ attitudes and
dispositions towards ICT use in various contexts. While proposing a fully-fledged
assessment framework for digital (or ICT) literacy is beyond the scope of this work, a
roadmap for such an assessment is suggested.

3.5.1. Students’ cognitive achievement and well-being

PISA’s approach to measuring students’ cognitive achievement consists “in assessing not
only whether students can reproduce knowledge, but also whether they can extrapolate
from what they have learned and apply their knowledge in new situations. It emphasises
the mastery of processes, the understanding of concepts, and the ability to function in
various types of situations” (OECD, 2017[20]). Thus, rather than assessing mathematics,
science and reading per se, PISA aims to document mathematics literacy, science literacy
and reading literacy, where literacy refers to students’ capacity to apply knowledge and
skills in key subjects, and to analyse, reason and communicate effectively as they identify,
interpret and solve problems in a variety of situations (OECD, 2017[20]).
Adolescents’ well-being can be defined as the quality of students’ lives and their standards
of living. Well-being is a multi-dimensional construct with both objective, material
components and subjective, psychological facets. The PISA 2018 framework for the
assessment of well-being integrates these different perspectives. In addition to students’
overall perceived quality of life or life satisfaction, the well-being framework covers three
other dimensions, each of which incorporates both objective and subjective components:
self-related well-being, well-being in school environments, and well-being outside of
school environments (OECD, 2016[22]).

3.5.2. Students’ digital literacy

The growing importance of students’ ICT literacy is reflected in the frequent inclusion of
a variety of ICT competencies in curricula (European Commission, 2013[14]). As measures
of ICT competencies become more widely recognised, education systems are shifting from
teaching ICT skills in isolation towards a more horizontal approach, integrating specific
ICT tasks and competencies across subjects (European Commission, 2013[14]). This
highlights the crosscutting and complex nature of ICTs, which are often used as a tool to
support instruction, but are also recognised as a subject of learning in themselves.
Although the PISA 2021 ICT framework does not provide a full-fledged assessment of ICT
competencies, it proposes the foundations for integrating ICT literacy as a specific domain
in future PISA cycles. It relies on existing assessments of ICT literacy to identify the main
methodological challenges and key competency areas that should guide the development
of such an assessment.
The feasibility study for the PISA ICT literacy assessment defines ICT literacy as “the
interest, attitude and ability of individuals to appropriately use digital technology and
communication tools to access, manage, integrate and evaluate information, construct new
knowledge and communicate with others in order to participate effectively in society”

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
34 │ EDU/WKP(2019)10

(Lennon et al., 2003[27]). This definition shares many similarities with the approach
developed in other ICT literacy assessment framework, such as the International Computer
and Information Literacy Study (ICILS) and Australian Curriculum, Assessment and
Reporting Authority (ACARA) ICT Literacy, among others (Fraillon, Schulz and Ainley,
2013[28]; Fraillon et al., 2015[29]).
A comprehensive approach to assessing ICT competencies should focus on five main
competency areas (Redecker, Punie and European Commission. Joint Research Centre.,
2017[13]; Fraillon, Schulz and Ainley, 2013[28]; Fraillon et al., 2015[29]):
 accessing, evaluating and managing information and data
 sharing information and communicating
 transforming and creating digital content
 individual and collaborative problem solving in a digital context and computational
thinking
 appropriate use of ICTs, which embeds knowledge and skills related to security,
safety and risk awareness

Competency area 1: Accessing, evaluating and managing information and data

Accessing information and data focuses on the extent to which individuals can identify the
desired information, data or digital content, and understand how to find and recover
computer-based information from various sources by using ICTs (ACARA, 2014[30];
Fraillon et al., 2015[29]). Evaluating information and data is an integral step in accessing
information and even more so with the development of search engines and artificial
intelligence. This involves the process of filtering through multiple information sources,
and assessing their relevance, integrity and usefulness (Fraillon, Schulz and Ainley,
2013[28]; ACARA, 2014[30]).
Managing information and data refers to the ability to organise and store various types of
digital information (ACARA, 2014[30]). It involves the ability to adopt and develop systems
for organising and classifying information in such a way that the information can be
retrieved and reused efficiently (Fraillon, Schulz and Ainley, 2013[28]).

Competency area 2: Sharing information and communicating

Sharing information and communicating refers to one’s ability to exchange information,
share knowledge, and customise such communication for a specific audience, context and
medium (Fraillon, Schulz and Ainley, 2013[28]; ACARA, 2014[30]). This includes detailed
knowledge regarding the real and digital contexts in which information is shared and thus
requires awareness about ICT-based communication platforms, including e-mail, instant
messaging and group chat, media sharing and social-networking websites, among others.
Given the wide range of ICTs for communication, those who want to communicate
effectively with ICTs need to understand information-based social conventions and be able
to adapt and modify selected modes of communication depending on the intended
recipients.

Competency area 3: Transforming and creating information and digital content

Transforming and creating information involves the use of ICTs and ICT-based data, digital
content and information to develop new information or knowledge. Successful individuals

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 35

can take existing information and derive new understandings by adapting, applying,
designing, inventing or authoring (Fraillon, Schulz and Ainley, 2013[28]). Individuals may
transform information with ICTs, either to produce or expand upon existing information,
by modifying its presentation for improved understanding in specific contexts. This process
often requires the ability to use ICT-based formatting, graphics and multimedia to simplify
and enhance the communication of information. Information transformation and creation
are also related to the quality of information, specifically with regards to how structure,
layout and design are used to support overall comprehension.

Competency area 4: Problem-solving in a digital context and computational

thinking
PISA 2012 defines problem solving as individuals’ capacity to engage in cognitive
processing to understand and resolve problem situations where a solution is not
immediately obvious. In the context of ICT literacy, the focus should be on solving
technical problems, identifying technical responses and solutions, and creatively using
digital technologies to solve a problem.
According to ICILS 2018, computational thinking can be defined as the “ability to identify
a problem, break it down into manageable steps, work out the important details or patterns,
shape possible solutions and present these solutions in a way that a computer, human or
both can understand” (IEA, 2017[31]). Although computational thinking and problem
solving in a digital environment strongly overlap and share many thought processes, one
key difference can be that computational thinking focuses on how to rely on digital and
computing possibilities to solve problems.

Competency area 5: Appropriate use of ICTs (online security, safety and risk
awareness and skills)
Online safety and security issues incorporate the appropriate use of ICTs across multiple
contexts and platforms. Using ICTs appropriately requires making critical and thorough
assessments of ICTs use while considering the social, legal and ethical issues in different
settings (Fraillon et al., 2015[29]). With increased information sharing, students must be
aware of methods for handling and protecting information.

3.5.3. Students’ attitudes and dispositions towards ICTs

The assessment of students’ and potentially parents’ ICT-related attitudes and dispositions
rely extensively on existing measures developed for previous PISA cycles. More precisely,
it follows the PISA 2021 taxonomy, which revolves around six dimensions: i) attitudes;
ii) values and beliefs; iii) task performance; iv) emotional regulation; v) collaboration;
vi) and open-mindedness and engagement with others (OECD, 2018[23]).
Although all dimensions listed above are not equally relevant to ICT use, they are related
to ICT literacy in two different ways. First, students’ attitudes, behaviours, beliefs and
aspirations related to ICTs are likely to be correlated with ICT literacy and students’ ability
to use ICTs for learning and leisure. This suggests exploring how students feel or behave
when using ICTs in general, but also when using ICTs in specific contexts, including
learning and leisure.
Second, the use of ICTs for teaching and learning can alter students’ attitudes and
dispositions towards learning in general or for a specific subject. This is often one of the

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
36 │ EDU/WKP(2019)10

reasons why ICTs are used for learning. These two relationships between students’ ICT use
and dispositions are investigated in the following ways:
Self-efficacy refers to students’ beliefs regarding their ability to execute a specific task or
to achieve a given goal. Research suggests that higher levels of ICT self-efficacy are
associated with higher levels of learning outcomes (Fraillon et al., 2014[4]). In the absence
of a proper assessment of ICT competencies, self-efficacy constitutes the primary source
of information about students’ ICT skills. It would therefore be of great value to ask
students to evaluate their own abilities based on a set of tasks and situations that reflect the
five competency areas mentioned above.
Interest, enjoyment and intrinsic motivation in a particular subject are in general
positively associated with learners’ achievements in that subject. Results from the
International Computer and Information Literacy Study (ICILS) 2013 suggest similar
conclusions for ICTs. In parallel, research suggests that ICT use in the classroom can also
affect students’ motivation and interest in learning a specific subject (Lajoie and Azevedo,
2006[32]).
Emotional regulation and task performance cover aspects of students’ emotions and
emotional control (i.e. their capacity to curb anxiety, handle stress, develop and maintain
positive expectations, etc.), and aspects related to students’ diligence and commitment,
including setting high standards, working hard and avoiding distractions (OECD,
forthcoming[7]). Knowing whether students are anxious or stressed when using ICTs, and
whether they are committed to understanding how to conduct specific tasks with ICTs in
different contexts would provide insights into their abilities to use ICTs, particularly for
learning purposes.
In addition, emotional-regulation and task-performance items could be constructed with
the intention to document students’ risky behaviours with ICTs. Notably, aspects related to
students’ self-control, dependence and abilities to regulate their engagement in specific ICT
activities (including addiction and overuse of ICTs).
Collaboration, open-mindedness and students’ engagement with others cover students’
approaches to connecting with other people and the perceived value of those connections;
openness to new experiences, perspectives and eagerness to learn and experience; and
enjoyment of social connections and assertiveness in voicing their own views (OECD,
2018[23]).
Metacognition refers to students’ knowledge of learning strategies for a specific subject.
For example, metacognition in reading refers to students’ awareness and ability to use a
variety of appropriate strategies when processing texts in a goal-oriented manner (OECD,
2009[33]). Metacognitive reading strategies have been positively associated with students’
reading proficiency (Waters and Schneider, 2010[34]; OECD, 2017[20]). When ICTs serve as
a means to learn reading, science or mathematics, students are exposed to new learning
strategies and practices. Thus, it seems important to document students’ awareness about
the effectiveness of ICT-based learning techniques.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 37

4. Upgrading the ICT questionnaire items for PISA 2021

The development of the ICT framework serves as a basis for a thorough revision of the
scope and focus of ICT-related questions for PISA 2021. The lessons drawn from the
analysis of previous PISA cycles (Section 2), and from an extensive review of the academic
literature, and policy reports and surveys on ICTs in education, guided the selection of key
policy issues to document in PISA 2021. Under the guidance of the ICT expert group, this
revision process was perceived as an opportunity to consider the ICT-related questions with
a fresh eye, with the objective of providing a new, forward-looking model for the ICT
questionnaire and its future iterations.
This section details the main changes to the questionnaire and explains the reasons guiding
them. The revisions concerned the three main areas of the questionnaire: access to ICTs,
use of ICTs and students’ self-reported proficiency in the use of ICTs.
Three major constraints affect the revision of the ICT questionnaire items. First, limited
space and time is available for ICT questions in the ICT familiarity and the student, parent,
teacher and school questionnaires. Administering the ICT familiarity questionnaire should
not take more than 15 minutes in the field trials and no more than 10 minutes during the
actual data collection. Moreover, since ICT is one among many areas to cover in the
student, parent, teacher and school questionnaires it is only possible to include a limited
number of ICT-related items in these questionnaires. Second, the ICT familiarity
questionnaire should maintain a certain degree of continuity over time, particularly to show
trends on key measures; at the same time it should give priority to new developments
affecting the use and spread of ICTs in education. Therefore, the relevance of a question
was the main criterion in the decision to keep or drop an item and was given precedence
over the opportunity to preserve trends. Third, the questionnaire should remain relevant to
a large number of countries and economies with various ICT experiences and
environments.
Engaging in an in-depth revision of ICT items for PISA 2021 provided useful lessons that
can guide the development of ICT questions for future cycles. A structural challenge for
the development of the ICT questionnaire is to keep up with a fast-changing and
unpredictable digital environment. The schedule for developing new ICT questions and
items is slower than ICT evolution. Therefore, the ICT familiarity questionnaire had to
identify and prioritise documenting stable and long-lasting ICT constructs. The
pervasiveness of ICTs in education also raises questions on the role of the ICT
questionnaire in future PISA cycles. The ICT familiarity questionnaire could become a
laboratory for uncovering innovative uses of ICTs for learning and teaching while
mainstream aspects of ICT in education would enter the core contextual questionnaires as
they become universal.

4.1. Access to ICTs

With the rapid development of new digital technologies, documenting the availability of
ICT resources for students in 2021 proved challenging and required making some important
changes in the approach to the ICT questionnaire. Importantly, the mere availability of
digital resources was deemed insufficiently informative about the digital practices and
needs of the 21st century student. Therefore, the questionnaire had to go beyond
availability to put the emphasis on ICT use. Moreover, the range of ICT resources

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
38 │ EDU/WKP(2019)10

documented in the questionnaire had to be adjusted and complementary information on the

“quality of resources” was sought.

4.1.1. Availability and use to be documented simultaneously

In previous versions of the questionnaire, students were asked whether a list of ICT
resources (or digital devices) is “available to them for use”. The analysis of PISA data
revealed that having access to ICT resources (whether at school or at home) does not
necessarily imply that students actually use them (OECD, 2015[6]). Moreover, in a context
of universal availability of ICTs, access to digital resources is less of a constraint
for students.
As a result, the new questionnaire focuses on the frequency with which students use a set
of digital resources at and outside of school while including a response option (i.e. “This
resources is not available to me”) to document the availability of the ICT resource. This
option also has the advantage of reducing the space devoted to a full set of ICT availability
questions. The cost of this change is that the new question is slightly more complex and
would increase the cognitive load on students to answer the questionnaire.

4.1.2. Focus on main digital devices and educational digital resources

The scope of ICT resources investigated in the questionnaire changed in two ways: It
narrowed the coverage to document only students’ use of generic digital resources and gave
special attention to “digital learning resources” expected to be particularly relevant to
students’ learning outcomes.
Instead of aiming at full coverage of a variety of ICT resources, the PISA 2021 ICT
familiarity questionnaire focuses on generic digital devices and technologies. It documents
the availability and use of “desktop and laptop computers”, “smartphones”, “the Internet”,
“tablet devices or e-book readers” and “video games”.
This choice was guided by several considerations:
 In a context of rapid technological innovation and integration of several digital
technologies into single devices, it becomes increasingly difficult to interpret the
results based on how frequently students use each device or generation of
technologies. For example, what does infrequent use of portable music players
reveal about a student in the era of smartphone technology? Similarly, how could
one interpret a student’s intense use of a desktop computer when laptops are the
universal option? Since the primary objective of PISA is not to report the evolution
of students’ use of different technologies but rather to document how ICTs are
related to learning outcomes, there is little additional value in documenting
students’ use of specific devices compared with the general availability and
frequency of use of generic ICT resources.
 Moreover, given the rapid evolution of digital technologies and practices, it is not
possible to identify the digital resources that will make a significant difference
years from now.
 Generating a classification of digital devices and resources is cumbersome as
similar devices can be used in different ways and different devices can serve a
similar purpose.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 39

 Given the limited space available in the questionnaire, investigating students’

practices with digital resources rather than how frequently they use a specific
device was deemed more cost-effective.
However, the questionnaire highlights digital learning resources. Those specific resources
are of special interest compared with other digital resources because they might have a
strong effect on students’ learning outcomes. Moreover, documenting the extent to which
students engage with digital learning resources at school and at home could inform how
much a country’s student population relies on those new learning technologies. Therefore,
the questionnaire documents three broad types of digital learning resources: “school
portals”, “educational software, apps and games” and “learning management systems”.

4.1.3. Accessibility and quality of resources

Another consequence of the rapid technical change and expansion of ICTs is the
diversification of ICTs in terms of quality. Having access to the latest generation of
computers – that embed brand new technologies and allow using the latest software without
slackening – improves learning possibilities far beyond what students could do with slow
and outdated computers. Nowadays, the major differences between students, schools (and
countries) revolve around issues of quality rather than quantity of ICT resources. In this
context, quality not only refers to the technical capacities of ICTs but also include aspects
such as whether the resources are fit for purpose and flexible enough to be used in various
contexts. Moreover, the degree of accessibility of ICT resources at school is also important
in determining how they are used to support learning (Redecker, Punie and European
Commission. Joint Research Centre., 2017[13]). For example, PISA 2015 data shows that
out of the 81% of students across OECD countries with access to a desktop computer at
school, only 58% actually use it.
Thus, the PISA 2021 ICT familiarity questionnaire investigates the quality of students’
access to ICTs at school. It does so by asking students’ their opinion about different quality-
related dimensions:
 the technical capacity of ICT resources, covering aspects such as the speed of the
Internet connection, whether digital resources are connected to the Internet or
function properly
 the modality of access to ICTs, comprising the availability of sufficient digital
resources per students, but also whether the digital resources are easily available
within the classroom
 the relevance of ICTs and overall guidance on how to use them, which documents
whether the students find digital learning resources engaging, whether they have
sufficient support to use ICTs and consider teachers to be motivated and skilled to
use ICT resources
Adding this new question in the ICT familiarity questionnaire helps document important
policy topics. It provides information on the association between the quality of ICT
resources and students’ learning outcomes, which could subsequently inform policy on how
education authorities should invest in and select ICT resources. In particular, it could help
answer the question: to what extent should authorities invest in providing average ICT
resources or providing less ICT resources but of better quality? Moreover, teachers and
students might also be more willing to use ICT resources for learning purposes if they meet
higher quality standards.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
40 │ EDU/WKP(2019)10

4.2. Use of ICTs

A critical finding from the literature on ICTs and education is that increasing students’
access to and use of ICTs does not generally improve their cognitive performance. There
is a consensus that the potential positive effect of ICT use on learning depends on how
students use ICTs. The analysis of PISA data from previous cycles is in line with these
conclusions. Therefore, documenting in detail how students use ICTs, particularly for
learning, constitutes the major additional contribution of the revised PISA ICT familiarity
questionnaire. Two components of ICT use have been updated in the questionnaire: the
intensity of use in different places and for different purposes, and the modality of ICT use
for supporting teaching activities.

4.2.1. Intensity and frequency of use in different contexts

An important lesson from PISA 2015 is that students spend an increasing amount of time
using ICTs. While most students already have access to ICTs, one can expect the intensity
of ICT use to keep increasing in the coming years. As such, it is very important to track the
evolution of ICT use across countries and ensure that the trend is carefully monitored over
PISA cycles.
Yet, the previous measurement of the time spent online or using digital resources suffered
from an important drawback. Although the latest versions of the ICT questionnaire
documented the intensity of ICT use for different tasks, it did not distinguish systematically
between ICT use for learning or for leisure. Students’ constant access to the Internet
requires making this distinction in order to capture students’ ICT use in more detail. Indeed,
in some countries students are increasingly using ICT resources during class for leisure.
The two types of ICT use in class might have opposite effects on learning outcomes and
could explain the unclear connection between ICT use and students’ performance.
Thus, the new ICT questionnaire was developed in several ways to account for the
evolution of ICT practices:
 With the increasing amount of time students spend using digital resources, new
categories of ICT use were added to the response scale in order to accurately
capture higher levels of ICT use.
 The coverage of the time students spend using digital resources now distinguishes
between ICT use for learning and leisure activities at school, before and after school
and on weekends.2 This allows for tracking how long students use ICT resources in
total and therefore maintains a trend with previous PISA cycles while
distinguishing between the different types of use. This question is included in the
student main questionnaire rather than the ICT familiarity questionnaire. The
former covers all participating countries and economies.
 The previous version of the ICT questionnaire documented in detail how frequently
and how long students used digital resources in different subjects. The updated
version of the questionnaire focuses only on the frequency of ICT use in the
subjects assessed in PISA (i.e. mathematics, science and reading). While this

2
Note that the questionnaire does not distinguish between the use of digital resources at school
during class and after class as it might be too cumbersome for students to recall this information.
Yet, this distinction could be of importance if students’ use of ICT for leisure during class is
particularly detrimental to learning.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 41

version provides less information on the time students spend using ICTs in different
courses, it still documents the relationship between ICT use in a specific subject
and performance in that subject.

4.2.2. Detailed coverage of learning activities with ICTs

As mentioned above, PISA data reveal an ambiguous bell-shaped relationship between ICT
use and students’ cognitive performance. Students’ performance in mathematics, science
and reading is the highest when students use ICTs moderately (everything else held equal).
The results hold for both ICT use at school and outside of school. One of the main
objectives for the revision of the PISA ICT familiarity questionnaire is to disentangle these
puzzling results. To do so, the selected approach will cover in detail how students use ICT
resources in relation to specific teaching practices.
Previous versions of the ICT familiarity questionnaire documented various types of ICT
tasks related to different learning contexts, such as “chatting on line”, “using e-mail”,
“browsing the Internet for schoolwork” and “practicing and drilling”. The revised
questionnaire takes a different route and focuses on the support function of ICTs in
learning. ICTs have a great potential for widening learning possibilities, by enabling
enquiry-based learning, providing more autonomy to students and facilitating group work,
formative assessment, and feedback and communication between students and teachers.
Thus, the revised questionnaire investigates whether students use ICTs in association with
specific teaching practices, including teacher-directed, student-centered and enquiry-based
teaching, and for teacher feedback. This approach has the advantage of linking ICT-related
practices to teaching strategies documented in the student and teacher questionnaires.
Students are asked how frequently they use ICTs while engaging with particular teaching
practices. The question on students’ use of digital resources for teacher-directed activities
focuses on mathematics (the core subject in PISA 2021) and is included in the core student
questionnaire. It includes activities such as using digital resources to “solve equations”,
“draw geometric figures or functions”, “read an explanation of mathematical concepts” and
“coding”. The ICT familiarity questionnaire investigates classroom activities related to
enquiry-based teaching, receipt of teacher feedback, formative assessment and school-
related activities after class. Enquiry-based teaching activities include items such as “create
a multi-media presentation”, “write or edit text for a school assignment”, and “analyse data
that you have collected yourself”. The activities do not specifically refer to mathematics
lessons and can be relevant to science and reading.
Another question documents modalities of teacher feedback. It provides information on
whether the students “read or listen to feedback sent by teachers, by other students or
automatically selected by a learning app” and whether students “revise their work based on
feedback” or “work on practice exercises using educational software”. The ICT familiarity
questionnaire documents another potentially important contribution of ICTs to education:
the capacity to enhance out-of-school learning in effective and innovative ways. Hence, the
questionnaire examines how frequently students “browse the Internet for schoolwork”,
“receive or download assignments or instructions from their teachers” and “communicate
with their teachers”.
Most of the time students spent using digital resources outside of school is dedicated to
leisure activities (OECD, 2017[18]). Using ICTs for leisure can also contribute to learning.
For example, social media provides students with opportunities to develop reading and
comprehension skills, and certain video games might encourage collaborative problem
solving. Frequent users of digital technologies may also become more efficient in using

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
42 │ EDU/WKP(2019)10

digital resources for learning purposes. However, excessive engagement in certain

activities might deter students from doing their homework or engaging in activities of
higher learning value. Therefore, the questionnaire examines how much time students
spend doing various digital activities ranging from playing video games, to browsing social
networks or looking for practical information online, for example, both during a weekday
and on a weekend day.

4.3. Attitudes and disposition towards ICTs

Most PISA-participating countries and economies are increasingly concerned with

developing students’ digital competencies so they will be able to thrive in the knowledge
economy. Indeed, most education systems include ICT or digital literacy in their national
curriculum, either as a stand-alone discipline or integrated within other subjects (Conrads
et al., 2017[3]). A successful 21st century student should be able to use ICTs to research,
manage, analyse and communicate information online, understand how to solve problems
and adopt new ICTs as they arise while ensuring that they use ICT resources safely.
Moreover, students should be able to shift between digital platforms and combine different
ICT tools to achieve their objectives. Previous versions of the ICT familiarity questionnaire
included basic questions about students’ attitudes, dispositions and self-efficacy in the use
of ICTs. The new ICT questionnaire develops questions on self-efficacy in order to reflect
the diversity of digital competencies identified in the ICT framework. Moreover, it
investigates students’ opinions about school rules for ICT use, their attitudes towards the
accuracy of information online, their experience with inappropriate digital content and their
general interest in acquiring ICT skills.

4.3.1. Students’ opinion about ICT use for learning

ICTs bring great learning opportunities to the classroom but also entail some risks. In
particular, students’ constant connection to the Internet and to social media via their
smartphones can interfere with their learning activities. When working with ICTs, students
might divert ICT resources from their initial learning purposes to either play, communicate
via social media or simply browse the Internet. They can also use their smartphones to
communicate at any time during class hours, thus creating noise and disorder. Since
distraction caused by ICT use during lessons is becoming an increasing concern, the ICT
questionnaire included a question on students’ opinions about school rules regarding ICT
use. The question provides information about the degree of autonomy students have when
using digital resources in class. For example, it asks students whether they should be
allowed to bring their mobile phones or laptops to class, whether the school should set up
filters on the Internet, and whether teachers and students should jointly decide on the rules
for using ICTs. Moreover, two items on the potential effects of ICT use have been included
in the classroom-climate question in the main student questionnaire.

4.3.2. Students’ exposure and response to ICT risks

Facing safety issues when using ICTs is an inevitable by-product of the increased exposure
to digital resources (Hooft Graafland, 2018[2]). The more time students spend online, the
more likely they are to encounter inappropriate digital content (i.e. violent or pornographic
content), to come across discriminatory content, and to face cyber-bullying. The ICT
familiarity questionnaire covers those risks and asks students whether they faced such
situations and how upset they were when they did. In this way, the questionnaire will

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 43

provide information on the frequency of student exposure to these situations and on the
emotional consequences of this exposure.
Today’s students also have to deal with a plethora of information coming from different
sources. Being able to assess the quality and accuracy of the different types of information
is one of the core digital competencies students should acquire. In order to address this
question, the ICT questionnaire examines how much students trust the information
available online, whether they have opportunities to discuss its accuracy, and how they
react when they encounter false or inaccurate information.

4.3.3. Students’ interest in ICTs and ICT self-efficacy

Ensuring that students possess an adequate set of digital skills has become an important
objective of education systems; yet, there are only few sources of information on the actual
performance of students in terms of ICT or digital literacy. The International Computer and
Information Literacy Study (ICILS) of the International Association for the Evaluation of
Educational Achievement (IEA) and ACARA’s ICT literacy assessment are among the few
nationally representative tests that rely on a well-defined assessment framework. The
DigComp.2.1 digital competency framework identified a set of digital competencies. The
PISA 2021 approach draws on this framework and proposes a self-efficacy assessment of
students’ digital competencies that encompasses a broad range of skills.
The self-efficacy questions included in the 2015 and 2018 versions of the questionnaire
were general in scope. They documented students’ overall confidence, independence,
interest and collaboration when using, helping relatives or learning about digital devices.
While these questions can be positively correlated with students’ digital abilities, they
mainly reflect their technical digital skills. The questionnaire items document whether
students feel comfortable “using digital devices”, “solving a problem with a digital device”,
“installing a software” and “selecting an application”.
Several components of digital competencies omitted in previous versions of the
questionnaire are now accounted for. The self-efficacy questions document students’
ability to perform a variety of tasks that refer to the competency areas identified in the ICT
framework. Students’ self-reported ability to “search and find relevant information on line”
and to “assess the quality of information” is document competency area 1 (i.e. accessing,
evaluating and managing information and data). Competency area 2 corresponds to the
ability to share information and communicate, and is documented by students’ perception
of their ability to “use digital resources to share practical information with a group of
students”, “use digital resources to collaborate with other students on a group assignment”
and “explain to other students how to share digital content on line or on a school platform”.
Competency area 3 corresponds to students’ ability to transform and create information and
digital content, and assesses students’ capacity to “edit digital photos or images”, “write or
edit text for a school assignment”, “collect and record data”, “create a multimedia
presentation” and “create, update and maintain a webpage”. Students’ self-efficacy in
problem-solving in a digital context and computational thinking (i.e. competency area 4) is
documented through students’ ability to “change the settings of a digital device to improve
the way it operates”, “select the most efficient program or app to carry out a specific task”
and “create a computer program”. Finally, students’ ability to use digital resources
appropriately (i.e. aware of online safety and security risks) is assessed by their ability to
“change the settings of a device or app in order to protect their data and privacy” and by
the question on their attitudes towards information quality descried in the previous
sub-section.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
44 │ EDU/WKP(2019)10

Defining relevant items for all aspects related to digital problem solving and computational
thinking proved to be challenging. A self-efficacy assessment requires simple but well-
defined tasks that would make sense to all students, regardless of their engagement with
complex digital tasks. Yet, computational thinking abilities are usually assessed in a context
and by a succession of activities. Thus, these aspects of digital competencies are not well-
reflected in the self-efficacy questions. Another important challenge when defining a self-
efficacy measure is to ensure that it is associated with students’ actual digital literacy.
Previous studies suggest that such measures might not be correlated with students’ ICT
skills (Fraillon et al., 2014[4]). The coverage of a relatively broad range of digital
competencies should reduce this risk. Moreover, a large number of items will be tested
during the field trials, and will eventually lead to the selection of the most relevant ones.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 45

5. Lessons for the future development of ICT questions in PISA

Findings from the analysis of ICT questions in previous PISA cycles, together with a
thorough review of the literature on the subject and the inputs from the expert group, guided
the development of the PISA ICT framework and subsequently the revision of the ICT
questions for PISA 2021. Overall, the new updated ICT questions are more reflective of
current digital practices and concerns. In particular, the PISA 2021 ICT familiarity
questionnaire accounts for the fact that access to ICTs is now almost universal, and
therefore the questions of interest for policy making revolve around issues of quality of
ICT resources and students’ use of those resources. The questionnaire also examines the
possible negative side effects of ICT use, students’ attitudes towards the accuracy of online
content and online risks, and their opinions about their own digital skills.
This in-depth revision process revealed opportunities and challenges regarding the future
development of ICT questions in PISA. In terms of opportunities, some of the important
dimensions identified in the PISA ICT framework are not covered extensively due to
constraints related to questionnaire length. For example, the selection and quality of digital
learning resources, the role of parents and teachers, and a full test-based assessment of
digital competencies might deserve more thorough coverage in future questionnaires.
Upgrading ICT items also unveiled significant challenges for documenting ICTs in the
context of PISA. The challenges include the rapid, unequal and irreversible expansion of
ICTs, the international nature of PISA, the long time needed to prepare for a cycle of data
collection and the difficulty to cover countries in different stages of ICT development.
The set of ICT questions selected for the PISA 2021 field trials does not cover all relevant
aspects of ICT in education and would need to be revised in the future to cover new
developments in the field. With the expansion of ICTs, access becomes less important and
the quality of resources takes central stage. In particular, future questionnaire items should
assess the quality of ICT resources and how they were chosen by schools and teachers. A
comprehensive assessment should examine the suitability of the digital resources for
learning, that is, whether they are adequate for diverse education settings and are fit for
purpose. For example, the Norwegian Centre for ICT in Education assesses the quality of
digital resources for education along three dimensions: the quality of the interface between
users and the resources (e.g. the digital resources should generate interest and be accessible
and inclusive); the possibilities and limitations of the digital resources (e.g. adaptability to
different contexts); and the education and evaluation potential (e.g. relevant for the
curriculum, enable relevant evaluation, suitable to different education contexts)
(Norwegian Center for ICT in Education, n.d.[35]).
Regarding ICT use, the modalities surrounding the use of digital resources for learning at
school could be further examined. For example, it would be interesting to understand
exactly how students use ICT resources during class. Do they use ICT resources many
times in a single lesson for various purposes at different times or do they have an entire
class during which they practice on digital devices? To what degree are students
independent when using digital resources? Do they follow the teachers’ instructions? Do
they have to solve problems by themselves, or in groups?
The PISA ICT framework highlights how ICTs can support teaching activities by enabling
parents and teachers to monitor students’ work both at and outside of school. There is a
great potential for ICTs to contribute to enhancing students’ engagement with their home

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
46 │ EDU/WKP(2019)10

assignments and ensuring continuity between school and home learning activities. These
types of practices could be investigated in more detail in future ICT questions as they are
likely to become very common in the near future. The role of parents in shaping their
children’s ICT practices is also important and is not investigated in detail in the
questionnaire. Moreover, ICT co-ordinators might play an increasingly important role in
schools by selecting and maintaining ICT equipment and by providing support and training
to teachers. Future questionnaires should reflect these organisational changes.
The increased use of digital learning tools is likely to trigger another phenomenon that
could affect education in the future: the harnessing of education data to enhance teaching.
Although collecting data on students raises ethical and security concerns, it is not surprising
that in the near future some education systems might decide to equip schools with
educational software to better track their students’ strengths and weaknesses.
An increasingly important concern for policy makers is students’ digital literacy.
Developing a proper assessment of students’ digital skills beyond self-efficacy could be
very valuable. Yet, as digital competencies become increasingly essential, developing an
optional assessment tool for one PISA cycle may not be the best option. Instead, digital
competencies could be tested as part of the mathematics, science and reading assessments.
Indeed, many components of digital literacy are similar to reading, science and
mathematics literacy in a digital environment. The assessment of digital reading in
PISA 2012 is a good example. Moreover, computational thinking involves processes that
are very close to problem solving and collaborative problem solving assessed in PISA.
Therefore, the suggested approach consists in developing test items within the assessments
of reading, science and mathematics that would cover digital literacy. This approach
coincides well with PISA’s definition of skills, which are more about practical knowledge
in real-life contexts.

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 47

References

ACARA (2014), National Assessment Program – ICT Literacy Years 6 & 10 Assessment [30]
Framework 2014, https://www.nap.edu.au/_resources/NAP-
ICT_Assessment_Framework_2014.pdf (accessed on 29 January 2019).

Bulman, G. and R. Fairlie (2016), Technology and Education: Computers, Software, and the [8]
Internet, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2777325 (accessed on
29 January 2019).

Campuzano, L. et al. (2009), Effectiveness of Reading and Mathematics Software Products, [12]
http://dx.doi.org/10.3201/eid1307.060537.

Conrads, J. et al. (2017), Digital education policies in Europe and beyond key design principles [3]
for more effective policies, https://ec.europa.eu/jrc/en/publication/eur-scientific-and-technical-
research-reports/digital-education-policies-europe-and-beyond-key-design-principles-more-
effective-policies (accessed on 29 January 2019).

Currie, C. et al. (2012), Social determinants of health and well-being among young people, [26]
http://www.euro.who.int (accessed on 30 January 2019).

Escueta, M. et al. (2017), Education Technology: An Evidence-Based Review, National Bureau [5]
of Economic Research , Cambridge, MA, http://dx.doi.org/10.3386/w23744.

European Commission (2013), Survey of Schools: ICT in Education - Benchmarking Access, Use [14]
and Attitudes to Technology in Europe’s Schools, http://dx.doi.org/10.2759/94499.

Fraillon, J. et al. (2014), Preparing for Life in a Digital Age, Springer International Publishing, [4]
Cham, http://dx.doi.org/10.1007/978-3-319-14222-7.

Fraillon, J., W. Schulz and J. Ainley (2013), International Computer and Information Literacy [28]
Study Assessment Framework, http://www.iea.nl (accessed on 29 January 2019).

Fraillon, J. et al. (2015), “National Assessment Program : ICT Literacy Years 6 and 10 Report [29]
2014”, ICT - Digital Literacy, https://research.acer.edu.au/ict_literacy/11 (accessed on
30 January 2019).

Hooft Graafland, J. (2018), “New technologies and 21st century children: Recent trends and [2]
outcomes”, OECD Education Working Papers, No. 179, OECD Publishing, Paris,
https://dx.doi.org/10.1787/e071a505-en.

IEA (2017), “What’s next for IEA’s ICILS in 2018?”, in The IEA’s International Computer and [31]
Information Literacy Study (ICILS) 2018, International Association for the Evaluation of
Educational Achievement (IEA), http://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf
(accessed on 30 January 2019).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
48 │ EDU/WKP(2019)10

Karam, R. et al. (2016), “Examining the implementation of technology-based blended algebra I [11]
curriculum at scale”, Educational Technology Research and Development, Vol. 65/2,
pp. 399-425, http://dx.doi.org/10.1007/s11423-016-9498-6.

Lajoie, S. and R. Azevedo (2006), “Teaching and Learning in Technology-Rich Environments”, [32]
in Handbook of Educational Psychology, Routledge,
http://dx.doi.org/10.4324/9780203874790.ch35.

Lennon, M. et al. (2003), “Feasibility Study for the PISA ICT Literacy Assessment: Report to [27]
Network A.”, Educational Testing Service, https://eric.ed.gov/?id=ED504154 (accessed on
30 January 2019).

Norwegian Center for ICT in Education (n.d.), Quality criteria for Digital Learning Resources, [35]
http://eqnet.eun.org/c/document_library/get_file?folderId=11090&name=DLFE-101.pdf
(accessed on 29 January 2019).

OECD (2018), PISA 2021 Mathematics Framework (First Draft), EDU/PISA/GB(2018)4, 45th [23]
meeting of the PISA Governing Board, Directorate for Education and Skills, Programme for
International Student Assessment, https://www.upc.smm.lt/naujienos/smm/penkiolikmeciu-
matematinis-rastingumas/GB-2018-4-PISA-2021-Mathematics-Framework-First-Draft.pdf
(accessed on 30 January 2019).

OECD (2018), PISA in Focus #83, https://www.oecd-ilibrary.org/docserver/1e912a10- [1]

en.pdf?expires=1548775159&id=id&accname=ocid84004878&checksum=CD46091CFF6D
EAE7985BF0545A3B5A88 (accessed on 29 January 2019).

OECD (2017), PISA 2015 Assessment and Analytical Framework: Science, Reading, [20]
Mathematic, Financial Literacy and Collaborative Problem Solving, PISA, OECD
Publishing, Paris, https://dx.doi.org/10.1787/9789264281820-en.

OECD (2017), PISA 2015 Results (Volume III): Students’ Well-Being, PISA, OECD Publishing, [18]
Paris, https://dx.doi.org/10.1787/9789264273856-en.

OECD (2016), PISA 2018 Draft Analytical Frameworks, https://www.oecd.org/pisa/data/PISA- [22]

2018-draft-frameworks.pdf (accessed on 30 January 2019).

OECD (2015), Students, Computers and Learning: Making the Connection, PISA, OECD [6]
Publishing, Paris, https://dx.doi.org/10.1787/9789264239555-en.

OECD (2014), TALIS 2013 Results: An International Perspective on Teaching and Learning, [15]
TALIS, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264196261-en.

OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, [21]
Science, Problem Solving and Financial Literacy, PISA, OECD Publishing, Paris,
https://dx.doi.org/10.1787/9789264190511-en.

OECD (2009), PISA 2009 Assessment Framework - Key competencies in Reading, Mathematics [33]
and Science, OECD Publishing, https://www.oecd.org/pisa/pisaproducts/44455820.pdf
(accessed on 30 January 2019).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified
 EDU/WKP(2019)10 │ 49

OECD (forthcoming), Skills Outlook 2019. [7]

Pane, J. et al. (2013), Effectiveness of Cognitive Tutor Algebra I at Scale, [10]

http://www.siia.net/visionk20/files/Effectiveness%20of%20Cognitive%20Tutor%20Algebra
%20I%20at%20Scale.pdf (accessed on 29 January 2019).

Redecker, C., Y. Punie and European Commission. Joint Research Centre. (2017), European [13]
framework for the digital competence of educators DigCompEdu.,
https://ec.europa.eu/jrc/en/publication/eur-scientific-and-technical-research-reports/european-
framework-digital-competence-educators-digcompedu (accessed on 29 January 2019).

Roschelle, J. et al. (2016), “Online Mathematics Homework Increases Student Achievement”, [9]
AERA Open, Vol. 2/4, p. 233285841667396, http://dx.doi.org/10.1177/2332858416673968.

Scherff, L. and C. Piazza (2008), Why Now, More than Ever, We Need to Talk about Opportunity [16]
to Learn, International Literacy AssociationWiley, http://dx.doi.org/10.2307/40058135.

Schmidt, W. and A. Maier (2009), “Opportunity to learn”, in Sykes, G. et al. (eds.), Handbook of [17]
education policy research, Routledge, New York.

Smith, P. et al. (2008), “Cyberbullying: its nature and impact in secondary school pupils”, [25]
Journal of Child Psychology and Psychiatry, Vol. 49/4, pp. 376-385,
http://dx.doi.org/10.1111/j.1469-7610.2007.01846.x.

Trucano, M. (2005), “Knowledge Maps: ICTs in Education - What Do We Know about the [19]
Effective Uses of Information and Communication Technologies in Education in Developing
Countries?.”, Online Submission, https://eric.ed.gov/?id=ED496513 (accessed on
29 January 2019).

van Deursen, A. and E. Helsper (2015), “The Third-Level Digital Divide: Who Benefits Most [24]
from Being Online?”, http://dx.doi.org/10.1108/S2050-206020150000010002.

Waters, H. and W. Schneider (2010), Metacognition, strategy use, and instruction, [34]
https://www.guilford.com/books/Metacognition-Strategy-Use-and-Instruction/Waters-
Schneider/9781606233344 (accessed on 30 January 2019).

UPGRADING THE ICT QUESTIONNAIRE ITEMS IN PISA 2021

Unclassified