15 Years of (Who)man Robot Interaction: Reviewing the H in Human-Robot Interaction

This article paper is structured as follows. We first aim at establishing a common ground with the reader by defining what is meant with the words “gender”, “sex”, and “bias” in the current article (Section 1.2) before stating our research questions (Section 1.3). In Section 2, we then present related work on the topics of gender, diversity and critique of HRI research practice. Section 3 describes the research participation dataset we created (and how we did so) in order to examine research participation in studies published at the HRI conference to date. Section 4 presents the complementary survey we administered to HRI researchers in order look for any links between researcher identity, HRI sub-topic and research participation. Our findings are presented, with signposting to our research questions, in Section 5. We discuss key implications from these findings in Section 6, resulting in a summary of practical suggestions (at the individual researcher and research community levels) in Section 6.3. Finally, we provide a brief summary conclusion in Section 7.

Bias is a word that has many meanings in different fields, so for clarity, we here outline what we mean by gender, and gender bias. When defining gender, it is important to note that sex and gender are separate things, but frequently conflated. Offenwanger et al. [59], referred readers to the definition provided by the Canadian Institutes of Health Research, which clearly differentiates the two: Sex refers to a set of biological attributes in humans and animals. It is primarily associated with physical and physiological features including chromosomes, gene expression, hormone levels and function, and reproductive/sexual anatomy. [...] Gender refers to the socially constructed roles, behaviours, expressions and identities of girls, women, boys, men, and gender diverse people. It influences how people perceive themselves and each other, how they act and interact, and the distribution of power and resources in society [1]. This is consistent with definitions from research institutes in other governments such as Sweden [3], where the majority of the authors of this article are currently based. In this work, we refer to gender and gender representation, rather than sex, but acknowledge that (a) sex is also relevant, as sex-related differences (that in some cases might correlate with gender) can impact research results and (b) researchers might be conflating sex and gender when reporting participant demographics, exasperated by the fact that so few articles specify how questions surrounding gender and/or sex were asked (see related commentary under Section 6.2).

As per Offenwanger et al. [59], we knew that it was unlikely we would be able to identify exactly how different authors were operationalising gender, i.e., did they ask participants about gender, or sex? How was this question asked, if multiple choice, what answer options were given? Previous work on gender sensitivity (or a lack thereof) in HCI research has previously called attention to this issue, and called for researchers to be upfront in their understanding of gender when writing about it [18]. In our case, we also want to state our definition of gender before we are faced, in the literature we review, with conceptions of gender that might be incompatible with it because if we do not, our article reinforces a conception of gender that we do not agree with. As such, we also extend our definition of gender to include that [g]ender identity is not confined to a binary (girl/woman, boy/man) nor is it static; it exists along a continuum and can change over time. There is considerable diversity in how individuals and groups understand, experience and express gender through the roles they take on again per the Canadian Institutes of Health Research [1] and again in line with information from the national health service in Sweden.⁵ Whilst some fields typically group gender variables with sexual orientation under the umbrella term Sexual Orientation and Gender Identity (SOGI), we saw no evidence of this in the HRI literature and limit our discussions here to sex and gender only.

We want to explicitly affirm our viewpoint that gender goes beyond binary, even though we have not always been able to treat it as such in this work due to the historically binary assumption/treatment of gender evidenced in those papers we’ve reviewed, and in (some of) the diversity measures we have replicated (i.e., Offenwanger et al. Distance from Even Representation measure [59], see Section 2). The binary assumption of gender exists both in government systems [37, 75] and in research, and causes problems for many reasons, among them the difficulty of tracking the participation of non-binary people in research [59]. While this binary treatment is now widely acknowledged to be problematic, and steps are being taken to rectify it legally [2, 3, 43, 76, 79, 80] and in research [50, 71], this is a slow process, and we are still confronted with binary treatments of gender.

With regard to the definition of gender bias, previous definitions include: any set of attitudes and/or behaviors which favors one sex over the other [10, p. 83] and: a systematically erroneous gender dependent approach related to social construct, which incorrectly regards women and men as similar/different [66, p. ii46]. In the first instance then, when we speak of gender bias in research participation, we are talking about systems and practices that result in an over-representation of one gender over others. The latter definition alludes then to biases arising from failing to account for gender differences, or from looking for (and perhaps post-hoc rationalising) gender differences where one should not—a theme we discuss in the context of when (not) to do gender analyses and the seemingly divergent views HRI researchers hold on this topic.

This article aims to investigate gender representation and diversity in HRI research participation and factors which might influence it. To this end, we address the following questions, via a systematic review of HRI literature and a survey of HRI researchers. The literature review represents an (extended) conceptual replication of Offenwanger et al. HCI work [59]. Based on their findings and resultant hypotheses, we examine variations in research participation across sub-topics of HRI and reported participant recruitment practices, both of which Offenwanger et al. linked with variation in gender bias. The survey adds value to this review, first and foremost, by allowing for examination of Offenwanger et al. hypothesis that variation in researcher identity might correlate with diversity in research participation.

We further take the opportunity to reflect on what HRI researchers are doing with gender data by asking:

Together, these questions represent a thorough reflection on the who, how and why of HRI user studies, which are a cornerstone of HRI research. As we discuss in more detail later (Section 5), the majority of HRI conference papers each year report some sort of user study, by which we mean any study including human subjects, therefore including e.g., design studies and basic usability testing.

Relatively few works in HRI explicitly set out to investigate how user gender may influence HRI: only 31 of the 684 papers we reviewed were identified as addressing explicit research question(s) and/or testing specific hypotheses regarding user gender. However, a larger number of papers (n = 64) were identified as including gender in their analyses e.g., as a confounding variable. Across all of these works, just over half (n = 50) reported a significant effect of user gender with the remaining (n = 45) reported the lack of such (see full results under Section 5.3). Many such analyses pertain to whether perceptions of a particular robot, robot behaviour or application vary across participant gender (e.g., [5, 21, 68, 72]) but others pertain to behavioural differences (or lack thereof) when interacting with a robot (e.g., [30, 31, 54, 74]). As such, previous literature paints a mixed picture with regards to if, when, and how a person’s gender might directly impact their experience of HRI in the context of these HRI experimental settings and measures.

Given the relatively small overall number of papers discussing gender in our corpus (a total of 95 out of 684) as well as potentially divergent views within the HRI community (see further comments on this under Section 6.2.1) a broader question might be: (when) is gender analysis appropriate in the first place? As previously noted, Tannenbaum et al. claim that, per the title of their Nature perspectives article, “sex and gender analysis improves science and engineering” [85]. The authors draw attention to the necessity of gender and sex analysis for equality e.g., in designing for physical safety (car safety devices which account for sex differences in physical size) and reducing gender bias in AI systems (persons in a kitchen being default tagged as women by image labelling systems). Notably, however, on this last point, they note that a first challenge in algorithmic bias is to identify when it is appropriate for an algorithm to use gender information. In some settings, such as the assignment of job ads, it might be desirable for the algorithm to explicitly ignore the gender of an individual as well as features such as weight, which may correlate with gender but are not directly related to job performance. However, others might argue that gender should explicitly be considered in the context of job applications (although arguably by trained human recruiters, rather than any sort of algorithmic system) as a way to account for effects of gender bias in the workplace [65] and/or to support affirmative action policies which work towards diversity goals of an organisation [86, ch. 7].

This importance of context over categorisation is taken up in another recent Nature perspectives article which calls for researchers to move away from social categories (including e.g., race and gender) and instead consider social context [23]. The authors Cikara et al. note that gender and racial or ethnic categories are often treated as purposive social groups, as though tagging a person with a category comprehensively capture their thoughts, feelings and perceptions [...] Repeatedly relying on categories in research gives rise to illusory essences—the notion (even among experts) that these categories represent objective, definable and fixed constructs—which in turn, reifies the categories. A better question then might be: how can we conduct gender sensitive research in HRI? Here we can take inspiration from our colleagues in HCI who have a head start on us in critically considering these issues, and we want to spotlight Burtscher and Spiel’s article on this topic as an excellent starting point [18].

Regardless of the above, diversity in research participation is important from both a technical and ethical perspective with regards to delivering effective HRI that has positive impact in the real-world. As summarised by Offenwanger et al. [59], failure to ensure diversity (with respect to gender, in the first instance) can have a negative impact on underrepresented populations through unintentional exclusion [62, 87], which can lead to role stereotyping [13, 77] and new technologies being difficult to use for underrepresented groups [4, 16, 53, 56, 70, 82]. Under-representation can be especially problematic for people with non-normative gender identities, as exclusion can be easily caused by inadequate analysis tools. Most gender analyses, particularly statistical analyses, are confined to only analysing men and women, and it is not unusual for (even the most well-intentioned) researchers to exclude participants with non-normative identities due to sample sizes being too low for inclusion in statistical tests. Even the Distance from Even Representation (DER) measure used by Offenwanger et al. to illustrate the under-representation of women in HCI studies “is confined to comparing the representation of men and women and cannot be used to analyse more complex gender representation” [59]. Alternative metrics such as diversity measures from ecology [33, 41] have been proposed as a way to address this problem, and we discuss the practical applicability of these under Section 2.3.

HRI researchers have noted the importance of conducting user and design studies with diverse sets of participants that reflect the eventual target population [14, 67]. The publications in the first seven editions of the HRI conference had, however, an over-representation of men among their participants [89]. To our knowledge, this is the only equivalent previous work specifically concerned with examining gender representation in HRI research participants. However, HRI researchers have been very active in critiquing the field’s typical research practices. For example, work has been done on examining author diversity [20], classifying HRI publication methodology [9], reviewing data collection tools [22, 74], calling out the need for replication of studies [44], and pointing out a troubling lack of reporting with regards to ethics practices [64], recruitment, compensation, and participant gender [24].

Moving from HRI to our neighbouring field of HCI, previous works have examined researcher diversity [11, 41, 92] and its relation to publishing [57], evaluation methods in HCI [7], participant sample sizes [19], participant gender [18, 59, 71], and intersectionality [18, 73]. Existing guidelines for collecting and reporting participant gender [71] and commentary on how researchers might practice (gender) sensitive research [18] are essential reading for those in HRI interested in this topic, and very much inform the recommendations we later present under Section 6.3. In the fields of computer science and AI more broadly, initiatives like the UNESCO report on the digital gender divide [90] and UNICEF’s policy guidance on the development of AI for and with children [28] further call for gender diversity in those involved with the development and evaluation of AI-enabled systems.

Diversity is notoriously difficult to define, partly due to its close relation to complexity, and to the hierarchical interdependence and fuzziness of many categories [49, 60]. Although no definition is perfect there are many useful ways to define diversity in measurable ways, whether within or between categories.

To facilitate monitoring of diversity and representation in scientific communities several ways of quantifying diversity have been considered. Inspired by ecology, a Gender Diversity Index (GDI) has been proposed [33, 41], in which weighted Shannon indices [81] were calculated to measure gender diversity of keynote speakers, authors and conference organisers. The proposed benefit of the GDI is that one number in the range [0;1] will capture several aspects of diversity, where a higher number corresponds to better diversity, making comparison and tracking over time easier. The boundaries are achieved by normalising the Shannon index \(H^{\prime }\) , that is, calculating a Pielou index \(J^{\prime }\) ;in which \(p_i\) is the relative abundance of species \(i\) , and \(S\) is the number of species. To capture more aspects of diversity Hupont et al. [41] created a Conference Diversity Index consisting of the average of the three Pielou indices for gender, institute affiliation and geographic provenance.

In ecology, there is a plethora of diversity measures, and many of them can be written in the form of a Hill number,in which \(p_i\) is the relative abundance of species \(i\) , and \(q\) is a constant defining different diversity measures [47]. At the limit when \(q\) approaches 1, the diversity measure becomes the exponential of the Shannon index. The diversity can be measured within a community (alpha diversity), between communities (beta diversity), or a single measure can be calculated to capture both (gamma diversity) [48]. However, the use of diversity measures in ecology is contested [39]. The simplicity of such indices makes comparisons easy, but the lack of context or depth can make the comparisons unfair or misleading. After all, the clarity of the index is a consequence of a lot of information being excluded by the compression. The various indices promote different aspects of diversity, such as number of species (richness), the relative abundance of the species (evenness), or presence of rare or endangered species. The Shannon index depends on both the richness and the evenness, but the Pielou index is achieved by discarding the richness.

The diversity measures mentioned thus far all rely on entropy to conceptualise diversity. An alternative, for instance, used by Offenwanger et al. [59] in their DER measure, is to look at distance. The DER measure was specifically developed to measure inequality of representation between men and women, defined asThis results in a number ranging from [–1;1] where 0 corresponds to equal representation, negative values to an over-representation of men, and positive values to an over-representation of women. Although this metric lacks the ability to handle more than two genders, or intersectional aspects of gender, its simplicity makes both interpretations and limitations clear.

In the current article, we report both DER and GDI measures from our collected data, to allow a direct comparison to Offelwanger et al. [59] work on the CHI and Hupont et al. [41] work on the Affective Computing and Intelligent Interaction (ACII) communities. However, we are aware that none of these measures are well suited to capture the complexities of gender spectra and dynamics. We would therefore like to take this opportunity to call for novel and more appropriate measures of diversity. We would, at the same time, point out that all aspects of diversity cannot be captured by an index, so when using a diversity measure, it is important to complement it with a discussion about what aspects that specific measure highlights, and how to appropriately interpret the results.

We contacted the authors of Offenwanger et al. [59], who provided us with a copy of the Machine Assisted Gender Data Annotation (MAGDA) tool for this analysis. The MAGDA tool was designed to complement the data schema developed by Offenwanger et al., which we adhered to in our data collection. In this way, our dataset (which we release with this publication) complements and extends the dataset released with their publication. In addition to the data captured by Offenwanger et al. (number of participants and participant information, including both demographics and recruitment practices), we also extracted text that relates to analysis of participant gender (this would typically be found in the discussion or results section, e.g., we used gender of the participants as a control variable). 10% of the data (70 papers, randomly selected) were annotated by four coders to calculate inter-rater agreement. Agreement was high for overall number of participants (ICC = 0.97), and number of female and male participants (ICC = 0.84 and 0.82). Due to the very low number of non-binary participants, it was not possible to calculate agreement for this number. Agreement was also good for whether recruitment information was reported (Fleiss’ \(\kappa\) = 0.62), and whether gender was discussed and/or analysed (Fleiss’ \(\kappa\) = 0.66). The four coders then annotated the remaining 614 papers, randomly allocated so that each coder annotated approximately the same number of papers. In cases where some participant data were reported as being excluded from analyses, we aimed only to count and report on those participants whose data was actually used in analysis. However, this was not always possible; where some papers reported the demographics of participants recruited, then indicated they dropped data from e.g., n = 10, others reported the demographics of participants whose data was actually analysed. We hence flagged and captured the dropping of participants as additional participant data (see below).

Offenwanger et al. paper provides full detail on the guidelines underpinning the data collection process [59]. Here, we summarise key decisions and assumptions important for interpreting the results presented in Section 5.

In order to analyse participant sources as Offenwanger et al. did [59], we tagged all text that contained additional information about study participants. This typically included items such as age, nationality, familiarity with robots, and so on. Notably, we paid particular attention to reporting (or lack thereof) of the participant recruitment method, given Offenwanger at al. suggestion that gender bias in research participation might stem from recruitment method. We undertook post-hoc classification on these items, creating a set of labels which were applied to papers based on the collected data. The full list of labels and criteria for them to be applied can be found in Supplementary Materials Table 1.

We also identified papers that conducted some form of analysis of participant gender. Three coders (the three authors most familiar with HRI research) individually annotated these papers, which were then explicitly discussed in order to reach an agreement on what the analyses pertained to. This was required due to ambiguity in the way we saw such analyses being reported. As a result, we classified papers that had a clear research question or hypothesis related to gender as “main gender discussion”, and further separated these into papers that analysed the relevant results qualitatively and/or quantitatively. The remaining papers, which treated gender as “confound”, “controlled” for gender when conducting statistical analysis, or did some post-hoc analysis, were labelled as “confound”.

In order to classify the papers by HRI sub-topic, we followed the method used by Offenwanger et al. [59], applying probabilistic topic modelling [12] to our 684 papers using the MALLET library [35]. Based on coherence and perplexity measures, we determined that a suitable set of topics would most likely lie in the range of 8 to 20 topics. Topic sets in these counts were visualized as word clouds, and independently labelled, based both on the wordcloud results as well as which papers were associated with each topic, by three of the authors who have been part of the HRI community for several years. The results were discussed to select a final list of 15 (non-exclusive) HRI sub-topics (see Table 2, original wordclouds are provided in Supplementary Materials Figure 1(a)). We excluded two topics (“User Study” and “Data, Systems”, respectively) from final analyses due to their high paper count and high generality (all but two papers tagged with one of these labels were also tagged with at least one other). The paper count for each topic can be found in Table 2. The distribution of sub-topics for each year of the conference can be seen in Supplementary Materials Figure 5.

Concerning RQ1 on HRI participation, we find (similar to [59]) that gender reporting is increasing (Figure 2). Also, our analysis reveals that, of the papers that report gender, men have made up the majority of HRI research participants to date, with a small but consistent gap between men and women (Figure 3 left) that shows no indication of changing (Figure 3 right). Across all HRI conference papers between 2006 and 2021, average DER was \(-\) 0.085, and has fluctuated around an average of \(-\) 0.066 since 2010.

Mirroring Offenwanger et al. findings from HCI [59], non-binary participants represent only a tiny proportion of participants in HRI research studies (making them barely visible in Figure 4). Only in 2020 and 2021 have more than two papers at the conference reported studies including non-binary participants, with these participants representing 0.18% and 0.46%⁷ of the total participants described in those years, respectively. From the current data it is not possible to know the extent to which this under-representation might be caused by authors providing only binary answer options in gender demographics questions. Furthermore, only 6 of the 17 papers across 2020 and 2021 reporting studies with non-binary participants avoided othering by specifically utilising gender terms such as non-binary rather than simply listing anyone who did not identify with pre-defined items under an “other” category, deviating from published best practices for the inclusive capture and reporting of participant gender [71].

Concerning RQ2a on variation in gender bias patterns across different research topics, Figure 5 (left) shows (by use of the DER measure) that participant gender diversity varies across sub-topics of HRI, with a Kruskal–Wallis test demonstrating some of these differences to be significant ( \(\chi ^2(13)=36.639\) , \(p\lt 0.001\) ). Specifically, we found that the subtopic of “control and teleoperation” had significantly lower DER compared to “groups”, “gaze and attention”, “gender and anthropomorphism”, and “design and design studies”. The mean and standard deviation of DER for each subtopic, and post-hoc pairwise comparisons using Dunn tests, are provided in Table 2 and Supplementary Materials Table 3.

Concerning RQ2b regarding the impact that recruitment method might have on gender diversity in research participants, we found that only three recruitment practices were referred to in five or more papers across multiple years of the conference (see Table 1 in the supplementary materials for our coding schema). These were community and local institution recruitment—which appear to have remained consistently common across all years of the conference—and crowdsourcing—the majority of which specifically refers to Amazon Mechanical Turk (MTurk), which has increasingly appeared since 2015. Specifically, of the 65 papers that used crowdsourcing methods, 56 (86%) used MTurk.

The COVID-19 pandemic likely made in-person user studies infeasible for many researchers in 2020 and 2021; however, given that papers published at HRI 2020 were submitted in autumn 2019, this alone does not explain the increase in MTurk studies between 2019 and 2020. Similarly, if the pandemic was specifically responsible for a (temporary) increase in the use of MTurk for recruitment, then a larger increase than demonstrated might also have been expected between HRI 2020 and HRI 2021. As such, it seems reasonable to conclude there is a steady increase in the use of MTurk for recruiting HRI research participants, regardless of the pandemic.

Figure 5 (right) shows DER across the different recruitment types reported in papers, which was significantly different (Kruskal–Wallis test, \(\chi ^2(3)=12.24\) , \(p\lt 0.01\) ). We replicate Offenwanger et al. finding that crowdsourcing seems to represent a potential source of gender bias within participants because papers citing its use have significantly lower DER than papers utilising other recruitment methods (post-hoc Dunn test, \(Z = -2.92, p = 0.01\) . The full table of post-hoc pairwise comparisons can be found in Supplementary Materials Table 4).

Of the 113 survey respondents, 61 identified as men, 48 as women, 2 as non-binary, 1 as genderfluid; 3 researchers opted not to share their gender (note that participants could select more than one gender descriptor, in line with best practice guidelines [71]). Respondents’ gender distribution is shown in Table 1. Respondents reported working across 22 countries, distributed as follows: USA (41), Sweden (14), the Netherlands (10), Germany (9), the UK (8), Canada (5), Austria (3), Portugal (3), Switzerland (2), Spain (2), New Zealand (2), Japan (2), Italy (2), Denmark (2), France (1), Israel (1), India (1), Singapore (1), China (1), Turkey (1), Belgium (1) and Australia (1). As previously mentioned in Section 4, 73 respondents (65%) reported publishing at least one first or co-authored paper at an HRI conference. Figure 6 reports the number of HRI researchers working on the subtopics we previously identified during the systematic review. Due to the low number of respondents who identified as non-binary, genderfluid, or preferred not to disclose this information, we removed them from this figure as it would potentially constitute identifiable information.

In the supplementary materials, we provide additional results regarding the number of sub-topics participants reported working on (Supplementary Materials Figure 2) and their different educational backgrounds versus current fields (Supplementary Materials Figure 3). Supplementary Materials Figure 3, in particular, represents a study in what more intersectional consideration of diversity and representation issues in HRI might “look like”, as one might consider how educational background and gender differently impact on movement and/or belonging (building on [83]) within different sub-topics of research.

Regarding researcher diversity, we computed the GDI proposed in [41] (Section 2.3), with data from the author survey. Based on a count of self-identified men, women, non-binary, genderfluid respondents (note that we did not include people that preferred not to say), we calculated a GDI of 0.58. In comparison, the average GDI of ACII authors from [41] was 0.85, which would imply a higher diversity in ACII compared to HRI. However, this comparison needs to be interpreted with some caveats. While we calculated the GDI based on several self-reported gender categories, whereby respondents could also enter free text, [41] inferred binary gender categories from authors’ names. As mentioned in Section 2.3, smaller categories are penalised in the Pielou index; thus, even though our current GDI appears to be faring worse than the one computed for the ACII conference, it might be a by-product of trying to evade binary assumptions. Calculating the GDI of our respondents while only considering men and women, the result becomes 0.99, which is supposedly much more diverse. Furthermore, the Shannon indices (before normalisation) of our respondents are 0.81 and 0.69 when including four and two genders, respectively. At first glance, it might seem that such unweighted indices might be better suited for variables with many categories, however, the lack of an upper boundary makes this number prone to misinterpretation and hijacking. This strengthens our belief that current diversity measures do not properly account for social diversity, and that novel measures are sorely needed. Since it is not possible to capture diversity in general with any one simple measure or model [49, 60], the work of developing such measures for HRI needs to start by identifying what diversity should mean in HRI and based on that design the measures [39]. This might result in several measures to be used in parallel. When using these measures, they should always be presented in conjunction with an appropriate qualitative interpretation grounded in the underlying definition of diversity.

Addressing RQ4, Figure 7 shows the relationship between average paper DER (based on our dataset of HRI papers 2006–2021) and average author DER (based on our HRI researcher survey considering only those n = 73 respondents who identified as having published at the conference). There is a moderate correlation between these two values ( \(r = 0.55, p = 0.054\) ), suggesting a connection between gender diversity of authors within a sub-topic and gender diversity of the participants taking part in user studies relating to that topic.

We might expect these relationships to further interact with researcher educational background and specialism, given known gender trends across e.g., psychology versus robotics and AI [59], which ought to be considered in line with a fully intersectional approach to understanding these trends. More diverse research teams might be more aware of under-representation issues, and therefore strive to attract a more representative sample. Based on the responses of our survey, the most commonly reported backgrounds and current fields among HRI researchers are computer science, engineering, social science, and psychology, with the first two being biased towards men and the latter two being biased towards women (see Table 3). There is, however, a fair bit of mobility between researcher background and (current) researcher specialism/field (see Supplementary Materials Figure 3).

Several survey respondents highlighted their aspiration for representative and diverse sampling, both for scientific rigour and for more meaningful impact. A smaller number of respondents specifically highlighted that race/ethnicity needs more attention in HRI—again echoing similar critique published by HCI researchers [18]. Only 10 papers did such reporting; most of them are recent (the first being in 2010 and 5 being published since 2018), which might indicate an improvement due to raised awareness. With more participant information required for a richer understanding of context, one respondent proposed that: Maybe the conferences could encourage reporting of detailed participant information if it somehow did not count against the page limits and there was a standard list of demographic questions that people could work from.

On RQ5, the vast majority of papers reporting gender did not conduct gender-based analysis (the number of papers per year conducting such analyses can be seen in Supplementary Materials Figure 6). Of the 95 papers that did conduct gender analyses (representing 21% of the total papers which reported participant gender, Figure 8(a)), the majority (64 papers, representing 67%) treated it as a confound, with only 31 being identified as having a gender-related research question or hypothesis (Figure 8(b)). Results on the impact of gender are mixed (Figure 8(c)). This figure is further broken down by sub-topics in Supplementary Materials Figure 7). Concerning the extent to which gender is of explicit experimental interest in HRI overall, papers with an explicit research question or hypotheses represent 7.4% of all papers describing a user study in our systematic review. This is in line with additional findings from our researcher survey regarding respondents data collection and reporting, presented in the Supplementary Materials.

Our findings suggest HRI researcher mirrors HCI research with regards to gender bias in research participation, as we document an over-representation of men alongside very few non-binary participants to date. We similarly found these bias patterns to vary across (i) recruitment method, in particular replicating a concerning trend of decreasing women taking part in (increasingly popular) crowdsourced studies and (ii) HRI sub-topic, which in turn likely links to diversity of researchers (in terms of gender but also educational background and specialism) working on those sub-topics.

Whilst the majority of user study papers we analysed reported participants’ gender, it is perhaps surprising that this was not universal (even in papers from the most recent editions of the conference) given e.g., the APA guidelines we refer to in the Introduction. Whilst we did not replicate the entirety of Offenwanger et al. detailed analysis of gender/sex language being used to report participants [59], we did identify that many papers engaged in the othering of non-binary participants. Therefore, we encourage HRI researchers to integrate current best practices for the inclusive collection and analyses of participant gender into their work [71]. We also note that more widespread utilisation of these practices would improve the clarity of gender reporting.

Whilst the majority of papers we reviewed reported gender to some extent, and our researcher survey also appeared to suggest the main reason for collecting participant gender was for reporting purposes, it was surprisingly difficult for us to consistently extract quantitative data about exactly who had taken part in the studies described. Similarly, we do not know from our current data to what extent the low number of non-binary participants stems from authors failing to provide participants with a non-binary option in their gender demographics question, or due to the explicit recruitment of men and women only (although this was made explicit in some works e.g., [45, 91]. We, therefore, suggest that authors always report the percentage of participants who selected each gender option presented, even when that percentage amounts to 0, and make it clear whether (and why) any gender screening was applied at the point of participant recruitment or data analyses.

Here, we pull together and summarise some starting recommendations on how we, the HRI research community, might be able to improve on participant (gender) diversity and representation moving forwards. Notably, these recommendations cover actions at different levels (e.g., from individual researchers/reviewers through to conference/journal organisation committees), and are intended to provide a starting point for further discussions and initiatives. These recommendations further build on existing guidelines [71] and recommendations [18] coming from our neighbouring field of HCI.