Educative Social Media for Informal Science Learning: Effective Message Design Across Two Digital Niches Lisa Lundgren & Kent J. Crippen Abstract: The majority of science learning occurs in informal environments such as museums or aquaria, but also in online spaces such as forums or social media (Falk & Storksdieck, 2010). In a design-based effort, we report here on the examination of the behavioral engagement of a community of followers with social media messages that were systematically produced by researchers on XXXX, an NSF-funded project focused on building knowledge and relationships that center on paleontology (i.e. the study of fossils). Focusing Twitter and Facebook, we investigated the following research question: what messaging elements lead to increased behavioral engagement? In this presentation, we concentrate specifically on quantifying behavioral engagement with social media messaging and refining Falk and Dierking’s (2013) Contextual Model of Learning (CMoL) as it applies to the social media landscape. We find that community engagement varies dependent on platform, messaging elements such as hashtags, URLs, mentions, and post type. In particular, the use of hashtags without the inclusion of other messaging elements on Twitter showed significantly lower engagement than when used on Facebook. While these findings are significant in and of themselves, we argue that our study provides empirical evidence for use of CMoL with social media. Citation: Lundgren, L. & Crippen, K. J. (to be presented 2018, March). Educative social media for informal science learning: Effective message design across two digital niches. Annual International Conference for the National Association for Research in Science Teaching (NARST), Atlanta, GA Subject/Problem: Current research recognizes that science learning occurs throughout a person’s lifetime, across time and space in both formal and informal contexts (Bell et al., 2009). In sum, the majority of science learning occurs in informal environments such as museums or aquaria, but also in online spaces such as forums or social media (Falk & Storksdieck, 2010). Wenger, White and Smith (2009) define these online spaces as digital habitats and though they have been shown to emerge naturally, our goal is to understand how to design them in order to maximize their educative potential for science. In a design-based effort towards achieving our goal, we report here on the examination of the behavioral engagement of a community of followers with social media messages that were systematically produced by researchers on XXXX, an NSF-funded project focused on building knowledge and relationships that center on paleontology (i.e. the study of fossils). Facebook and Twitter comprise two niches, or distinct platforms upon which community members interact within the digital habitat. Best practices from a variety of fields were merged to create quality messages, including: graphic design principles, use of messaging strategies established by the marketing department, and educative strategies to sustain engagement beyond a single message. Focusing on these niches, we investigated the following research question: what messaging elements lead to increased behavioral engagement across the niches of Twitter and Facebook? In this presentation, we concentrate specifically on quantifying behavioral engagement with social media messaging and refining Falk and Dierking’s (2013) Contextual Model of Learning (CMoL) as it applies to the digital habitat. We find that community engagement varies dependent on niche, messaging elements such as hashtags (e.g., #paleontology), URLs (e.g., www.nsf.gov), and mentions (e.g., @NSF), and post type. In particular, the use of hashtags without the inclusion of other messaging elements on Twitter showed significantly lower engagement than when used on Facebook. While these findings are significant in and of themselves, we argue that our study provides empirical evidence for use of CMoL in digital niches such as Twitter and Facebook. The CMoL accounts for learning in myriad informal education contexts, but has yet to be applied in informal, digitized learning environments. We close with recommendations for an empirically grounded social media messaging strategy for informal science education and propose additional more robust applications of CMoL in informal, digitized science education practice. Theoretical Framework: This study was guided by the CMoL, a learning theory that takes into account three interdependent contexts relative to informal science learning: sociocultural, personal, and physical (Falk & Dierking, 2013). Each context uniquely influences learning: the sociocultural context accounts for interactions between learners and their perceptions of societal norms related to learning, the personal context details individual interests and motivations for learning, and the physical context establishes how a designed space affects learning. Though no existing research has applied CMoL to social media, Falk and Dierking suggest that social media is the new “word of mouth” that will “bring together the personal and sociocultural contexts” (Falk & Dierking, 2013, p. 88). User interaction via the conventions offered by social media platforms (i.e. likes, shares, comments) result in direct behavioral measurements can be affiliated with the personal and sociocultural contexts of CMoL. Following the lead of Hwong and colleagues (2017), who describe social media messaging as a way for scientists and scientific institutions to “effectively communicate their messages to the general public” (p. 480), we make use of CMoL to understand the ways in which social media messages bridge community interest in scientific topics, namely, in paleontology. Falk and Storksdieck (2010) describe the personal context of CMoL as the identity-related motivations for learning in informal environments. Interest is deeply connected to identity in these environments (Falk, 2006); measuring the relationship between messaging elements and engagement rates can explore the extent to which the personal context applies to the digital habitat. While the personal context relates to interest, the sociocultural context also molds those interests as it represents the societal norms and cultural practices that permeate social media messages as interested members of a community attempt to build knowledge or connect to one another (Wang et al., 2017). Methodology: In line with our theoretical framework, this study is oriented as an exploratory quantitative study that sought to determine which aspects of social media messaging enticed community members to participate in (i.e. like or share posts) or contribute to (i.e. comment on posts) purposefully designed messages. We focused on content-based elements, specifically, hashtags, mentions, and web URLs as well as a categorical framework for post type. Data included 1,450 designed messages over a two-and-a-half-year period (May 2014-Dec 2016) that were posted to Facebook and Twitter. Engagement rates were reported independently by each niche for each message. Though calculated slightly differently due to the conventions of each niche, the reported engagement rate represents a similar construct and is computed by dividing the total number of engagements (likes, comments, shares, clicks) by the total number of members a message reached (Bugeaud et al., 2016). We view the reported engagement rate as a form of behavioral engagement, an observed manifestation of cognitive activity (Azevedo, 2015). Facebook posts were collected via administrator access to the Project’s public page. Researchers exported monthly “post level” data from the Insights tab, then information about each post was tabulated in a spreadsheet. For Twitter, researchers accessed the Project’s account analytics with administrator access, focusing on the tweet activity tab. As with Facebook, Twitter data were extracted monthly. Then, message elements were determined and tabulated through hand coding. Messages were classified based whether they included some combination of three specific elements that have been shown to increase messaging engagement: hashtags, mentions, and URLs (Naveed et al., 2011). Hashtags, a metadata technique that allow users to organize information around a certain topic/s, as well as including URLs have been shown to predict Twitter message popularity (Petrovic, Osbourne, & Lavrenko, 2011). Messages with mentions, or the act of including a user’s Twitter handle (e.g., @NSF), have been shown to lead to increased citations of pre-prints of peer-reviewed work (Shuai, Pepe, & Bollen, 2012). In a separate analysis, the post type analytical framework (Authors, 2016; 2017) was applied to all messages regardless of which elements they included. This framework involves five categories that delimit posts based upon message content. For example, an information post includes general resources for paleontology while a research post focuses on an aspect of scientific research, such as a link to a peer-reviewed journal article. A constant comparative method was used by two researchers for this coding (Lincoln & Guba, 1985). Following data cleaning and processing, the average engagement rate for each category was used for comparison. A one-way ANOVA with post hoc comparisons using the Tukey HSD test was used to determine statistical differences among the averages within each niche. All averages were compared to the benchmark engagement rate for the non-profit education sector as determined by Bugeaud and colleagues (2016). Since the reported engagement rates were computed in a slightly different fashion, the averages were only compared descriptively across the niches. Results: The categorization of messages led to three main findings: (a) some messaging elements were used frequently while others more scantily, (b) engagement rates differed by niche and message element, and (c) post type highlights differential engagement rates within and across niches. Following an overall description of most-used and least-used elements, we examined engagement rates for messaging elements, focusing on the engagement rates when messages employed singular elements and combinations of elements (Figure 1). For both Twitter and Facebook, there were significant differences in engagement rates when messaging elements were analyzed (F (6, 1252 = 6.85, p <.001). Post hoc comparisons using the Tukey HSD test indicated that the mean score for messages with only hashtags (M = 4.37, SD = 3.48) was significantly different than messages with mentions (M = 2.36, SD = 2.02), messages with mentions and URLs (M = 2.55, SD =1.60), messages with hashtags and mentions (M = 1.96, SD = 1.28) and messages with hashtags, mentions, and URLs (M = 2.55, SD = 1.60). However, messages with hashtags did not significantly differ from messages with hashtags and URLs (M = 4.23, SD = 3.33). A list of top performing hashtags, mentions, and URLs for each niche will be included in the presentation. While this study sought to examine messaging elements, we also included messages without messaging elements, finding that these messages averaged a 1.2% engagement rate on Twitter and a 4.9% engagement rate on Facebook. Messages that included solely URLs showed the largest disparity between niche engagement rates, with Twitter messages that included URLs only garnering an average engagement rate of 1.3% while Facebook messages that included only URLs averaged a 5.6% engagement rate. While most Twitter messages surpassed the benchmark engagement rate, over half of the Facebook messages did not. Benchmark engagement rates are functionally useful; however, they provide a broad view on the subject of social media messaging whereas we sought to determine more detailed reasons for engagement. Similar to engagement rate, post type indicates the essence of social media messages while messaging elements offer opportunities for community members to interact with such messages. With this in mind, we sought to determine which post types elicited more engagement. Examining post types across niches revealed that messages which were coded as information posts, research posts, or news posts garnered more engagement than opportunity posts or comment posts. There was a significant effect on engagement rate at the p<.05 level for different post types (F (4, 1420) = 16.4, p <.001). Post hoc comparisons using the Tukey HSD test indicated that the mean score for messages coded as information (M = .044, SD = .032), messages coded as research (M = .041, SD = .036), and messages coded as news (M = .041, SD = .033) were significantly different than messages coded as opportunity (M = .032, SD = .025) or messages coded as comment (M = .028, SD = .026). Information, research, and news posts all explicitly focus on aspects of paleontology which could be of interest to our community, whereas opportunity and comment posts may not. Findings and Discussion: We sought to elucidate the ways that social media messaging elements relate to behavioral engagement within social media niches. Our analysis also found that engagement rates differed between niche and by message elements, with Facebook messages that included only URLs engaging a much higher percentage of members than Twitter messages which included the same. Lastly, we tracked the relationship between post type and the inclusion of hashtags, finding that engagement rate varied within niche with comment type posts garnering a significantly higher engagement rate on Facebook, although this was not the case on Twitter. For the work on post type, we postulate that the higher engagement with research and information post types on Facebook relates to two of CMoL’s contexts for learning: the personal and sociocultural contexts. Research post types and information post types provide the community with generalizable content related to paleontology or with links to blogs or photos from scientists, amateur paleontologists, and organizations. The sociocultural context of learning postulates that informal education environments exist as independent constructs that reside “in the minds of individuals living within a community” (Falk & Dierking, 2013, p. 78). This means that in order to build knowledge with communities, social media messaging must build upon the perceptions of community members. The post types of research and information do this by linking to people within the community members might know, or highlighting paleontological constructs that the community already has familiarity with. In a similar manner, the personal context of learning is deeply tied to individual experiences, knowledge, and interests: information post types can serve to highlight these experiences and interests in ways that an opportunity or comment post type cannot. Contribution to Teaching and Learning: Research into public engagement with science via social media has been done in multiple contexts including YouTube (Welbourne & Grant, 2016), Twitter (Daume & Galaz, 2016), and Facebook (Fauville et al., 2015), yet these empirical pieces do not necessarily capture the ways in which social media can be used for educative purposes. Recent work on social media focused on space science found that hashtags, along with communication styles capture attention more so than other messages, however, this focused on predictive forecasting and the data were not separated by social media niche (Hwong et al., 2017). Our current study therefore furthers the field of creating educative, science-specific social media messaging by focusing on another gateway science (i.e. paleontology), acknowledging the differences in niches, and adding to our understanding of how messaging elements contribute to engagement rates. We also add to the body of work concerning the CMoL as we show that digital, social media niches are informal science learning spaces in which the CMoL’s personal and sociocultural contexts affect engagement levels, especially in messages that were coded as information and comment post type. Interest to NARST: The current study will be of interest to NARST members who use social media in their research, to those members who are interested in digital education environments, and to members with an interest in informal science education. All content, regardless of social media niche, competes for engagement in this age of constant digital marketing. By providing evidence for which messaging elements and post types within which niches produce higher behavioral engagement, NARST members can integrate such practices into their own research. Furthermore, we provide evidence for the application of the CMoL, a robust and well-established theory, in digital habitats through establishing a relationship between post types and the CMoL’s personal and sociocultural contexts. By determining engaging messaging elements and post types, scientists and educators can work to build messages that highlight key science issues while reaching diverse networks. Tables and Figures Table 1 Engagement rates with messaging elements by niche Facebook Twitter Messaging Element n M (SD) n M (SD) Only hashtags 180 .056 (.036) 75 .013 (.012) Only URLs 116 .037 (.022) 48 .024 (.017) Only mentions 6 .026 (.022) 46 .023 (.019) Hashtags and mentions 2 .035 (.017) 69 .019 (.012) Hashtags and URLs 331 .049 (.036) 136 .026 (.017) Mentions and URLs 13 .038 (.011) 70 .023 (.018) Hashtags, mentions, and URLs 12 .033 (.020) 147 .025 (.015) No element included 195 .049 (.037) 4 .025 (.011) Total 855 595 Figure 1: Facebook and Twitter engagement rates of messages with examined elements Selected References Azevedo, R. (2015). Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues. Educational Psychologist, 50(1), 84-94. Daume, S., & Galaz, V. (2016). “Anyone know what species this is?” - Twitter conversations as embryonic Citizen Science communities. Plos One, 11(3), e0151387. Falk, J. H., & Dierking, L. D. (2013). The museum experience revisited. Walnut Creek, CA: Left Coast Press, Inc. Fauville, G., Dupont, S., von Thun, S., & Lundin, J. (2015). Can Facebook be used to increase scientific literacy? A case study of the Monterey Bay Aquarium Research Institute Facebook page and ocean literacy. Computers & Education, 82, 60–73. Hwong, Y.-L., Oliver, C., Van Kranendonk, M., Sammut, C., & Seroussi, Y. (2017). What makes you tick? The psychology of social media engagement in space science communication. Computers in Human Behavior, 68, 480–492. Naveed, N., Gottron, T., Kunegis, J., & Alhadi, A. C. (2011). Bad news travel fast: A content-based analysis of interestingness on Twitter. In Proceedings of the 3rd International Web Science Conference WebSci  ’11(pp. 1–7). New York, New York, USA: ACM Press. Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: article downloads, Twitter mentions, and citations. Plos One, 7(11), e47523. Wang, R., Kim, J., Xiao, A., & Jung, Y. J. (2017). Networked narratives on Humans of New York: A content analysis of social media engagement on Facebook. Computers in Human Behavior, 66, 149–153. Welbourne, D. J., & Grant, W. J. (2016). Science communication on YouTube: Factors that affect channel and video popularity. Public Understanding of Science (Bristol, England), 25(6), 706–718. Wenger, E., White, N., & Smith, J. D. (2009). Digital habitats: stewarding technology for communities. Portland, OR: CPsquare. 1 Educative social media for ISE| NARST 2018