Evaluating the effectiveness of 360 videos when teaching primary school subjects related to environmental education

The study presents the results of a project in which 360 o videos were used for informing students about subjects related to environmental education. The target group was forty-four, nine- to ten-year-old students, who used three teaching tools, namely printed material, web pages, and 360o videos. The project lasted for twelve two-teaching-hour sessions (four for each tool), and data were collected by means of evaluation sheets and a questionnaire for recording their views and attitudes. It was found that the 360o videos yielded better learning outcomes compared to the other tools. The sense of presence was also greater in this tool. Then again, all tools were considered as equally motivating and effective. Moreover, the 360o videos were considered more enjoyable and easier to use only when compared to the printed material. Though the findings confirmed that 360 o videos can be effective in raising students' awareness of environmental problems, they also point to the need of finding innovative methods and teaching frameworks that would allow educators to fully exploit their educational potential.


Introduction
Well-informed and sensitized citizens are expected to be able to understand and be actively involved in resolving environmental issues vital for our well-being, as well as for the planet's future. Such issues that seek our immediate attention are climate change, global warming, pollution, and the loss of biodiversity caused due to human activities. The publicity these issues receive in the mass media has raised the public awareness of such problems. Then again, the importance of understanding environmental concepts and the intricate inter-relationships of all the factors/parties involved, emphasizes the need for well-organized, innovative, and efficient educational frameworks and practices (Markaki, 2014). In this respect, Environmental Education (EE) should play a central role in educating people and in raising their awareness. It is generally agreed that EE is rather effective in infusing individuals with knowledge, values, experiences, and -most importantly-with the determination to act (Stevenson, 2007). Alas, it is not an easy task. Children, as well as adults, find it difficult to identify some of the critical environmental issues, either because they are not directly visible or because of the time distance between cause and effect (Ahn et al., 2016;Hsu et al., 2018). The literature demonstrated that these difficulties can be eased if individuals acquire first-hand experiences (Ahn et al., 2016). Through personal experiences, it is more likely that people will regard the negative effects of environmental problems as directly connected to them, develop positive behaviors, and become more actively involved in their solution (Akerlof et al., 2013;Zaalberg & Midden, 2010). Although the bulk of the EE resources is mostly conventional (e.g., printed material), a wide range of ICT based resources is becoming increasingly available, including, but not limited to, web sites, augmented, and virtual reality applications (e.g., Ahn et al., 2016;Fokides & Chachlaki, 2019;Hsu et al., 2018;Huang et al., 2016).
Videos are one of the most widespread forms of entertainment, as well as educational material (Smith et al., 2012). Carr-Chellman and Duchastel (2001) claimed that the success of videos, both as entertaining media and as learning tools, lies in the fact that viewers/trainees identify themselvesto some extent-with what they watch in the video. Despite their obvious advantages, videos lack certain features that would probably make them more interesting and effective. For example, viewers are passive receivers; they cannot view the environment from different angles since there is only one (chosen by the cameraman or the director). In this respect, 360 o videos seem to offer an interesting solution. In short, panoramic cameras are used for recording such videos; these cameras can capture images not only from a limited field of view, but from a full circle or, more correctly, from a sphere. It is remarkable how easily anyone can produce 360 o videos without any prior or specialized knowledge, by simply using a 360 o camera. Processing and editing them is not different from that of an ordinary video. Users can view them either on their smartphones or by using head-mounted displays (HMDs). In essence, one has to imagine a sphere in the center of which the users are positioned. They can turn their heads in any direction they want and see a portion of the sphere. In addition to the imposing presentation of the visual material, users have the illusion that they participate -albeit on a second/indirect level-in the experience, as there is not a single spot that cannot be depicted. Interactions can also be added through embedded hotspots. 360 o videos have begun to find their way in many areas where a high degree of realism is necessary (e.g., Health Sciences, Biology, and Engineering), but also in education (Ardisara & Fung, 2018). Most research reported encouraging results in knowledge acquisition (e.g. Pham et al., 2018;Ritter III et al., 2019), enjoyment, and motivation to learn (e.g., Lee et al., 2017;Wu et al., 2019;Xie et al., 2019).
Given the above, it was considered an interesting endeavor to examine the impact of 360 o videos on primary school students' knowledge about environmental issues. For that matter, a project was implemented, which compared the learning outcomes from the use of 360 o videos, printed material, and web pages. The rationale, methodology, and results of this project are presented in the sections to follow.

Environmental Education
Together with the growing interest in environmental problems, the interest in EE is also growing steadily (United Nations Environment Programme, 2013). In a nutshell, through a combination of organized interventions, the objectives of EE is to inform learners of all ages about the functions and processes of the ecosystems, to motivate them to try to solve existing problems (United Nations Educational, Scientific, and Cultural Organization, 1978), to foster behaviors and attitudes that would allow them to balance the protection of the environment and development (Tisdell, 2013), and to help them understand how to avoid new problems (Buckler & Creech, 2014).
The topics that EE examines are -by default-multi-disciplinary; Physics, Geography, Chemistry, and Social Sciences all contribute, rendering it as one of the most complex teaching/learning subjects (Fauville et al., 2014). Moreover, critical thinking and the capacity to make informed decisions are also required, so as to reach a deeper understanding of the intricate interrelationships of all the factors involved in the environmental issues (Wals et al., 2014). Given that EE deals with a multitude of subjects, educational systems across the world implement it differently and put emphasis on different issues. For example, in Turkey, Australia, Singapore, Ireland, and Canada, students are taught subjects related to biodiversity, pollution, man-made environment, and ecology (Derman & Gurbuz, 2018). Although within the European Union EE is compulsory, the member states follow different approaches. In some (e.g., Poland and Portugal), it is embedded in science courses (Pitarma et al., 2018;Pytel et al., 2016). In others, such as Denmark, there is an interdisciplinary approach, while in Sweden there are specialized environmental courses (Fauville et al., 2014;Skolverket, 2011). As far as the Greek educational system is concerned, EE is not a distinct course. Educators are encouraged to design and implement initiatives related to environmental issues, but this is done in an unsystematic and, mostly, not well-organized way (Flogaiti, 2005). On the other hand, some teaching units do exist, that present and examine environmental issues in both primary and secondary level, but only in the context of science-related courses. For example, in the fourth grade of primary school, there is a course called "The study of nature," in which there are several units that refer to the ecosystems, endangered species, pollution, and recycling.
Diverse settings, methods, and tools are utilized for teaching EE subjects. Besides formal settings (i.e., at school), non-formal ones are not uncommon. For example, there are specialized training courses for specific professional groups, such as fishers, on how to deal with endangered species (Notarbartolo di Sciara et al., 2009). Informal interventions are also common, for example, programs offered for national parks visitors (Miller et al., 2013). Conventional means (i.e., printed material, project toolkits, comics, leaflets, and brochures) seem to be the most frequently used tools. ICT based tools (e.g., web sites and digital eco-games) are also widely available (Fokides & Chachlaki, 2019). Moreover, virtual reality (e.g., Ahn et al., 2016;Hsu et al., 2018;Metcalf et al., 2018) and augmented reality applications (e.g., Huang et al. 2016;Lu & Lin, 2015) have also been reported. Although it is generally agreed that, regardless of the tools and methods, EE programs/activities are effective (Hill, 2013), the literature suggests that there are quite a lot of barriers hindering their efficacy. One such, is the difficulty individuals have on identifying environmental problems, the reason being that environmental degradation is not usually directly observable, either visually or temporally (Ahn et al., 2016). For instance, while ocean acidification reduces the pH of the sea and leads to significant disruption of marine ecosystems, people do not realize the risk, because they rarely have the chance to witness personally the effects of acidification on marine life (Ahn et al., 2016). Likewise, in their daily lives, people do not notice the reduction of water supplies and this lessens their concerns about its preservation (Hsu et al., 2018). A similar behavior is observed on the issue of climate change; as the period of time that intervenes between a harmful action, such as pollution, and climate change can be several decades or even hundreds of years, people tend to believe that an isolated phenomenon/event cannot have extreme effects (Markowitz et al., 2018). Even when people understand environmental problems, they tend not to feel directly responsible, because of the psychological (Uzzell, 2000) and temporal distance between cause and effect (Ahn et al., 2016).
Coming to environmental pollution and climate change, the relevant literature identified several concepts and ideas that students have trouble understanding. For instance, research suggested that they tend to relate the concept of "environmental pollution" almost exclusively with the presence of waste and rarely refer to air, soil, water, and noise pollution, or to the problems caused to flora and fauna (Kilcan & Çepni, 2015). In addition, other researchers suggested that students are not able to make the distinction between the concepts of "climate change," "global warming," and "greenhouse effect" (Boylan, 2008). Not only this, but it was found that they believe that climate change can be attributed only to anthropogenic factors; natural factors are excluded (Chang & Pascua, 2016). Although sometimes they seem to have a clear picture of the effects of climate change, they are quite confused about the solutions and, most importantly, its causes (Liarakou et al., 2011). Finally, some studies concluded that even though students recognize concepts related to the environment (e.g., ecosystem, species, environment, food chain, biodiversity, global warming, and acid rain) they cannot define them, nor specify their causes or effects (Visintainer & Linn, 2015;Yücel & Özkan, 2015).
Regarding ecosystems, literature suggested that students think that organisms at a higher level in the food chain consume all the organisms at the lower levels (Munson, 1994). Other studies noted that population changes also confuse them. For example, they think that the variation in the population of a species affects only the species with which it is directly connected (Griffiths & Grant, 1985;Munson, 1994). Moreover, they believe that changes in the population of a species do not affect the entire ecosystem, because they consider some organisms as not that important (Munson, 1994). For some students, ecosystems are not fully functional entities but merely a collection of organisms (Brehm et al., 1986). Although they can predict the effects of removing a producer/prey from the food chain, they find it difficult to identify the effects of removing a consumer/predator at the top (Leach et al., 1996). Interestingly enough, the above problems are also present in young adults and even in students studying Biology (Butler et al., 2015). Dikmenli and Cardak (2017) as well as Ruckert (2016), found that students do not seem to grasp the concept of "endangered species", commonly confusing it with the term "extinct species." When asked to explain why an animal is threatened with extinction, the reasoning is either unclear or without coherence (Dikmenli & Cardak, 2017;Ruckert, 2016). In addition, students do not comprehend the main, as well as the secondary consequences of an animal's extinction (Dikmenli & Cardak, 2017). For example, they believe that the impact will be minimal if the animal is not "important" (Butler et al., 2014;Munson, 1994). Researchers found that students' ability to understand the globality of the phenomenon is also limited; they either think that it is an issue that affects local species or their knowledge is limited to species that have received considerable publicity (Giovos et al., 2018;Gomes et al., 2019). Moreover, students can usually name as endangered species animals belonging to the mammal class and not to other classes of vertebrate (Dikmenli & Cardak, 2017).
Collectively, the above troubles and difficulties probably signify problems in understanding and mastering both EE's core concepts (i.e., the conceptual blocks advancing a subject's understanding) and threshold concepts (i.e., essential ideas that their understanding leads to an irreversible transformation of an individual's perception of a phenomenon/subject) (Meyer & Land, 2003). It seems that students find concepts related to EE somehow "alien," intellectually challenging, and absurd; in short, they are "troublesome knowledge" as defined by Perkins (1999).

360 o Videos
As already mentioned, 360 o videos allow users to look at any direction they want, instead of being limited to the constant viewing angle of conventional videos (Rupp et al., 2016). While such videos can be viewed in any device capable of displaying regular videos, their advantages are fully accessed when they are viewed using smartphones or HMDs. The built-in accelerometers and gyroscopes of smartphones and HMDs track user head movement and, in turn, the portion of the video that corresponds to the relative viewing direction of the user is displayed (Pham et al., 2018). What made 360 o videos accessible was Google Cardboard (Curcio et al., 2016). As the name implies, it as simple device made from cardboard (Cardboard compatible HMDs made of plastic are also popular) with two lenses for the user's eyes, in which a smartphone is inserted (Hussein & Nätterdal, 2015). Quite reasonably, the use of 360 o videos attracted the interest of researchers and the literature reports a variety of fields of application. They are considered ideal for virtually visiting museums and archaeological sites, for viewing medical procedures or laboratory experiments (e.g., Argyriou et al., 2017;Lee et al., 2017;Queiroz et al., 2018;Pham et al., 2018), and for subjects such as Natural Sciences, Geography and Mathematics (e.g., Minocha et al., 2017;Wu et al., 2019). There is also research regarding their use for delivering virtual courses (e.g., Ardisara & Fung, 2018), for the study of foreign languages and culture (e.g., Blyth, 2018;Xie et al., 2019), and for supporting hοme-schooling of people unable to physically attend school (e.g., people with health problems or teenage mothers) (Minocha et al., 2017). While not extensive, there is also research for the teaching of subjects related to EE, such as Ecology (Wallgrün et al., 2019), endangered plants (Ahmad et al., 2019), protected areas and the impact of development on the natural environment (Minocha et al., 2017), and renewable energy (Ritter III et al., 2019).
Despite the wide scope of applications, research is still at a very early stage and, logically, there are contradictory results and several gaps in research. For example, the learning outcomes that 360 o videos produced, compared to other teaching tools, sometimes were found to be better (e.g., Pham et al., 2018;Ritter III et al., 2019;Wu et al., 2019) and sometimes there were no statistically significant differences (e.g., Karageorgakis & Nisiforou, 2018;Ulrich et al., 2019). Others noted that the pedagogical framework for their use is still extremely unclear (Fowler, 2015). The target group was usually young adults; research examining the effects of 360 o videos on primary school students is limited (e.g., Minocha et al., 2017;Wu et al., 2019).
Technical challenges are also present; one such being image resolution. While there are cameras that record ultra-high-resolution 360 o videos (8K, 7680 × 4320 pixels) their cost is still particularly high. More affordable ones can record 4K (4096 x 1714 pixels) or 5.5K (5632 x 2700 pixels) videos. Although such resolutions sound adequate, it should be noted that images are spread in a space covering 360 o and are halved given that they are divided into two (one for each eye). Usability also plays an important role. A hard-to-understand user interface interrupts the flow of users' experience, as they struggle to master it (Glaser & Schmidt, 2018). Although users find 360 o videos easy to use, as navigation is done by focusing on embedded hot spots in the scene and holding that position for about two seconds, hand-tracking devices are considered an even more natural way of interacting (Miller & Bugnariu, 2016). Another problem, reported by a quite large number of users, is the severe discomfort, vertigo, and nausea caused by the feeling of unnatural movement that is brought on when navigating, called simulator/motion sickness. The problem worsens when videos are recorded with a moving camera; users perceive that they are moving although they are actually still (Kasahara et al., 2014). Quite logically, this situation severely impairs one's cognitive abilities and learning experience, negatively impacting the learning outcomes (Rupp et al., 2019). Fortunately, motion sickness can be reduced when the camera is static (i.e., fixed on a tripod) and eliminated if the HMDs are removed for a short period. Drawing the attention of users to what is important, as well as dealing with disorientation, are also noteworthy issues (Ardisara & Fung, 2018). For example, users can easily get distracted. Moreover, there is no guarantee that they will always look towards the direction they are supposed to look; they may be engaged seeing something that intrigues them and miss something important that takes place at another part of the scene. One solution that has been tested is to insert time delays at key parts of the video, giving viewers the chance to reorient themselves (Kavanagh et al., 2016). Another idea is to shoot videos from afar so that items can be easily observed (Ardisara & Fung, 2018).
Researchers have also focused on the increased levels of enjoyment/fun 360 o videos offer (Lee et al., 2017) and on users' motivation to engage with the content (King- Thompson, 2017;Xie et al., 2019). As 360 o videos depict the real world and not a simulated/digital version of it, the feelings of immersion and presence seem to be strong (Argyriou et al., 2017). Presence is the feeling of "being there," of living within the virtual world (Falah et al. 2014) and it is important for a variety of reasons. First, it engages the user and has been associated with both the increased recall of the virtual experience and the increased awareness of the state of the virtual environment (Papadakis et al., 2011). Secondly, people who feel "present" in a virtual environment are more likely to have a realistic behavior, acting as if they were in a real situation. Some suggested that this phenomenon positively affects their behavior in the real world (Ahn et al., 2014). While the levels of immersion and presence were high (e.g., McKenzie et al., 2019;Rupp et al., 2016), their impact has not always been positive. The argument made is that users were distracted, missing what was important (McKenzie et al., 2019) or they were over-excited due to the novelty of the experience and ignored the content (Rupp et al., 2016). This signifies the need to find methods to engage users with both the environment and the content.
To summarize, the literature review presented in the preceding sections, revealed that: (i) students face quite a lot of problems in understanding basic concepts related to EE, (ii) 360 o videos are an emerging field of study not adequately systemized, as research is spread thin across diverse subjects and methods, (iii) primary school students were not usually the target group, and (iv) a considerable volume of the research on 360 o videos had rather small sample sizes and/or utilized pre-post study designs without comparing the results with other teaching tools commonly used in EE (e.g., printed material and web pages). Having the above in mind, a project was implemented, setting forth the following research hypotheses:

Method
For testing the research hypotheses, a within-subjects experimental design with three treatments/conditions was applied, meaning that the study's participants were exposed to three different teaching tools (printed material, web pages, and 360o videos). Compared to betweensubjects, within-subjects designs are considered more statistically efficient, more powerful, and require smaller sample sizes (Greenwald, 1976). The project lasted for twelve two-teaching-hour sessions (four for each tool, once a week), from mid-September 2019 to mid-December 2019. Details for the project are presented in the following sections.

Participants
As already presented in a previous section, the Greek program of study for primary schools dictates students to be taught subjects related to EE, for the first time, in the fourth grade (ages nine to ten). Thus, the study's target group was students of this age. Several selection criteria were applied for achieving an ordinary/typical sample (Creswell & Poth, 2017): (i) students to attend public primary schools, (ii) to have never before viewed 360 o videos, used HMDs, and to have never before been taught the same subjects or similar to the ones included in the project, and (iii) both students' ratio of boys-to-girls and spread of ability to be close to that of a typical Greek fourth grade. Several public primary schools in Athens, Greece were contacted and their fourthgrade teachers were interviewed for determining whether their students met the above criteria. The outcome of this process was the selection of two classes having a total of forty-four students (twenty-one boys and twenty-three girls). A statistical power analysis was performed for sample size estimation, based on the data from the studies presented in section "360 o videos." A medium effect size for this study was considered acceptable using Cohen's (1969) criteria. With an α = .05 and power = 0.80, the projected sample size needed with this effect size is approximately thirty students for a within-group comparison. Thus, a sample of forty-four students was more than adequate for the main objectives of this study.
As the study involved minors and in compliance with the rules for conducting research, the following measures were taken: (i) a research approval was granted from the University's ethical committee and (ii) following a briefing, students' parents granted their written consent for their children's participation in the project.

Materials
A number of units from the fourth-grade textbook "The study of nature" formed the basis for the project's teaching/learning material. These units present four major themes (i) Greece's main ecosystems, (ii) endangered species in Greece, (iii) pollution and prevention, and (iv) waste management and recycling. After assessing their content, it was decided to re-arrange it and add supplementary material, so as to thoroughly cover the subjects discussed. Moreover, as the topics discussed in these units are rather complicated and reflecting on the problems students and adults face in understanding environmental issues, it was decided to (i) put emphasis on core and threshold concepts, (ii) provide specific and easy to grasp examples, and (iii) modify the cognitive load to be within the capabilities of nine-to-ten-years-old students.
Another issue that had to be resolved was derived from the research design. While in betweensubjects designs the learning material is the same among groups, the same cannot be applied to within-subjects designs, as there is just one group of participants. Consequently, after using the first tool, students would have acquired some knowledge on a given subject, more after using the second, and even more after using the third, rendering the learning outcomes from each tool incomparable. At the materials design level, two measures were taken for resolving this issue (another three measures were taken related to the procedures that were followed and they are elaborated in the corresponding section). Firstly, the learning material, though different in each tool, had to be equipollent in terms of quantity, quality, and cognitive load (e.g., the same quantity of text and difficulty level, the same amount of terms, facts, and figures). Secondly, it was deemed appropriate for all the subjects presented in one tool to have corresponding subjects in the other two tools. For example, it was decided that students were to be taught about land pollution with the use of printed material, air pollution through web pages, and water pollution through 360 o videos. An outline of the learning subjects per teaching tool is presented in Table 1. The next step was to gather and edit the necessary texts and multimedia material (e.g., videos, animations, and photos), so as to form twelve discrete teaching units (four per tool). During this stage, both the authors and the participating teachers suggested material to be considered for inclusion. As for the project's 360 o videos, several freely available on the Internet were considered, but also several videos were recorded using a 360 o camera. As the water was the theme chosen to be presented through 360 o videos (see Table 1, 4 th column), these video clips presented areas near Athens in which the marine ecosystem is badly damaged due to heavy pollution or beaches full of garbage and other wastes.
For every unit, several drafts were formed and discussed during a series of group meetings and the final versions were formed. Furthermore, worksheets were written for reasons elaborated in the "Procedure" section. A booklet was written which included the learning material for the first four units. The units were split into two parts, the first presenting the main learning material and the second presenting additional/further readings. As it was not possible to include videos, they were replaced by a series of screenshots and text. Four web sites were developed presenting the material of the next four units. Again, some web pages presented the main material and other pages presented the additional one. Four mobile apps were developed for the last four units. For that matter, 3D Vista Virtual Tour was used (https://www.3dvista.com/). This software allows the -relatively easy-development of 360 o virtual walkthroughs by combining 360 o and regular videos and photos. Multifunctional hot spots and clickable objects were added for allowing the users to interact with the apps (e.g., loading another video, presenting a photo album, displaying text, and playing an audio file) ( Figure 1). As with the other tools, the apps had two parts, one for the main material and one for the additional. On the basis of the comments made by five students (not included in the study) who read the booklet and used the web sites and the apps, minor changes were made. As a final note, for viewing/using the apps the project provided smartphones and Cardboard compatible HMDs to the participating students ( Figure 2).

Procedure
As already stated in the preceding section, another three measures were taken, related to the procedures that were followed, in order to avoid problems associated with the research design that was applied in this study. First, prior to the beginning of the study, the teaching framework, described below, was presented to the two teachers involved in the project. Next, they were asked to simulate a teaching session, while the other teacher together with the researchers acted as "students." Moreover, in order to further ensure that the teachers were not biased in favor of one tool over the others, one of the researchers was present during teaching (but did not participate by any means). Second, all sessions were conducted on the same day of the week and at the same school hours. This was done in order for the effects of external factors (e.g., students' tiredness or loss of interest due to previous lessons) not to vary across sessions. Third, for avoiding any order effects, the use of the tools was randomized (meaning that they were not used sequentially). In addition, the participating students were not informed about which tool they were going to use in each session.
Two teaching hours were allocated for each session. That is because it was essential for students to have enough time to study the learning material and conduct all the activities. Moreover, it was important to make provisions for some extra time, in case there were delays or technical issues because of the use of computers, smartphones, and HMDs by the students. A constructivist teaching approach was utilized with all tools. Given that pair/group work is strongly advised when teaching science subjects, students studied/worked in pairs-with the only exception being when students used the apps (Harlen & Qualter, 2014). Bybee et al.'s (2006) 5Es instructional model was considered the most suitable teaching framework. It is a well-received framework that is based upon the findings of constructivist-learning theory, cognitive psychology, and best practices in science teaching. As Bybee (1997) declared: "using this approach, students redefine, reorganize, elaborate, and change their initial concepts through self-reflection and interaction with their peers and their environment. Learners interpret objects and phenomena and internalize those interpretations in terms of their current conceptual understanding" (p. 176). The activities that took place in each stage were the following:  During the Engage stage, the teachers made a short introduction, provided examples from everyday life, and initiated the first round of discussions among students. The purpose was to trigger their interest regarding what they were about to learn.  During the Explore stage, depending on the tool that was used, students studied a unit's main material either from the booklet, or from the web pages, or from the first part of the apps. Following that, they used the unit's corresponding worksheet for recording their views, opinions, and explanations.  Students communicated their ideas and opinions during the Explain stage. Each pair presented what they recorded at the previous stage, followed by discussions with the rest of the students. When a common consensus was reached for the topic that was discussed, students recorded their final thoughts in the worksheets.  The Extend stage allows further exploration of the ideas or issues presented during a lesson. For that matter, students studied the additional material from the booklet, the web pages, and the second part of the apps. As in the Explore stage, students used the worksheets for recording (once again) their views, presented their ideas, and discussed them with the rest of the class.  In the Evaluation stage, the teachers urged students to think and discuss issues related to the unit's subject, so as to evaluate the learning process.
It has to be noted that the teachers' role, as derived from the above framework, was minimal. This helped, even further, to avoid bias towards a tool. In essence, they acted as facilitators of the learning process. Although they initiated, coordinated, or joined students' discussions, they did not force their views. In addition, they avoided providing direct answers to problems or questions. Instead, they tried to guide students indirectly, by providing hints or by drawing their attention to the important parts.

Instruments
Twelve evaluation sheets/tests were devised and used for assessing the learning outcomes. A pretest was also devised for assessing students' prior knowledge of all the learning subjects included in the project. The pre-test was administered a week prior to the beginning of the project, while the evaluation sheets were administered right after the end of a session. All tests consisted of two parts. The objective of the first part (about one-third of the questions) was to examine the acquisition of declarative knowledge (e.g., definitions of terms and/or concepts, facts, and figures) through open-ended, fill-in-the-blank, multiple-choice, and yes-no questions. The objective of the second part (the remaining two-thirds of the questions) was to examine the acquisition of procedural, conditional, and functional knowledge. The questions in this part were rather "tricky," as they required critical thinking, attention to detail, and making connections with different pieces of knowledge. For example, students were asked to make concept maps, give their own examples, provide explanations, and apply to different situations what they had learned. The evaluation sheets were the outcome of a collaborative process similar to the one for the development of the learning material. The authors and the participating teachers suggested a number of questions to be considered for inclusion in the evaluation sheets, explaining their rationale and necessity. Following that, three draft versions for each evaluation sheet were assembled. These drafts were then discussed in a group meeting, in which changes were suggested (e.g., questions were removed, added, or reworded), and the final versions were established.
For examining H2a-e, five out of the twelve factors included in a modular scale, designed for examining digital educational applications  were selected. Specifically, the following factors/items were included: motivation (three items), ease of use (six items), fun/enjoyment (six items), subjective usefulness (six items), and immersion (four items). All questions were presented on a five-point Likert-type scale (ranging from "strongly disagree" to "strongly agree"). The questionnaire was administered three times, during the third session of each tool used. Also, the questionnaire regarding the apps/360 o videos included an additional openended question prompting students to report cases of motion sickness or any kind of discomfort when using the HMDs.

Results
As it was mentioned in the previous section, the sample was forty-four fourth grade students (twenty-one boys and twenty-three girls), who attended a total of twelve two-hour teaching sessions (four for each teaching tool). Data coming from all participants were included in the study, as everyone attended all sessions. All the evaluation sheets were graded (on a 0-100 scale). Following that, for each student, three new variables were calculated, representing their average performance when using the printed material, the web pages, and the apps. The resulting data were then inputted into SPSS 25 for further analysis. Table 2 presents descriptive statistics for the average scores in the evaluation sheets and also the results in the pre-test.
For examining H1, a one-way repeated measures ANOVA test was to be conducted. Prior to doing so, it was checked whether the assumptions for this type of analysis were violated. No issues were found, given that the data were normally distributed, there were no outliers, and the Mauchly's test of sphericity was not statistically significant (Mauchly's W = .92, x 2 = 3.66, p = .160). The results of the analysis indicated there were indeed statistically significant differences between the three tools [F(2, 42) = 36.01, p < .001, η 2 partial = .456]. Post-hoc pairwise comparisons (Table 3), revealed that the use of the apps (M = 70.12, SD = 10.30) produced better results when compared to the web pages [M = 62.80, SD = 13.12, p < .001). The effect size was large (dCohen = 0.90). Moreover, the use of the apps produced better results when compared to the printed material (M = 57.70, SD = 14.75, p < .001). The effect size, in this case, was very large (dCohen = 1.12). It has to be noted that students' performance when using the web pages was better compared to their performance when using the printed material [p = .009, dCohen = 0.49 (medium)]. Given the above, H1 is confirmed; when teaching primary school students subjects related to EE, the use of 360 o videos yields better learning outcomes compared to other teaching tools, such as printed material and web pages. Prior to analyzing the results in the questionnaire, its internal consistency was assessed using Cronbach's alpha and was found to be very good (α = .817). The reliability scores of the five factors were also very good (α = .822 to .865). For examining H2a-e, fifteen variables were calculated, representing students' averages in each factor's items and for each tool (five factors X three tools) ( Table 4). Given that the data were not normally distributed and given that the sphericity assumption was violated in all cases, a series of Friedman's tests by ranks (the non-parametric equivalent of the one-way ANOVA with repeated measures) were conducted. The results indicated that H2b and H2d had to be rejected as there were no statistically significant differences in subjective usefulness (χ 2 = 3.38, p = .147) and in motivation (χ 2 = 2.08, p = .353). Therefore, it can be concluded that primary school students find 360 o videos, printed material, and web pages equally useful and equally motivating.
On the other hand, statistically significant differences were noted in immersion (χ 2 = 10.50, p = .005), enjoyment (χ 2 = 5.23, p = .033), and in subjective ease of use (χ 2 = 16.89, p < .001). Post-hoc pairwise comparisons in these factors (using a series of Wilcoxon's signed-ranks tests) revealed that:  Students had more fun and enjoyed the use of the apps more when compared to the printed material [z = -2.91, p = .004, dCohen = 0.65 (medium)]. On the other hand, students equally enjoyed the use of the apps and web pages (z = -1.78, p = .075). Moreover, students had more fun when using the web pages compared to printed material [z = -2.10, p = .036, dCohen = 0.46 (medium)]. Thus, H2a is partially accepted; primary school students enjoy the use of 360 o videos more only when compared to printed material.  Students considered the apps easier to use when compared to the printed material [z = -3.67, p < .001, dCohen = 0.85 (large)]. On the other hand, students found the apps and web pages as equally easy to use (z = -0.63, p = .526). Moreover, students regarded the web pages as easier to use when compared to printed material [z = -4.31, p < .001, dCohen = 1.04 (large)]. Thus, H2c is partially accepted; primary school students find 360 o videos easier to use only when compared to printed material.  The sense of immersion was stronger in the apps when compared to printed material [z = -2.96, p = .003, dCohen = 0.67 (medium)], as well as with the web pages [z = -2.56, p = .010, dCohen = 0.57 (medium)]. No statistically significant differences were observed when comparing the printed material and web pages (z = -0.110, p = .912). Thus, H2e is accepted; primary school students' sense of immersion is greater in 360 o videos than in printed material and web pages.
Finally, it has to be noted that none of the students reported motion sickness or severe discomfort when using the HMDs. Around half of them (N = 23) reported minor discomfort (e.g., irritation of the facial skin, pain in the nasal bone, and eyestrain) in cases where the headsets were used for more than thirty minutes.

Discussion
The data analysis, as presented in the preceding section, confirmed that 360 o videos certainly are of educational value. On the other hand, some rather perplexing results emerged, to be discussed in the following paragraphs. Students' scores in the pre-test highlight an issue that bears further discussion. Indeed, they were able to answer correctly about 40% of the questions (see Table 2), indicating a rather limited prior knowledge on environmental issues. Not only that, but a closer examination of their responses, revealed that their answers reflected most of the problems reported in past research, such as inability to connect cause and effect (Ahn et al., 2016), to define core environmental concepts (Visintainer & Linn, 2015;Yücel & Özkan, 2015), and to understand the functions of ecosystems (Brehm et al., 1986;Griffiths & Grant, 1985;Munson, 1994). Given the attention environmental issues receive in the mass media, one might have assumed that students had -at least-some basic knowledge about these issues. On the basis of the results, this assumption was not supported. Although the examination of the underlying reasons for such mistakes was beyond the study's scope, it is plausible that they indicate a lack of systematic teaching on the relevant subjects. In fact, according to the results in the questionnaire, students considered the three tools as equally useful. One way to interpret this result is that because students did not know much, they thought that all three tools helped them to learn something.
There was a noteworthy change in student scores in the evaluation sheets; depending on the tool that was used, 60% to 70% of the questions were correctly answered, signifying a 57% to 80% positive change. Whether these results are satisfactory is a matter of debate. Strictly statistically speaking, the significant differences between the three tools, point to the logical conclusion that 360 o videos are more effective in raising student awareness regarding environmental issues, confirming the existing literature on the learning outcomes from the use of 360 o videos in the context of EE (e.g., Ahmad et al., 2019;Ritter III et al., 2019). The fact that the effect sizes were large (when compared to web pages) and very large (when compared to printed material), reinforces the validity of this assumption. Also, one has to take into account that EE is a complex and difficult teaching/learning subject (Fauville et al., 2014) and that the evaluation sheets examined the acquisition of declarative knowledge, but the main focus was on procedural, conditional, and functional knowledge (as noted in the "Instruments" section). In light of the above, it would not be an overstatement to say that 360 o videos allowed students to develop a solid body of knowledge about environmental issues.
Then again, it cannot be overlooked that the use of web pages and printed material also had a measurable positive impact on student knowledge; thus, all the tools that were examined can be characterized as effective. Considering the learning outcomes together with the fact that the process for developing apps that utilize 360 o videos is both time-consuming and complicated and that some additional hardware is required (i.e., smartphones and HMDs), a skeptic might support that it does not merit the trouble and that it would be better to stick to tools that are already commonly used or that are easy to introduce in the school environment. On the other hand, it can be counter-argued that even the slightest advantage a tool has over other tools is crucial if we are to succeed in infusing young students with eco-friendly values. This debate is not among the ones that can be easily settled, as it reflects the broader and ongoing discussions for the educational value of ICT tools.
Nevertheless, what remains to be answered is why 360 o videos yielded better results. The fact that students acquired more declarative, procedural, functional, and conditional knowledge, leads to the logical assumption that they mastered both EE's core and threshold concepts. It has to be stressed that, as noted in the "Materials" section, the instructional material was extensively reorganized, the emphasis was put on certain concepts, easy to grasp examples were provided, and no assumptions whatsoever were made regarding students' prior knowledge on environmental issues, following the guidelines for threshold concept teaching (Cousin, 2006). In this respect, it can be assumed that 360 o videos were better aligned to the above organization/presentation of the teaching material.
Τhe teaching framework is also a probable explanation. Indeed, Bybee et al.'s 5Es (2006) are considered rather effective in the teaching of science subjects. On the other hand, the above framework was applied in all tools and not only in 360 o videos; thus, it provides a good explanation for the good results as a whole. Other explanations that have to be ruled out are motivation and the effect of enjoyment. Both are considered key advantages of 360 o videos (e.g., King-Thompson, 2017;Lee et al., 2017;Xie et al., 2019). Yet, in this study, it was found that, although 360 o videos were highly effective in motivating students to learn (M = 4.12, SD = 0.93) the three tools were equally motivating. In addition, although students enjoyed the 360 o videos quite a lot (M = 4.36, SD = 0.58), they equally enjoyed the use of web pages. Though these results per se are puzzling, it seems that the 360 o videos failed to produce the novelty effect, a term that describes students' increased interest and performance due to the introduction of a new technology in their teaching. That is because, in this study, it was found that students did not view 360 o videos as something exceptional, probably because the apps lacked some features as elaborated in the following section. While the lack of the novelty effect can be viewed as a limitation of the study, it can also be viewed as an advantage. This, in turn, can result in students not paying attention to what they are supposed to learn (Rupp et al., 2016).
Students did not report cases of motion sickness or severe discomfort. Minor problems were mentioned, similar to the ones reported by the majority of users (Carnegie & Rhee, 2015). Motion sickness was avoided perhaps because, in all the 360 o videos included in the apps, the user was static; moving in different areas or transitions between videos was instantaneous through the embedded hot spots. Moreover, students did not have difficulties in using the apps, given that they found them equally easy to use as the web pages. Ease of use is a key factor, affecting the quality of user experience, the learning outcomes, motivation, and enjoyment, not only in 360 o videos but also in other applications such as the ones based on virtual reality (Fokides & Atsikpasi, 2018).
The sense of immersion was found to be elevated in 360 o videos, in line with other studies that examined this factor (e.g., McKenzie et al., 2019;Rupp et al., 2016). This finding offers a strong explanation for the better learning outcomes with the use of 360 o videos. That is because, a body of literature suggested that it has a positive impact on the learning outcomes (Lee et al., 2010), as it has a positive effect on student attention and engagement (Ahn et al., 2016) and as it allows a higher personal connection with the environmental problems and nature as a whole (Gehlbach et al., 2015). Moreover, this finding has to be viewed together with the absence of the novelty effect, also found in this study. While over-excitement due to the novelty of the experience and distraction are associated with diminished learning outcomes, they are also commonly associated with the negative effects of immersion (McKenzie et al., 2019;Rupp et al., 2016). Thus, it is quite logical to assume that their absence allowed the positive impact of immersion to become even more prominent.

Implications for Research and Practice
Contrasting the results in the pre-test with the much better results in the evaluation sheets (regardless of the teaching tool that was used), the importance of EE interventions/projects becomes clear. Moreover, the study at hand extends the existing literature as it (i) utilized 360 o videos which are not commonly used in projects related to EE and (ii) comparatively examined and quantified the impact of three tools on student knowledge about environmental issues. A number of the study's findings might prove useful to the experts involved in the development of applications utilizing 360 o videos. It was found that students regarded all tools as equally motivating. In addition, enjoyment and fun were not that different in 360 o videos and web pages. The above can be interpreted as a partial "failure" to exploit the full potential of 360 o videos. A body of literature suggested that both enjoyment and motivation can be facilitated if the educational applications (regardless of the underlying technology) include game-like features (e.g., Faiola et al., 2013;Kozlov & Johansen, 2010). Consequently, it is advised that software developers make available such features in apps utilizing 360 o videos as well. Usability was not an issue, as students found 360 o videos and web pages as equally easy to use. In the apps with 360 o videos, navigation and interactions were triggered by focusing on embedded hotspots. There is a trade-off between this type of interaction triggering and the one through hand-tracking that developers should take into consideration. The former is relatively easily implemented and requires no additional hardware; the latter offers a more natural way to interact (Miller & Bugnariu, 2016), but hand-tracking devices are required.
Although educators can consider integrating 360 o videos into their everyday teaching, this does not come without a price. First, the use of 360 o videos does not necessarily mean that positive learning outcomes will be achieved; the content and the context in which a tool is used are important as well. Therefore, for a given subject, educators have to reflect on whether 360 o videos are suitable and if they have advantages over other tools. A well-defined teaching framework, similar to the one suggested in this study, is also strongly advised, so as to accompany the use of 360 o videos with meaningful activities and avoid students' distraction because of their novelty. Familiarization and some training in the use of HMDs are also advised, as with other technology tools used by young students (Fernández-López et al., 2013). Finally, time is of essence. Twoteaching-hour sessions were enough for conducting the activities and for using the apps. Alas, it was hard to find two consecutive teaching hours and to fit them in an oversaturated school timetable, even if this was temporary and for a short period. On a broader level, this points to the need to reform the primary school curriculum and timetable and come up with new ones that do not hinder the use of innovative and/or technology-based instructional methods.

Limitations and Future Research
Although the results quite clearly demonstrated that 360 o videos are effective teaching tools, certain limitations need to be addressed. Although the sample size was more than enough for the type of statistical analysis that was conducted, it could have been larger. Other age groups could have been considered for inclusion in the study. Given the complexity of EE, one might also argue that the number of sessions was limited. Therefore, some concerns and reservations may be raised for the generalizability of the results. Probably the study's most significant limitation is that it focused on knowledge acquisition; the impact of 360 o videos on EE related attitudes, views, and behaviors was not examined. All the above limitations can be attributed to the study's highly exploratory nature. As the literature is quite limited, the primary concern was to shape a -rather general-idea on the pros and cons of 360 o videos and, depending on the outcomes, to plan a series of follow-up studies. In this respect, future studies can target younger or older students. It would also be interesting to record educators' views on the usefulness and applicability of 360 o videos in everyday teaching. Their impact on attitudes and behaviors, in the context of EE, is definitely worth examining. Long-term interventions and the use of qualitative research tools (e.g., interviews) would be of great help in understanding 360 o videos' educational potential. To this end, comparing the learning outcomes with the ones of other technologies (e.g., immersive virtual reality) would also be useful. Finally, it would be interesting to examine the impact of 360 o videos on learning through the lens of the Technology Acceptance Model (Davis et al., 1989) and try to determine whether students' intention to use them during their teaching is correlated with the learning outcomes that this technology is able to produce.

Conclusion
In the context of EE, the use of 360 o videos was examined and the learning outcomes were compared to the ones of other teaching tools. Overall, it can be argued that 360 o videos are effective tools for promoting primary school students' knowledge about environmental issues. What is more, an instructional framework for integrating them into teaching was proposed and tested. In this respect, the study's findings might prove useful to educators and researchers alike in planning their future steps, as more research is needed for fully understanding their usefulness and for finding ways to exploit their educational potential.