A systematic review and mapping of the literature of virtual reality studies in earth science engineering education

Today, earth sciences engineering education programs face with serious problems. It is often not possible to provide experience on different geographical environments or materials. Considering these problems of earth science related engineering education programs, virtual reality environment potentially can provide technology enhanced training alternatives. However, in the literature there are not many studies showing how virtual reality can enhance the earth science related engineering education programs. Accordingly, the aim of this mapping of the literature and systematic review study is to explore studies dealing with virtual reality in the earth sciences engineering education programs. Through a rigorous screening process, the first search in the Scopus database provided 150 studies, where 19 of them were selected in the course of this study and the mapping of the literature report was drawn with all these articles. Additionally, with seven of these articles a systematic review is conducted. The mapping of the literature report shows that studies on this topic have increased in recent years and have been published in a variety of journals. The number of China and USA origins studies are remarkable among the examined studies, which are predominantly in the fields of mining engineering and geological engineering. As a result of the in-depth analyses, the systematic review report indicates that virtual reality applications in the earth sciences engineering education positively contribute to the learning outcomes and experiences of students/trainees from different perspectives.


Introduction
Recent developments in information and communications technology have brought along significant changes and transformations in the educational field. Many innovative technologies put into practice in this context have evidently changed and improved educational practices. Virtual reality (VR) is regarded as one of these technologies. The concept of VR became popular especially as of the late 1980s and has found important application areas for itself since then (Reznek et al., 2002). When the initial definitions of VR are considered, hardware components are seen to be featured. As a matter of fact, Steuer (1992) states that VR environment was initially defined as composed of certain technological hardware such as computer components, earphones, headmounted displays (HMDs), and motion detection gloves. Different definitions of VR have been made by researchers who have criticized the views associating VR only with hardware components and suggested that experiences should also be incorporated in its definition. According to Stone (1991), VR is a multimedia developed to increase human-machine communication. Heim (1998) VR a technology that persuades the participant that he or she is actually elsewhere by replacing primary sensory input with data received by a computer. In light of these definitions, VR is widely defined as using 3D environments generated by computers which address one or several of individuals' senses and in which individuals can immerse and interact in and utilize virtual or real objects or both of them together (Burdea & Coiffet, 2003;Guttentag, 2010).
VR combines HMD for head-tracking, multimedia components for sound and music, and various hardware components for allowing immersing, navigating, and interacting in the environment. With the use of such components, VR prepares and offers environments that enable individuals to immerse in a 3D simulated world (Li et.al., 2017;Huang & Chen, 2019). By these means, VR offers significant advantages to educators and trainees. Because of such advantages it provides, VR technology is considered to be a cost-effective and efficient tool compared to conventional methods (Li et al., 2017). Accordingly, VR technologies have a wide area of application from medicine to education, industry, and entertainment, among many others (Bracq et al., 2019;Gallagher & Cates, 2004;Martín-Gutiérrez et al., 2017;Palter & Grantcharov, 2010;Rizzo et al., 2014;Vergara et al., 2017;Yung & Khoo-Lattimore, 2019). By virtue of its unique characteristics (e.g., offering close-to-real-world experiences through an environment that is absolutely harmless), VR was born as an excellent tool for education that currently has a wide usage in a variety of fields such as aircraft, aviation, military, and surgery (Zhang, 2017).

VR Applications in Earth Sciences Engineering Areas
Recent developments in VR technologies, fall in their prices, and the advantages they provide such as 3D environment and interaction have attracted the attention of geoscientists, which, in turn, has paved the way for VR technologies to be used in earth sciences engineering (Havenith et al., 2019). In particular, VR has a significant potential in the field of earth sciences as it allows exploring inaccessible areas of the past, present, or future or existing remote areas not only by static objects such as videos and photos but also by processes reflecting their true dynamics (Halik & Smaczyński, 2018;Havenith et al.,, 2019;Lin et al., 2015). Data visualization and 3D modelling have been frequently studied in the field of earth sciences in recent years (Jones et al., 2009;Romañach et al., 2015;Yang & Wang, 2019;Yang et al., 2019). With the development and spread of HMD systems and relative fall in their prices, VR systems now allow visualizing, integrating, manipulating, and querying geospatial data through embodied experiences (Halik & Smaczyński, 2018;Huang & Chen, 2019;Zhao et al., 2019). In other words, VR is an environment that offers a lot of potential for data visualization. Immersing himself in data through VR, the user can benefit from the larger space offered and more natural interactions and analyze multidimensional data visually (Rink et al., 2014;Wang et al., 2020). Similarly, another advantage provided by the use of VR systems for the purpose of geovisualization is that it allows the analysis of terrain changes over time with no need for physical presence in the studied area and in a way that is closest to natural time (Halik & Smaczyński, 2018).
In parallel with these, earth scientists have used VR technologies in their attempts to deal with the problems that may be involved in visualizations and analyses of data and environments related to earth sciences (Zhao et al., 2019). Klippel et al. (2019) state that earth sciences are an observational discipline, and hence observing at any time will make significant contributions to people. In this regard, through use of the opportunities offered by immersive experiences at the highest level, individuals can be immersed in 3D environments that show how formations exist, experience germ-level processes, or develop a regional geological structure understanding (Lin et al., 2015;Klippel et al., 2019). All these experiences can be linked to learning objectives in the field of geoscience, where students need to learn and comprehend challenging content by synthesizing observations and data (Klippel et al., 2019). Many researchers consider VR as one of the most important tools for 3D geographic information systems (GISs) and data visualizations in different areas of earth sciences (Helbig et al., 2014;Huang & Chen, 2019). As a matter of fact, Havenith et al. (2019) report that their experience involving the application of VR in geoscience classes shows that VR will obviously help data visualization and approach multiple geoscientific problems containing complex geometries.
Among important advantages of VR technologies are the solutions they provide for training in hazardous working conditions without users being exposed to any real hazards (Isleyen & Duzgun, 2019). Mining is a typical high-risk and hazardous field, and it is important that individuals working in this field get adequate training (Nickel et al., 2019;Stothard & Laurence, 2014;Zhang, 2017). Accordingly, VR has attracted the attention of researchers in the field of the mining engineering education and industry needing to achieve high safety standards, and many studies have been carried out in this field (Chen et al., 2016;Grabowski & Jankowski, 2015;Mitra & Saydam, 2013;Zhang, 2017). VR applications have also become an ideal ground to improve decision-making processes because trainees and users experience various scenarios in this environment and may identify potential problems without facing any real-world risks and their consequences (Isleyen & Duzgun, 2019;Nickel et al., 2019;Stothard & Laurence, 2014). From this perspective, VR environments that focus on improving decision-making skills are realistic, safe, and reliable tools for training individuals in this field (Isleyen & Duzgun, 2019). These applications have made it clear that mining can be considered as one of the fields that will benefit most from the developments in the VR technology (Isleyen & Duzgun, 2019).
The appearance of VR applications in many areas from education and health to entertainment and tourism in recent years has led to systematic review and mapping of the literature studies for the description of the studies on this topic. Systematic review and mapping of the literature studies are important studies conducted to reveal the current status of the studies on a specific topic and to direct future studies (Seuring & Müller, 2008). Many researchers have carried out systematic review and mapping of the literature studies on VR applications in different fields. Some of these studies have been conducted in the fields of education (Martin-Gutiérrez et al., 2017;Mikropoulos & Natsis, 2011), healthcare/medicine (Bracq et al., 2019;Dascal et al., 2017;Malloy & Milling, 2010), surgery (Moglia et al., 2016, and tourism (Beck et al., 2019;Yang et al., 2017;Yung & Khoo-Lattimore, 2019). However, as far as we can see based on our literature review, there is no such study covering the field of earth sciences engineering. Given the fact that studies on this topic have become widespread in recent years, the present study is necessary and significant to describe the studies in the field, identify the current trend, and light the way for future studies. It is obvious that VR studies have gained importance in the field of earth science engineering in recent years. VR applications have the potential to contribute significantly to the field due to the dangerous nature of the earth sciences, the examination of hard-to-reach areas and the differences mentioned earlier. In this context, the rationale and importance of the study is clear. In this context, the purpose of the present study is to make a detailed examination of VR studies conducted in the field of earth sciences engineering education through mapping of the literature and systematic review.

Aim and Research Questions
This study aims i) to offer a mapping of the literature on VR applications in the earth sciences engineering education and ii) to provide a deeper insight into how VR can be used to enhance teaching-learning process in the earth sciences engineering education. While the first aim points to mapping of the literature, the second one refers to a systematic review. The mapping of the literature questions (MoLQ) to be answered in the context of the study are as follows: MoLQ1: How many articles were published over the years? MoLQ2: In which fields were the articles published? MoLQ3: In which journals were the articles published? MoLQ4: Which countries are the articles from? MoLQ5: What are the numbers of participants, types of data collected, and data collection tools in the articles? MoLQ6: What kinds of analyses were conducted in the articles?
The systematic review question (SRQ) to be answered in the context of the study are as follows: SRQ1: What are the effects of the use of VR on learning outcomes in the earth sciences engineering education?
Our ultimate goal in this study is to present a synthesis of the existing studies in this field to scholars, curriculum makers, and practitioners, thereby introducing a thorough description of state of the art and providing a perspective that will further studies in this field.

Research Design
This study employed mapping of the literature and systematic review approach together. Rowley and Slack (2004) define systematic review as an approach that is used for facilitating the identification, evaluation, and interpretation of studies in a selected field and explores concepts, practices, and theories. Similarly, Seuring and Müller (2008) emphasize that literature review has two main objectives in general: 1) summarizing existing studies by identifying patterns, themes, and problems; 2) identifying the conceptual content that makes up the field and contributing to its theoretical development. The study takes the PRISMA statement as a basis. The PRISMA statement was developed for authors to conduct systematic reviews and reporting concerning meta-analyses and to provide them with a systematic reporting way (Moher et al., 2009). For this reason, PRISMA was used as the protocol to be followed in systematic review studies and the method part of the article was developed according to this protocol.

Search strategy and terms
The study conducted a systematic electronic search in the Scopus database. When systematic review studies in the literature are examined, it is seen that more than one database is searched in general (Bramer et al., 2017). In addition, the number of repetitive articles and the efforts to eliminate them at the preliminary stage of the studies are noteworthy (Kwon et al., 2015). Scopus is the most comprehensive abstract and citation database of peer-reviewed literature with the intelligent tools it has to monitor, analyze, and visualize research. Scopus provides the most extensive insight into research results in such fields as science, technology, medicine, social sciences, and the arts and humanities (Scopus, 2020). Moreover, Scopus covers over five thousand publishers, Elsevier, Emerald, IEEE, Sage, Springer, Taylor & Francis, Wiley-Blackwell being in the first place, across the world, and this number is increasing every passing day (Scopus, 2020). Verifying this, Yung and Khoo-Lattimore (2019) determined that the same search string yielded the most results in the Scopus database. For this reason, the researchers conducting the present study deemed using only Scopus in the study.
For the search terms, an outline was drawn up by examining the articles published in this field under the leadership of the last author. This draft was then discussed with five domain experts, and the search terms were decided to be as follows: (-virtual reality‖ OR VR OR -immersive visual*‖) (-earth scien*‖ OR -earth sciences engineering‖ OR -earth sciences engineering education‖ OR -geological engineering‖ OR -geophysical engineering‖ OR -geomatics engineering‖ OR -geotechnic* engineering‖ OR -mining engineering‖) These two groups were combined with the conjunction AND. The search was conducted on the search field -Article title, Abstract, Keywords‖ on 10 October 2019. In order to access all studies in the field, no date limitation was applied in the search.

Eligibility criteria
In this study, inclusion and exclusion criteria were determined based on PICOS in accordance with the PRISMA protocol. Inclusion criteria were set as articles containing VR applications related to earth science engineering, -peer-review articles, -the articles written in English whose full-text versions are available.
Exclusion criteria, on the other hand, were determined as virtual learning environments without VR only 3d simulations and data visualizations pre-university level (k12) systematic reviews, trade publications, book chapters, position papers, and conference proceedings.

Study selection
Through the literature review performed in the Scopus database on 10 October 2019, 150 records which were exported to Excel for data management were reached. The first and third researchers took part in the whole screening process. In the first stage, based on the inclusion and exclusion criteria, non-journal peer-review articles (e.g., conference papers, book series/chapters, editorial notes, trade publications, review articles, non-English, unavailable full-texts, etc.) were eliminated (n = 106). Then the titles and abstracts were examined, and 23 articles were removed. After that, the full texts of the remaining 21 articles were read to ensure that they met the inclusion/exclusion criteria. At this stage, 17 articles were excluded. This being the case, only four articles remained. Finally, 15 articles were added through retrospective and prospective reference searching (snowballing). All steps were performed by the cooperation of two researchers to avoid research and researchers' bias. These researchers resolved their disagreements by working together on the articles and made the final decisions together.
Since there was not much research in the field, the researchers adopted using a combination of mapping of the literature and systematic review approach. It is stated that the use of these two approaches together gives the best result in terms of describing a field (Cruz-Benito, 2016). Hence, the researchers made a small adaptation in the conventional PRISMA protocol. Accordingly, 19 articles in the field were subjected to mapping of the literature. According to the conventional PRISMA protocol, only experimental studies are included in the systematic review as required by PICOC rules. In the present study, however, due to the scarcity of studies in the field and to provide a more thorough description of the field, systematic review was conducted with a total of seven studies involving evaluations. Figure 1 presents the adapted PRISMA flow diagram showing the detailed information flow throughout the literature review process.

Figure 1
Adapted PRISMA flow diagram showing the review research process *,**: The articles subjected to review are marked with * in the references section. All marked with *,** indicate the articles used for mapping of the literature while ** indicate only the articles used for systematic review among them.

Data Analysis
All the researchers first decided how to encode the articles, and the two researchers (first and third) analyzed the articles together through content analysis. Problems emerging in coding were solved by discussing together. Support was received from other researchers when necessary. Two consecutive process steps were followed for a total of 19 studies analyzed.
Firstly, as relevant metadata were entered in Excel, all articles were descriptively analyzed for the mapping of the literature analysis. With this descriptive analysis, the distributions of the articles by year, research area, journal, and country were revealed. In addition, the types of participants and data collection tools included in the articles and the analyses performed were determined. Afterwards, seven articles were subjected to in-depth content analysis for systematic review analysis. With this analysis, the effects of VR applications on learning outcomes as well as the key streams and trend of research in the field were identified and analyzed. In this way, significant opportunities for further research directions were highlighted. Finally, two other researchers randomly selected four of the articles and conducted the check process. This check process was performed to decide whether the articles would be included in the systematic review and to confirm their coding. By this means, research and researchers' biases were minimized. In general, in the analysis process, the first and third authors applied the coding scheme while the second and fourth authors monitored the process and provided suggestions about the method.

Findings
In this section, findings concerning mapping of the literature and systematic review analyses are given under separate titles.

Mapping of the Literature Report
19 articles selected within the scope of the study were analyzed, and findings to answer the mapping of the literature questions (MoLQ) were extracted. Figure 2 graphically illustrates the answer to MoLQ1. Among the articles (n = 19) reached within the scope of the study, the first article was published in 2008. Figure 2 shows the distribution of the publications from 2008 to 2020, with articles published in this field reaching the peak in 2019, when nine articles were published. An article was published in a forthcoming issue of 2020 before 2020 came.

Figure 2 Distribution of the articles by year
With respect to MoLQ2, it was seen that the articles were published in different sub-fields of earth sciences. Table 1 shows the distribution of the articles by field. Although five different subfields appear, mining (n = 9; f = 47.37%) and geology (n = 6; f = 31.58%) share the first two places.  To answer MoLQ3, the journals in which the articles were published were examined. Table 2 presents the journals in which the articles in this field were published. The articles were published in 15 different journals. Among these journals, Environmental Earth Sciences, International Journal of Digital Earth, International Journal of Mining Science and Technology, and Pure and Applied Geophysics contain two publications while the other 11 other journals include one article each. The journals are mostly related to earth sciences, and only three of them are about educational sciences. With regards to MoLQ4, the countries of origin of the articles were examined. They are graphically presented in Figure 3. It was observed that the articles were from seven different countries, with China (n = 6) and USA (n = 5) ranking first.

Figure 3 Distribution of the articles by origin
For MoQL5, the numbers of participants of the articles, the types of data collected in the articles, and the data collection tools were examined. In this respect, firstly, it was investigated whether the data were collected from students/participants. As can be seen in Figure 4, 12 of the articles did not contain any evaluation, whereas seven involved evaluations such as experimental study and collecting participants' opinions.  Figure 5 shows the number of participants in seven articles including evaluations. As seen in Figure 5, the numbers of the participants contained in the articles vary from 5 to 37.

Figure 5
The numbers of the participants in the articles Figure 6 provides number and frequency information on what kinds of data were collected in seven articles involving evaluations. As shown in Figure 6, quantitative data were collected in three studies, qualitative data in two studies, and mixed (quantitative + qualitative) data in two studies. Figure 7 shows the data collection tools used in seven articles involving evaluations and their relevant numbers. The data collection tools used in the articles were divided into two categories: questionnaire/scale and interview/open-ended questions form. As more than one data collection tool was used together in some articles, a total of 18 data collection tools were detected in the articles.  To answer MoLQ6, the data analyses made in the articles were examined. Figure 8 shows the analyses made in seven articles involving evaluation and the numerical information about them. As shown in Figure 8, a total of 15 analyses in nine categories were conducted in the articles. The most frequently conducted analyses are superficial qualitative data analysis (n = 4) and frequency distribution (n = 3).
Finally, Table 3 presents data summarizing the findings provided in the mapping of the literature analysis. This report provides, in a sense, a descriptive analysis of the articles and a map of the studies. In addition to this analysis, there is a need for a deeper analysis in order to provide a thorough picture of the selected articles in order to answer the systematic review research question. Therefore, the next section presents the findings concerning systematic review in detail.   Most articles were published in 2019 MoLQ2 Most articles were published in the field of mining

MoLQ3
Most articles were published in Environmental Earth Sciences, International Journal of Digital Earth, International Journal of Mining Science and Technology, and Pure and Applied Geophysics (two publications in each) MoLQ4 Most publications are of Chinese origin

MoLQ5
The highest number of participants is 37; most frequently used data collection tool is questionnaire/scale; and most frequently collected type of data is quantitative MoLQ6 The most common analysis is superficial qualitative data analysis

Systematic Review Report
To answer SRQ1, a total of seven articles involving evaluations or participant/student opinions were analyzed in depth. Information about each article including author, field, aim, method, and outcomes is presented in detail in appendix Table 1 (TA1) in the way such information is provided in the original articles. Five of the articles that were examined through systematic review were conducted in the mining field and two in the geological field. When the years of the articles are examined, it is seen that one article was published in each one of 2014, 2015, and 2017 while four articles were published in 2019. The numbers of participants in these articles ranged from 5 to 37; quantitative data were collected in three of them, quantitative data in two of them, and and mixed data (quantitative+qualitative) in two of them; questionnaire/scale and interview/open-ended questions form were used as data collection tools in the articles; and finally, various analyses, superficial qualitative data analysis (n = 4) and frequency distribution (n = 3) being in the first place, were made in such articles. A summary of each article can be presented as follows: Isleyen and Duzgun (2019), conducting a study in the field of mining, aimed to provide a methodology for developing VR-based training simulations for the assessment of fall of roof hazard in tunneling operations. With the simulation they developed, they planned to train people 0 1 2 3 4

Analysis type
in order to improve their decision-making skills for cases where quick decisions are necessary for an operation. They collected qualitative data from five participants through an interview form. According to the results of their study, VR provides a safe training environment to increase the efficiency of visual detection of all kinds of ground control problems, geological structures, and instabilities. The study also determined that the application of such VR simulations in the engineering education is likely to have a positive impact on students' understanding of tunneling operations and geological features and also has a high potential to increase work-place efficiency and improve decision-making processes. They also report that those who are trained in this environment have the chance to judge the consequences of their own hazard mitigation decisions and gain experience by making a self-assessment. Among the suggestions of the participants was that demonstration of the results of numerical analysis within the virtual tunnel would provide better and more intuitive visualizations. Moreover, changing the tunnel geometry based on the displacements before and after the support installation may provide a better representation to understand the impacts of hazard prevention and offer a more immersive environment.
In their geological study, Klippel et al. (2019) aimed to test how much virtual experience can suspend the participants' belief in being in the field compared to the actual experience. They formed intervention/control group with 37 participants and collected quantitative and qualitative data via questionnaires/scales and open-ended questions forms. Based on quantitative and qualitative data analyses, they concluded that the two groups did not display any biases against immersive experiences or potential for success in that lab, but a significant difference was detected in the assessment of learning experiences. The students using the immersive virtual environment (intervention group) attributed a significantly higher rank to their learning experiences compared to the students who participated in the physical field trip and responded to the question -I learned a lot from the field trip‖ (control group). The intervention group students also received very high lab grades. While the female students in the intervention group achieved statistically significantly higher scores than those in the control group, no difference of this kind was detected for the male students. Furthermore, though it was concluded that the intervention group students liked their experiences more than the control group students, the spatial situation model did not yield any significant difference. Among the results of the study that can be considered negative, though limited in number, are that the intervention group students suggested ways to improve instructional quality such as more interaction, note-taking, and allowing for peer work and that a student reported symptoms of a cyber-sickness in the postquestionnaire. There were also some students who expressed their concerns about inadequate image clarity. A general overview of the results of the study also revealed the advantages virtual field trips offered with regard to entertainment, learning experience, and actual lab scores.
Lütjens et al. (2019), in their geological study, aimed to develop a workflow for the purpose of investigating the advantages, constraints, and possible applications of immersive and intuitive 3D terrain visualizations through VR and presenting large-scale terrain datasets in VR for existing mid-range computers. In that study collecting qualitative data from 10 participants, the researchers concluded that VR was generally perceived as user-friendly and easy to use, that it gave all participants a better representation of the terrain topography in comparison to the twodimensional representation, and that no participant felt any discomfort/motion sickness while navigating the virtual world.
Carrying out a study in the field of mining, Nickel et al. (2019) aimed to investigate if the usage of a tutorial session before using the mining simulator could sufficiently decrease performance variability and raise participant performance metrics. In the study conducted with 19 participants, tutorial and non-tutorial groups were formed, and quantitative data were gathered through questionnaires/scales. The analysis of these data showed no significant relationship between personality questionnaire scores and the number of collisions or perception response time. However, while the two groups displayed high standard deviation scores for collisions and perception response time in the beginning, the variability in the tutorial group decreased over time, and the groups began to yield more consistent scores in the simulator after 10 minutes of use.
Zhang (2017), doing a study in the field of mining, aimed to develop and test an immersive and intuitive VR training system prototype to be used in the mining industry. Quantitative data collected from 10 participants via 5-point Likert-type questionnaire were analyzed. According to the results of the study, the HMD-based intuitive VR training system was 1.5 to 2 times better compared to the screen-based system in terms of immersion, intuition, interaction, and ease of use, provided better user experience, and was slightly better in terms of ease of learning. A negative result of long-time utilization of HDM can be mentioned as that the user feels tired or motion sick. Finally, it is indicated that the lack of high-quality content is the most critical disadvantage the current VR training system holds.
Grabowski and Jankowski (2015) aimed to conduct a pilot study to evaluate two different VR environments (moderate and high immersive environments) in the mining education and assess the results. The analysis of the quantitative data collected from 21 people via two different scales revealed that the simulation in high-immersive VR performed much better in all cases (interactivity, innovativeness, ease of use, user-friendliness, ease of learning, functionality, and method preferred). In addition, individuals who participated in the training found the system they used useful, and they felt the positive impacts of the training even after three months.
Finally, Stothard and Laurence (2014), in their study in field of mining, aimed to show that large-screen visualization and simulation technology can be used to describe sustainable mining practices. The analysis of quantitative and qualitative data collected from 12 people revealed that the students liked interaction and found it easy to learn, and they found visualization and simulation positive. Another conclusion of their study is that this visualization system has a great potential for the training of non-experts. Finally, the following research question addressed in the study was answered affirmatively: -Can large-screen visualisation and simulation technology be utilised to depict sustainable mining practices?‖ When the results of the researches, in which a total of seven studies were examined and SRQ1 was tried to be answered, the following findings were reached: VR provides a safe education environment in the field of earth sciences and has the potential to have positive effects on students' understanding of the subject and concepts related to the field. In addition, it is stated that these applications are fun and have a high user experience. It also stands out that it is perceived as userfriendly and easy to use. In addition to the mostly positive opinions, some shortcomings / problems or disadvantages regarding VR applications draw attention, even though they were voiced by a small number of students. High quality image / content problem, cyber disease symptom can be shown among these problems.

Discussion
This study aimed to analyze the articles dealing with VR applications in the field of the earth sciences engineering education through mapping of the literature and systematic review. A total of 19 articles were used for mapping of the literature, and seven of them involving participants' evaluations/opinions were included in the systematic review.
In terms of publication year, nine articles were published in 2019, and one in 2020. This can be considered as an indicator of the increase in VR applications in the earth sciences engineering education in recent years. In addition, the decrease in the prices of HMD-based devices, their getting more widespread to be used, and the abundance of studies in other fields may have led to studies in this field (Havenith et al., 2019;Zhang, 2017). Another remarkable point concerning the publications is that the articles are concentrated especially in the fields of mining and geological engineering. As a matter of fact, 15 of the 17 articles were published in these two fields. One of the main reasons for this may be the fact that especially mining engineering is a risky and dangerous field. Isleyen and Duzgun (2019), Grabowski and Jankowski (2014), and Zhang (2017) emphasize such aspects of mining engineering and industry. Moreover, the fact that 3D visualizations have been carried out in the field of geology, which forms the basis of the earth sciences field, for many years may have positively affected the integration of VR into the studies in this field (Havenith et al., 2019). Kellogg et al. (2008) state that VR systems will be an important tool for earth sciences engineering scientists. The results confirm this opinion.
As to the journals publishing the articles, 15 different journals were detected. The primary journals of publication were seen to be Environmental Earth Sciences, International Journal of Digital Earth, International Journal of Mining Science and Technology, and Pure and Applied Geophysics, which are well-known journals in the field. However, three journals are educational sciences journals. This is considered important because it implies the potential of not only the field journals but also the educational journals as VR studies that require interdisciplinary study and process become widespread. When the countries of origin of the articles are examined, it is seen that China and the USA are in the forefront with a total of nine articles. They are followed by Australia and Germany with three articles. This may be about that the articles were published in the mining field more. As a matter of fact, China, the USA, and Australia are the leading countries in the field of mining, and thus it is natural that most articles are from these countries (Zhang, 2017). Furthermore, a German-based research lab can be considered to have made significant contributions to this field (Bilke et al., 2014).
Only seven articles out of 19 included in the study involved participant evaluations/opinions. Therefore, the numbers of participants in the articles, the types of data collected, the data collection tools used, and the analyses performed were evaluated through these articles. The numbers of participants varied from 5 to 37. Questionnaires/scales were predominantly preferred as data collection tools. Interviews/open ended questions were also used for collecting qualitative data. Superficial qualitative data analysis was seen to be the most preferred one, but various statistical tests were also employed. In general, scarcity attracts attention in the articles in terms of the number of participants and on some other points. The main reason for this may be that many articles involved the preparation and implementation of various scenarios for the use of the system. It may take a long time to conduct such studies with a large number of participants. That qualitative data analyzes were conducted superficially could be due to the fact that these studies were generally conducted by domain experts rather than educational scientists and published in field journals.
As to the effects of VR applications on learning outcomes, the positive effects were featured in the journals in general, while there were also some negative effects reported, though limited in number. However, a number of suggestions and requests from the participants may be helpful in determining the steps to be taken to improve these systems. It is reported in the results of the studies in general that VR generally provides a safe, funny, easy to use, and easy to learn training environment, is likely to positively influence students' learning relevant subjects, offers students a high learning experience, and is user friendly. It is also reported that VR provides a better representation of the terrain topography than the two-dimensional representation; high-immersive VR environments perform better; and VR environments have a great potential for the training of non-experts. Only one student reported cyber-sickness symptom after the application. Another negative opinion is that users get tired and feel motion sick in long-term use. As a solution to these problems, Nickel et al. (2019) argue that the inversion level may be increased and more realistic environments may be produced as artificial intelligence features are included in these environments. According to the researchers, this may help users feel more immersed in the environment and produce more consistent and meaningful responses to the environment. Isleyen and Duzgun (2019), on the other hand, state that moving at scale would minimize motion sickness, which is the most common problem experienced by the participants in VR, and would give a feeling of full immersion. In this context, these methods can be applied and tried in order to solve such problems. Among highlighted suggestions concerning the technical aspects of VR environments are that there is the lack of sufficient quality content, that image quality needs to be improved, and that students should be allowed for more interaction, note-taking, and peer work.
Steps can also be taken to address these problems.

Conclusion and Future Works
The results of the study showed that using VR applications in the earth sciences engineering education has become popular in recent years. In other words, VR systems are a recent development in the earth sciences engineering education. The expansion of VR applications in the fields related to earth sciences, mining and geological engineering being in the first place, has brought along many educational potentials. Such applications, which allow especially students/trainees to be immersed in 3D virtual environments, enable them to gain important experiences especially on dangerous and risky tasks and difficult-to-access and large land areas. Positive views on VR may be expected to encourage researchers who are interested in VR applications in the earth sciences engineering field and thus trigger the increase of studies concerning these applications. Users/participants may be expected to have VR experiences closeto-reality with the improvement of their technical and usage limitations and the expansion of their use. Although there have been numerous systematic review studies on VR systems in different fields in recent years, the present study is significant as it is the first one in the field of earth sciences engineering. Accordingly, the results of this study offer a promising perspective for the education of earth sciences engineers. The potential of transforming a wide variety of scenarios into VR applications, especially in earth sciences fields, may contribute to the integration of VR simulations with earth sciences engineering.
Though this study is significant as it is a first, it covers only peer-reviewed journals contained in the Scopus database in accordance with the systematic review protocol. This may be regarded as a limitation. Different sources such as conference proceedings and theses may be included in the research. This may allow a broader description of the field from different perspectives. In addition, by carrying out more detailed studies in different fields of earth sciences engineering, different studies can be put forward in a more specific field.