The effects of a STEM-based intervention on middle school students' science achievement and learning motivation

The goal of STEM education is to integrate the disciplines of science, technology, engineering, and mathematics in a way that facilitates holistic learning. Students are also supported in developing creative solutions to problems using the STEM approach. Other STEM fields are often linked through engineering and engineering design. Throughout this study, students participated in engineering design-oriented activities that connected STEM disciplines. This study aims to reveal the effect of a learning environment based on the STEM approach in science education on students' academic achievement and motivation. This study used a quasi-experimental, non-equivalent pretest-posttest control group design. Seventh-graders who attended a public middle school in the Aegean Region of Türkiye participated in the study. The research findings showed a significant difference in science achievement between the control group in the curriculum-based learning environment and the experimental group in the STEM-based learning environment. It was found, however, that motivation toward science was not significantly different. The results indicated that the STEM-based learning environment positively affected science achievement.


Introduction
Economic competition has forced countries to revise their educational systems, as it has done in the past.Over the past few years, governments have placed engineering-based ideas at the top of their agendas.Therefore, raising entrepreneurial and creative individuals with advanced problemsolving skills who can question and produce scientific solutions to problems is an essential part of these agendas (Ministry of National Education [MoNE], 2016; Yıldırım & Selvi, 2017).The Organization for Economic Cooperation and Development [OECD] (2010) also emphasized the importance of arranging curricula and pedagogies in a way that would enable students to improve new skills in their lifetime (p.11).
In the United States (US), the National Science Foundation [NSF] implemented an approach (SMET) that combined the disciplines of science, mathematics, engineering, and technology that could contribute to economic growth in the 1990s.This approach was later named STEM [Science, Technology, Engineering, Mathematics] (Portz, 2015).Especially in the 21st century, because many occupations require science, technology, engineering, and mathematics skills, STEM education has a fundamental role in preparing individuals for business (Akyıldız, 2014).In addition, Marginson et al. (2013) emphasized that STEM skills are needed in STEM occupations and other economic sectors.In this context, it can be stated that STEM education emerged due to economic rationale (Williams, 2011;Yıldırım & Selvi, 2017).As noted, STEM education plays an essential role in developing the 21st-century skills that students need for the workforce in the future (Singh, 2021).Because the STEM approach helps to improve 21st-century competencies while providing opportunities to use knowledge of multiple disciplines (Asian Development Bank [ADB], 2021; Moore, Stohlmann, et al., 2014;Peters-Burton et al., 2020).Therefore, it is not surprising that studies have reported improved skills such as problem-solving and collaboration in STEM-based learning environments (Guzey et al., 2020).
Singapore, one of the top-ranking countries on international exams in recent years, such as TIMSS and PISA, has switched to an inquiry-based STEM curriculum (Bagiati et al., 2015).Similarly, Türkiye updated the science curriculum based on an inquiry-based learning approach from an interdisciplinary perspective (MoNE, 2018).Thus, the STEM approach was adopted in Türkiye and included engineering and design skills in the Turkish science education curriculum.Korean STEM education emphasizes STEM's integration-oriented approach to solving real-world problems (Hong, 2022).The Korea Foundation for the Advancement of Science and Creativity [KOFAC] (2016) stated that the rationale of Korean STEAM [A-Art] education is that most realworld problems cannot be solved with knowledge of a single field but can be solved by connecting and using knowledge from various fields.In addition, the Australia Education Council (2015) released the Australia National STEM School Education Strategy Report focused on STEM literacy and skills for 2016-2026.Overall, countries recognize the importance of STEM education and have taken different steps to support STEM education.
STEM integration combines science, technology, engineering, and mathematics disciplines to improve student understanding and increase interest in STEM fields (Harwell et al., 2015;Roehrig et al., 2012;Wang et al., 2011).According to Bybee (2010), the STEM approach requires placing real-life problems in a central position and using STEM fields to solve them.Thus, integrated STEM disciplines help students understand the connections between disciplines (Wang & Knobloch, 2018).Similarly, Shaughnessy (2013) underlined that any STEM activity must have a problem requiring knowledge and approaches from several disciplines for solutions.Moore and Smith (2014) defined STEM integration as merging STEM fields into a single class, unit, or lesson.Öner and Capraro (2016) emphasized that STEM education should integrate four subject areas in an applied context.From a similar perspective, Delen and Uzun (2018) underlined that the basis of STEM is to reveal the applied aspect of the taught content from an engineering perspective.In this framework, some studies emphasize that STEM integration should be design-oriented (Moore et al., 2020).For example, Guzey et al. (2020) underlined that STEM learning experiences based on engineering or technology design-oriented tasks might enrich education multi-dimensionally.Moore, Glancy, et al. (2014) emphasized that most STEM integration efforts are shaped within the framework of learning science, mathematics, and technology content using engineering and engineering design.According to Harwell et al. (2015), the STEM approach uses engineering as a tool to teach other STEM disciplines.Therefore, design-oriented STEM learning environments might help students learn the basic ideas in STEM disciplines (Krajcik & Delen, 2017).Moreover, the engineering design process not only helps to acquire subject knowledge and skills but also creates a platform for developing high-level skills (Richard et al., 2006).
Engineers routinely create solutions to real-world problems using science and mathematics in the design process (National Research Council [NRC], 2009;Sanders, 2009).Engineering is an essential element in STEM education that points to the design process and unifies other fields (Jolly, 2017).According to Moore et al. (2020), engineering is the glue that holds the other STEM disciplines together.Similarly, Truesdell (2014) pointed out that a meaningful way of effectively integrating STEM disciplines is through engineering projects that allow the search for solutions to real-life problems.Students engaged in engineering projects may generate creative solutions to engineering-based problems using mathematics and science disciplines (Rogers & Portsmore, 2004).The engineering design process can help students identify and solve problems, and students can confront their understanding and misunderstanding of science concepts in engineering designbased processes (Capobianco et al., 2018).So, it can be said that engineering frequently provides a context for the STEM approach (Australian Curriculum-Assessment and Reporting Authority [ACARA], 2016).From this perspective, engineering design could be helpful in the implementation of STEM education (Lin et al., 2021).Students should be given opportunities to participate in engineering design activities because engineering design activities could help connect STEM disciplines and develop 21st-century skills (Moore, Stohlmann, et al., 2014).Also, many studies noted that learning activities based on engineering design enhance learning in STEM (Capobianco et al., 2018;Lin et al., 2021).As Clough and Olson (2016) emphasized, connecting science and engineering practices can foster meaningful learning in science education and build a basis for new learning in various contexts (p.373).For example, English and King (2019) reported in their STEM integration study that students also understand fundamental engineering principles while using their knowledge of mathematics and science in a bridge design task.
It has been emphasized by the NRC (2009) that engineering does not have to be a teacherdirected design activity.Instead, it can involve solving an open-ended problem with mathematical and scientific knowledge.Sanders (2009) suggested a "purposeful design and inquiry" (PD&I) pedagogy that combines design with scientific inquiry in problem-solving for integrative STEM education (p.21).Corbett and Coriell (2014) described the engineering design process for middle school students as follows: (1) defining the problem, (2) researching the problem, (3) brainstorming for the solution, (4) choosing a solution, (5) creating and developing a prototype, (6) testing and evaluating the prototype, and (7) improving and redesigning.Similarly, MoNE (2019) mentioned a cycle like the engineering design process, which includes defining needs or problems, researching, developing possible solutions, choosing the most appropriate solution, creating prototypes, testing, evaluating, and sharing solutions.Although a consensus definition of STEM integration has not yet been established, there are some highlights from various applications.For example, STEM integration requires; (1) using real-world problems or context, (2) ideas and skills to connect the disciplines, (3) the disciplines should be integrated (Moore et al., 2020), and (4) using engineering and design as a tool or context to teach other STEM disciplines (Harwell et al., 2015).Based on these highlights, activities incorporating engineering design and real-world problems can help implement STEM education.Engineering design-based science teaching experiences can make STEM learning easy as students reveal solutions to real-life problems (Capobianco et al., 2018).
Today, STEM education and its implications are still on the agenda in many countries for the reasons mentioned above.Therefore, researchers worldwide are conducting various research on STEM education.For example, studies have investigated the effects of STEM education on variables such as achievement, science content knowledge, motivation, attitude, problem-solving, higher-order thinking skills, and career interest (e.g., Angwal et al., 2019;Anwar et al., 2022;Cotabish et al., 2013;Fan & Yu, 2017;John et al., 2016;Kopcha et al., 2017;Lam et al., 2008;Lin et al., 2019;Ngo, 2021;Nugent et al., 2015).As stated earlier, STEM has become a focus area for education in Türkiye in recent years.As a result, many institutions and organizations have established STEM centers and conducted activities and research on this subject.For example, studies have investigated the effects of STEM education on variables such as academic achievement, motivation, scientific creativity, and career interest (e.g., Çalışıcı, 2018;Gülen, 2016;Kırıcı & Bakırcı, 2021;Kızılay, 2018;Koca, 2018;Tekbıyık et al., 2022;Yıldırım, 2016).
STEM education studies provide educators and policymakers with evidence-based information.In this way, authorities may obtain information about the effectiveness of STEM education, which they consider necessary for economic development and include in their agenda.Wahono et al. (2020), in their meta-analysis study, sought the effectiveness of STEM education in Asian student learning outcomes and underlined that many future studies are needed to verify the results.De Loof et al. (2022) proposed that researchers investigate the impact of integrated STEM education on cognitive and affective outcomes.Likewise, Anwar et al. (2022) indicated that more studies are needed to compare the effectiveness of integrated STEM and traditional teaching approaches on student achievement.The authors also suggested that future research also consider the motivation variable.
Electricity is one of the core subjects in the science curriculum.However, electricity subjects are challenging to learn because it contains complex and abstract concepts, and studies consistently report poor student understanding of electricity subjects (Mulhall et al., 2001).In addition, primary school students often have alternative concepts and ideas about the electricity they experience in daily life, and some of the alternative concepts are usually transferred to the secondary education level (Preston et al., 2022).For these reasons, a sample integrated STEM-based learning environment was designed in the electrical energy unit in this study.Therefore, this study aimed to reveal the effect of a STEM-based science learning environment designed in the electrical energy unit on students' academic achievement and motivation, and the following questions were investigated: RQ 1) How does STEM education affect students' science achievement?RQ 2) How does STEM education affect students' science learning motivation?

Research Design
This study used a quasi-experimental, non-equivalent pretest-posttest control group design.Each of the two 7th grade classrooms was randomly assigned to the control and experimental groups.
The research design is shown in Table 1.

Participants
The participant group of the study included 7th-graders attending a public middle school in the Aegean Region, Türkiye, in the 2017-2018 academic year.A sample of 30 students was randomly assigned to the experimental group (6 females and 9 males) and to the control group (7 females and 8 males).

Data Collection Tools
The Electrical Energy Achievement Test (EEAT), which includes nine open-ended questions, was created considering the electrical energy unit learning outcomes of the 7th-grade science curriculum.In addition, two science teachers and a physics education specialist reviewed EEAT regarding comprehensibility and learning outcomes.Students' motivation toward science learning (SMTSL) questionnaire used in this study is a fivepoint Likert-type scale developed by Tuan et al. (2005, as cited in Yılmaz & Huyugüzel-Çavaş, 2007).While the original questionnaire had 35 items in English, two of them were excluded in the adaptation process into Turkish.Therefore, the 33-item Turkish version of the SMTSL was used in this study.

Teaching Material and Procedures
In this study, an engineering design-oriented learning environment was created to solve daily life problems within the framework of STEM disciplines.As stated in the introduction, the engineering design processes defined by Corbett and Coriell (2014) and MoNE (2019) were taken as a guide.In addition, three STEM activities were designed based on the 7th-grade electrical energy unit in science to create a STEM learning environment for the experimental group.In these activities, students seek a design-oriented solution to problems by considering STEM disciplines.Additionally, some criteria were determined for STEM activities to ensure compatibility with learning outcomes.For example, in STEM activities, criteria such as "The tool you will design/produce must convert electrical energy into heat energy" were used so that students could use scientific concepts compatible with the learning outcomes in their problem solutions.For each STEM activity, individual student and group worksheets were prepared, including instructions, criteria, and areas to draw designs and write explanations about the design.In addition, forms were created asking students to explain how and where they used STEM disciplines in problemsolving and were shared with students in each activity.A sample of STEM activity used to support the STEM-based science learning environment in this study is presented in the appendices (see Appendix 1).Before the intervention, the experimental group was given a seminar on the nature of the STEM approach for two hours.Then, the learning-teaching process was conducted with activities based on STEM for six weeks.In the experimental group, the students sought answers to the problems in STEM activities in groups of 3-4 students.Throughout the process, the teacher asked questions to guide the groups about solutions when necessary.The teacher also guided small group discussions and students' problem-solving participation in each lesson.The process in the learning environment, based on STEM activities, is illustrated in Figure 1.
In the control group, the learning environment was designed within the actual curriculum, and a student textbook published by the Ministry of Education was used as teaching material.During this process, activities, experiments, and chapter questions provided in the textbook were used.

Figure 1
The process in the learning environment during STEM activities

Data Analysis
A scoring key was prepared for the electrical energy achievement test (EEAT).A researcher and two science teachers examined the scoring key and made necessary arrangements.The answers given by the experimental and control group students to the EEAT questions were independently evaluated by two science teachers using the scoring key.The agreement rate among the coders was 83% (Miles & Huberman, 1994), and a reconciliation process was experienced when needed.
SMTSL questionnaire is a five-point Likert-type scale of twenty-five positive and eight negative items.In this framework, items are coded by assigning points between one and five.
The normality distribution of the data obtained from the pre-and post-tests of EEAT and SMTSL and the homogeneity of variance were examined through the use of Shapiro-Wilk test and Levene F test, respectively.Table 3 contains the Shapiro-Wilk test results of the EEAT and SMTSL.The Levene F-test p-values were .59and .11for the EEAT pre-and post-test, respectively, and .54 and .99 for the SMTSL pre-and post-test.The tests revealed that each dataset exhibited a normal distribution, and the variances were homogeneous.Independent group t-tests were conducted using pre-and post-test data to determine the effects of the experimental procedure on achievement and motivation.Özsoy and Özsoy (2013) emphasized that effect size is a measure of practical significance that allows for the correct interpretation of results by removing the possible effects of sample size on research results.
This study calculated and reported the effect size using eta square ( 2 ) statistics.An effect size value of .01 was considered as small effect, while it is moderate for .06 and large for .14(Green & Salkind, 2014, p.158).

Pre-test and Post-test Comparison (Achievement)
The achievement pre-test scores of the experimental and control groups were analysed using an independent group t-test, and the results are shown in Table 4.The EEAT post-test scores of the groups were analyzed using an independent group t-test to determine the effect of the intervention on the achievement variable.The results are shown in Table 5.

Pre-test and Post-test Comparison (Motivation)
The motivation pre-test scores of the experimental and control groups were analysed using an independent group t-test, and the results are shown in Table 6.The SMTSL post-test scores of the groups were analysed using an independent group t-test to determine the effect of the intervention on the motivation variable, and the results are shown in Table 7.The results of the independent group t-test revealed no significant difference between the experimental and control groups' post-test scores on the SMTSL [t(28) =1.15,  >.05].After the intervention, the mean SMTSL score of the experimental group was Mexp=130.20,and that of the control group was Mcon=122.80.The average motivation score of the experimental group remained at the same level, but that of the control group decreased.

Discussion and Conclusion
Design and scientific inquiry are routinely used to solve real-life problems from an engineering perspective (Sanders, 2009, p. 21).In this study, a design-oriented learning environment was created to solve the problems within the framework of STEM disciplines.In this context, the effect of a STEM-based learning environment on achievement and motivation in science education was investigated.
As stated in other sections, the Turkish science curriculum adopted an inquiry-based approach from an interdisciplinary perspective.There is also an emphasis on engineering and design skills in the curriculum.Although the science curriculum supports the STEM education approach, teachers seem to lack knowledge about its implementation, and some problems arise in practice (Özcan & Koştur, 2018).Similarly, Karakaya and Yılmaz (2022) underlined the difficulties of science teachers in designing STEM-based science learning environments.For example, studies reported that teachers had expressed problems such as time constraints, curriculum density, inadequacy in engineering knowledge, low cooperation among teachers, and lack of materials for STEM implementation (e.g., Bozan & Anagün, 2019;Eroğlu & Bektaş, 2016;Özbilen, 2018;Özcan & Koştur, 2018).In addition, in some implementations, a concern arises that STEM activities are 'cookbook' type workshops that lack STEM disciplines and skills, and the interdisciplinary nature of the STEM approach is not fully understood.Okulu and Oguz-Unver (2021) emphasized that STEM education should include inquiry and engineering experiences, but the design should not be limited to cookbook-type activities in which students are guided through each step.Some studies have reported that teachers do not integrate STEM activities into their lessons and associate STEM only with student-centred activities, laboratory materials, and technological devices, ignoring interdisciplinary connections (Özcan & Koştur, 2018;Timur & İnançlı, 2018).Therefore, comparing a curriculum-based learning environment with a STEM-based learning environment makes sense.
The research findings revealed a significant difference between the experimental and control groups' mean achievement scores in the post-test.Accordingly, these results indicated that the STEM-based learning environment is more effective than the curriculum-based learning environment in improving student achievement.This result is compatible with literature showing that STEM learning environments positively effect academic achievement and science learning (Akkaya, 2019;Angwal et al., 2019;Anwar et al., 2022;Büyükbastırmacı, 2019;Cotabish et al., 2013;Çalışıcı, 2018;Çetin, 2019;Dedetürk et al., 2020;Doğan, 2019;Fan & Yu, 2017;Gazibeyoğlu, 2018;Green, 2012;Gülen, 2016;Kapan, 2019;Koca, 2018;Yıldırım, 2016).Angwal et al. (2019) reported that the STEM learning environment, which includes the engineering design process, is more effective than the traditional learning environment in improving students' science achievement.Cotabish et al. (2013) revealed that a STEM intervention statistically increased elementary school students' science content knowledge and science process skills compared to students in the comparison group.Similarly, Wendell and Rogers (2013) reported that elementary school students' science content knowledge increase in the engineering design-based learning environment.As stated earlier, the engineering design process can help students confront their understanding and misunderstanding of science concepts (Capobianco et al., 2018).Therefore, the STEM-based learning environment designed in this study might have contributed to the students' deepening of their knowledge and, thus, their success due to engagement in engineering-based processes.However, unlike the findings of this study, some studies reported that STEM-based learning environments do not significantly affect the achievement variable (Bircan, 2019;Dumanoğlu, 2018;Hiğde & Aktamış, 2022;Nağaç, 2018;Neccar, 2019).Nağaç (2018) reported no statistical difference in achievement between STEM and curriculumbased learning environments at the 6th-grade level.In addition, Nağaç (2018) stated that most participants did not answer the reason part of the two-tier test and underlined that this might be why the difference in achievement did not occur.Similarly, Neccar (2019) reported that the STEM learning environment did not make a statistical difference in achievement and underlined that the multiple-choice test used in her study might have caused this.Hiğde and Aktamış (2022) also reported that science achievement did not differ significantly between groups in their experimental study based on STEM.Similarly, a multiple-choice test was used to measure achievement in their study.Dumanoğlu (2018) reported that the STEM learning environment designed in the 7th-grade electrical energy unit did not make a statistical difference in the achievement variable compared to the learning environment designed based on the curriculum.In the same study, the data collection tool, a multiple-choice test, was used to measure the effect on the achievement variable, unlike our study.This may raise the question of whether a measurement tool affects the achievement findings.
The affective domain is important in science education because affective factors such as motivation and attitude correlate with students' science achievement (Cavas, 2011).Furthermore, students' motivation to learn science is a multidimensional construct affected by many factors, such as learning environments, curriculum, and teaching methods (Yılmaz & Huyugüzel-Çavaş, 2007).For example, Brophy (2010) stated that students' motivation is affected by their interactions with teachers and classmates, so a classroom climate to support collaborative learning supports motivation to learn.However, there is no statistically significant difference in motivation scores between the groups in this study.Similarly, Karcı (2018) and Yıldırım (2016) reported no significant difference in motivation between the groups in their experimental STEM studies.Julià and Antolí (2019) also reported that students' motivation changed slightly in their research examining the effect of a long-term STEM-based learning environment on 6th and 7th-grade class students' motivation.The authors underlined that the students' motivation was maintained throughout the intervention.De Loof et al. ( 2022) reported that students' motivation in integrated STEM-based learning environments was positive but lower than expected.According to the researchers, students might have faced internal and external pressure to perform well in their first experience in STEM-based learning environments.Karcı (2018) noted that students' motivation sometimes decreased in STEM activities requiring group work and focus.Büyükbastırmacı (2019), in an experimental study with 7th-grade students, stated that students' motivation to learn science in the STEM learning environment did not reveal a significant difference compared to the control group.In addition, it was underlined that conducting the study for five weeks was insufficient and could cause this situation.Ulusoy (2019) emphasized that asking students to fill out many papers, such as worksheets for research purposes in the learning environment, may negatively affect students' motivation.The points emphasized by Büyükbastırmacı (2019), Karcı (2018), and Ulusoy (2019) may be one reason why the initial motivation scores did not increase further, considering the similar processes experienced in this study.
Moreover, the high initial motivation scores of both groups in this study may explain why there was no significant difference.After the experimental process, the average motivation score of the experimental group remained at the same level, but that of the control group decreased slightly.Although there was a decrease in students' science learning motivation in the control group, there was no statistical difference between the groups.This study's findings indicated that students' motivation was maintained throughout the intervention.In this context, it can be asserted that the STEM learning environment does not negatively affect students' science learning motivation.Unlike the results of this study, Lin and Tsai (2021) reported that students' science motivation in a STEM-based learning environment significantly differs from students in a traditional learning environment.However, compatible with the findings of this study, the authors noted that students' science learning motivation declined in traditional instruction.Some studies have reported that STEM learning environments increase motivation toward science (Hiğde & Aktamış, 2022;Kapan, 2019;Lin & Tsai, 2021;Ngo, 2021).For example, Kapan (2019) reported a significant increase in students' motivations toward science in STEM research conducted with a one-group pretest-posttest design in the 7th-grade electrical energy unit.Similarly, Green (2012) also reported that students' motivation increased in the STEM study on engineering design.
The sample size of this experimental study can be considered as a limitation.However, a large sample in an experimental study focusing on STEM-based learning environments may cause some problems regarding the implementation of the research.Although the sample size was small, evaluating the current study's and other STEM education studies' findings together could contribute to our understanding of STEM education.Therefore, meta-analysis studies can add depth to our understanding of STEM education.Another limitation of this study is that the intervention period was limited to six weeks and a science unit.Therefore, future long-term studies, including qualitative data, may contribute to better understanding effects of STEM education.

Table 1
Table 2 contains information on the STEM activities.

Table 3
Shapiro-Wilk test results of the EEAT and SMTSL

Table 4
Comparison of the EEAT pre-test means between the experimental and control groupsAnalysis of the EEAT pre-test data before the intervention revealed no significant difference between the achievement scores of the experimental and control groups [t(28) =1.23,  >.05].

Table 5
Comparison of the EEAT post-test means between the experimental and control groupsAfter the intervention, the mean EEAT score of the experimental group was Mexp=46.73, and that of the control group was Mcon=31.60.The independent-group t-test results revealed a significant difference between the EEAT post-test scores [t(28) =2.28,  <.05].

Table 6
Comparison of the SMTSL pre-test means between the experimental and control groupsAnalysis of the SMTSL pre-test before the intervention revealed no significant difference between the motivation scores of the experimental and control groups for science learning [t(28) =−0.134,  >.05].Before the intervention, the mean SMTSL score of the experimental group was Mexp=130.53,and that of the control group was Mcon=131.26.

Table 7
Comparison of the SMTSL post-test means between the experimental and control groups