Development, application, and evaluation of an online competitive simulation game for teaching electricity markets

The decline of fresh‐enrolled students and the increase in the number of dropouts in electrical engineering schools might be related to motivation, engagement and differences in learning preferences. This paper provides a detailed description of the development, application, and evaluation of an online, asynchronous simulation game to teach electricity markets concepts named EMGA. The EMGA aims to introduce students to the short‐term electricity market structure, highlighting the importance of forecasting tools for decision‐making. The EMGA has been deployed in a Master's course with 27 students. The learning effectiveness was assessed with a survey at the end of the exercise. Questions aiming at experience generation, conceptual understanding, skills development and affective evaluation were enquired. Positive results towards experience generation, conceptual understanding and affective evaluation were obtained. Students felt optimistic about the platform's potential. The main complaints from students were their lack of programming experience and the allocated time for the exercise during the course. Results from this small test, along with previously obtained results, are encouraging and might be of potential use for further developing the EMGA and student's experience.

increase by 12.1%12.1% by 2025, and according to the analysis in [6], to match the needs carried with their expansion growth, regions like Africa and Latin America need to invest more in engineering. However, as stated in [27] and [31], the number of dropouts in engineering schools at higher education institutions have increased over the last two decades, joined by a decline in the number of enrollments.
Several studies, as highlighted in [26] or in [14], classify three main categories of potential barriers to retaining students in engineering programs as climate and belonging, grades and conceptual understanding, and pedagogy and learning preferences. The first category involves inadequate teaching and advising and a culture of competition over cooperation found in most engineering programs. The second category deals with factors stemming from high-school preparation, low college grades, time-consuming coursework and grading scores. The last category addresses factors such as students' low levels of self-efficacy and self-confidence and their relationship with the climate of engineering classrooms (competitive, individualistic). The barriers mentioned prevail particularly in electrical engineering programs, where traditional lectures are the rule, that is, students passively receive information from their instructor, curricula are rigorously founded on mathematics and physics, and classrooms are usually competitive and individualistic [36].
The concept of game-based learning (GBL) is quite broad, involving game content following specific objectives towards the enhancement of knowledge and skills, as stated in [33]. Although it cannot be said that it is sufficient to replace traditional instruction, GBL opens opportunities for learning to be experiential, dynamic, situated, problem-based, and that it allows for immediate feedback. Such characteristics are shared in modern theories of active, effective, and experiential learning, as in [35]. According to [9], components of GBL include, but are not limited to, digital simulation, video games, interactive tasks, toys, role-play and modelling. GBL is often addressed with gamification ingredients, for example, leaderboards, points, and badges, to provide an engaging atmosphere. Literature reports that GBL allows positive cognitive and affective outcomes while integrating digital games into teaching and learning [3]. Thus, an extensive number of applications use GBL concepts in educational contexts, from healthcare, as in [18] to teaching history in [40] and quantum mechanic concepts in [2]. However, although electronic components, electrical machines, and electrical appliances allow hands-on experiences, other theoretical subjects, such as electricity markets, might be more complex to emulate in a classroom environment. In this sense, electrical engineering programs may benefit from applying GBL.

| Literature review
The use of GBL applications has increased over the last three decades in the context of power systems in general, and more recently electricity markets in particular, as an alternative to teaching subjects with limited hands-on experiences. A lab experience aimed at teaching poolbased electricity markets, rewarding the students according to their acquired revenue was proposed in [8]. The authors claimed that blending decision-support tools with technical intuition stimulated students' creativity. A local graphic user interface (GUI) for teaching economic dispatch was proposed in [30] and considered environmental constraints. Users were able to choose dispatching schemes and fuel types to obtain different objectives. Similarly, the authors in [10] developed a simulation game for teaching short and long-term dynamics in electricity markets. It was shaped as a web-based game consisting of several competing power generation companies based on publicly available information.
A simulation tool for electric power markets was proposed in [39] as a web-based evaluation toolbox for different market designs. The paper shows a detailed description of the web architecture and applications. In [5], a day-ahead electricity market realistic simulator was introduced using a graphic user interface and was aimed at higher education and personnel training. A web-based power market simulator was introduced in [1] for dayahead or real-time market. A comparison of their application with other simulators was shown. According to the authors, the simulator's proposed architecture enables different market models, rules, and time frames, making it more flexible. In [24], the authors discussed the development of a web-based simulator for the spot market in South Korea. The clearing mechanism in South Korea is described, along with the implemented dispatch method and the simulator's structure. Two simulation games were presented in [11] for market simulation and electricity generation. Both games were used in laboratory sessions introducing trading strategies of electricity markets and power plant technical constraints.
Similarly, a web-based platform for wholesale electricity markets was developed in [37] as a strategy game. Players were able to create generation portfolios and compete in a daily electricity spot market. A GUI educational software package was presented in [12] aiming at teaching the forecasting methods and the day-ahead market structure. It should be noted that besides their main topic (electricity markets), all the works mentioned above share the lack of a learning efficacy analysis, which is crucial to assess the achievement of learning objectives.
Some studies have also performed a learning efficacy analysis to evaluate the performance of the proposed approaches. As reported in [38], students had to compose an energy portfolio with different energy sources. Moreover, the most profitable group was the winner based on a leaderboard. A post-questionnaire was applied to 12 students enquiring about the game's implementation as an alternative to traditional teaching, showing a positive perception. A computer-assisted method for teaching and understanding transmission pricing techniques was developed in [15] using a GUI. The GUI was used in a power system economics course focused on transmission cost allocation methods and tested with 185 students. Previous years' results from traditional evaluations at the National Technical University of Athens were used for comparison. Results showed that grades obtained after introducing the computer-assisted interactive learning approach increased. A simulation game for teaching electricity markets was proposed in [19] and tested using the wholesale Chilean electricity market. The test was performed with 14 groups at the University of Chile and the Pontifical Catholic University of Chile. The authors evaluated the expected competencies using a posttest which indicated positive results.

| Contributions
This paper presents a detailed description of the development, application, and evaluation of an online, asynchronous, simulation game to teach electricity markets concepts, named Electricity Markets Game (EMGA). The game platform has been developed by an electrical energy systems group of a university in the Netherlands, applied as a graded exercise in a Master's course, and evaluated using a survey at the end of the exercise. The main objectives of the EMGA platform are to introduce players/students to the short-term electricity market structure, specifically, the day-ahead market, and to highlight the importance of forecasting tools for decisionmaking. The hypothesis to be tested using the EMGA is if the subject of day-ahead pool trading for energy portfolios can be taught effectively using GBL as a complement to traditional lecture-based learning. The learning effectiveness has been assessed with a survey at the end of the exercise, including questions on experience generation, conceptual understanding, skills development, and affective evaluation. Compared to similar approaches, this paper's highlights are: • A detailed description of the development of a webbased, asynchronous simulation game focused on teaching electricity markets, including its back-end and front-end. • A detailed description of the application, and evaluation of the EMGA in a Master's course using a post-survey. • Obtained results are encouraging and might be of potential use for further developing the EMGA, and can be used for comparison purposes in similar applications.

| METHODOLOGY
The EMGA was conceptualised as a web-based, asynchronous, serious game, outlining a variety of electricity market structures, for example, market rules, commodities and instruments, market-clearing algorithms and so forth. The EMGA platform deems actual power system limitations to cope for a realistic implementation, that is, network conditions, uncertain sources such as renewable energy, unexpected network events and so forth. The game engine has been designed modularly, so it is versatile, extendable and functional to complement different electrical engineering courses related to electricity markets. The main characteristics of the EMGA platform are presented in Table 1.
The EMGA platform is based on the experiential learning theory proposed in [23]; hence, students are expected to reach specific learning goals within four main target components, namely, experience generation, conceptual understanding, skills development and affective evaluation, as listed in Table 2. Experience generation is tackled in the EMGA platform by allowing students to have hands-on practice with forecasting algorithms and decision-making techniques, experimenting with different algorithms and fine-tuning parameters. After playing the game, students are expected to recognize, calculate and estimate uncertainty sources in electricity markets, the dynamics of short-term electricity markets, and the concept of risk aversion. At the same time, as students have access to prefilled Python scripts, it is expected that they will be able to practice with different algorithms to analyze time series and use that information to formulate their own hypotheses. Moreover, successful participation in the EMGA platform is expected to be an additional incentive for students to be more engaged in the course's material, while the handson experience is expected to produce a sense of achievement after being exposed to a real-life problem in a safe environment. Finally, prefilled code is expected to help reduce possible coding anxiety for inexperienced students while allowing their effective interaction with the platform.

| DEVELOPMENT
The EMGA has been entirely developed in Python language using the Dash framework, described in [21]. Dash is a framework for building web analytic applications written on top of Flask, Plotly.js and React.js. The game was deployed using a Heroku container (see [28]) and linked to a designed PostgreSQL database to store players' crucial information. Since the game is based on HTML, CSS and JavaScript, the platform can be rendered in most commercial web browsers. Thus, players only need an internet connection to participate and they can do it asynchronously at their own pace. Moreover, a Jupyter notebook has been designed and delivered to students with relevant forecasting and decision-making algorithms, so the programming anxiety is reduced. However, it must be pointed out that the platform is completely independent of the Jupyter notebook; thus, users can come up with their own decision-making tools. The general schematic representation of the EMGA platform is depicted in Figure 1, where the data from the EMGA represents shared information, for example, historical data, revenue, leaderboard and so forth and the bids from players represent the file they must upload to trade.

| Front-end
The EMGA platform is conformed of five pages, namely, Login page, Portfolio page, Trading page, Results page and Final page. The layout of the pages was designed to be T A B L E 1 Characteristics of the electricity markets game platform Feature Electricity markets game Educational content Electricity market concepts, mechanisms and participation.
Purpose Complement of traditional lecture-based learning.
Game techniques Scores, performance graphs, and leaderboard.

Storytelling
• Each team assumes the role of an energy portfolio owner that participates in the day-ahead electricity market.
• The game features an environment with several unknown quantities (prices, available wind energy).
• Rules: use historical data to forecast unknown variables and develop a market participation strategy (bids). Any deviation from the bid to actual realization is settled using a balancing mechanism (typically penalized).
• Objective: To maximize the revenue by bidding during a number of days to obtain a top position on the leaderboard. A short description of each page can be found below: Login page: The login page is the entry page of the EMGA and requires user identification and authentication. The validation is performed by entering a registered username and password combination. The username is directly stored in the PostgreSQL database, while the password passes first through an MD5 encryption algorithm. The login page also allows registration for firsttime users, a section for forgotten credentials, a link to the game manual, and a link to download the login tutorial. The login page is shown in Figure 2a.
Portfolio page: This page allows players to determine their energy portfolio. The selection involves the energy sources to be administrated considering their main characteristics, that is, power capacity, capacity factor, dispatchability, variability, direct emissions, indirect emissions and construction cost. However, players need to choose carefully since their investment cost is limited to a fixed budget. Once the player clicks on the Continue button, the information is stored in the database. The portfolio selection page can be seen in Figure 2b.
Trading page: On the left of the trading page, the current team's performance (current position and accumulated revenue) is displayed in Figure 3a. The players can find the Leaderboard button below the performance information, where the teams ranking is displayed. Both the performance and the leaderboard data come from the most updated information of the database at the time of querying. Players have different information on the Toolbox section, such as their portfolio characteristics, a link to the game manual, a template of the bid file, as well as different links to video tutorials. Furthermore, there is a Help button where players can find the guidelines of the game. Over the same header, the players find the username and a logout button on the right-hand side.
The main part of the page is occupied by the current trading date and the buttons to download historical data such as wind power information, day-ahead prices, and imbalance factors. Downloadable data is in .csv format and stored in the cloud. A drag-and-drop or select file feature is available for players to upload their bid file and it is stored locally in the player's session. Finally, the page counts with a button to confirm the player's trade and a section with information messages. The information messages work as an indicator for players of pending tasks or to continue. For example, in Figure 3a, the warning message 'WARNING-Files pending to be downloaded' in yellow, indicates that at least one file has not yet been downloaded to calculate the bid file. Two other messages can be displayed, this is, 'WARNING-Upload Bid File' in red, and 'You are ready to click TRADE!' in green. The colours indicate that the user cannot continue (red), the user must be aware (yellow) and the user can continue (green).
Results page: The bid file stored locally on the Trading page is used to calculate the player's imbalance and revenue compared to the actual realization of the day. The results page displays four figures, as shown in Figure 3b, identified as Day-ahead information, Imbalance & prices, Revenue per period and Day accumulated revenue. The calculations of each plot are described in detail in Section 3.2. A clickable link is available for players to download the current results in .csv format. Once players have finished analyzing the data, they must click on the 'Go to next day's bid' button on the top-left side of the page to continue playing.
Final page: Once a player finishes the available playing days, they are immediately redirected to the final page. In this instance, users can see their final accumulated revenue, their current position in the leaderboard, and the logout button. Being on this page represents the end of the game for a particular player.

| Back-end
Once a player has selected its portfolio, they need to consider the operational limitations for each source. The operational limits of each player's portfolio are stored in the database and can be downloaded as a .csv file on the trading page. Students are guided to consider the operational limits in their decision-making algorithm and obtain feasible bids, that is, bids that satisfy all operational constraints. These limits are: • Ramp-up and ramp-down, identified as E ramp and E ramp , respectively. • Minimum and maximum power, identified as E and E, respectively. • Minimum and maximum state of charge (SOC), identified as SOC and SOC, respectively.
A pre-processing step consisting of solving an optimization problem checks the feasibility of submitted bids and modifies the periods violating any of the constraints considered in Table 3. These possible modifications result in a reduction in the total perceived revenue. Jupyter notebook client: The prefilled code included some techniques to help students confirm their forecasting algorithm, for example, linear, ridge, epsilonsupport vector and random forest regression, all of them from the Scikit-learn library (see [32]). Students also had a simple artificial neural network example with ten neurons using the Keras library (see [7]). Besides the forecasting method, students were available to experiment by changing different important hyperparameters and settings to stimulate experimentation and creativity. Some of these options include the feature selection method, the size of the training and test datasets, the number of scenarios generated, the number of neurons, and the neurons' activation function. For instance, the default code was a recursive forecasting strategy; however, students were repetitively encouraged during the lecture to try different configurations (e.g., multi-output Thermal unit regression) or to implement other available techniques on their own, as the used libraries are broad and well documented on the internet. The default decision-making technique was a deterministic optimization model implemented in the Python-based, open-source modelling language Pyomo (see [20]). The model was cast as a mixed-integer linear programming problem aimed at maximizing the total revenue f . The solution of the optimization problem obtains the optimal decisions, that is, binary variables y such as the operating states of the thermal unit (on/off) and the energy storage system (charging/discharging); and continuous variables X such as the power outputs from the thermal unit and the energy storage system, and its state of charge. All this, considering the operational limits defined by each player's portfolio (L U , ). A general representation of the problem is shown in Equation (1).
where h X y ( , ) represents the equality constraints such as the state of charge calculation, while g X y ( , ) stands for the inequality constraints like maximum and minimum power injections. Students were recurrently encouraged to experiment with different decision-making techniques considering uncertainty discussed in the lecture, such as stochastic, robust, or risk-aware optimization.
Expected accumulated day revenue: The game evolves within a fixed number of trading days PG Ω = {1, 2, …, # } D and the trading period of each trading day is discretized in 1-h intervals Ω = {1, 2, …, 24} T . Players' main motivation as Energy Portfolio administrators is to participate in the day-ahead electricity market and maximize their revenue. Players have a pricetaker position in the market; thus, their main task is to submit 1-h energy blocks ∈ t Ω T for the next trading day ∈ d Ω D , namely, energy bids E t d , bid , considering day-ahead The expected accumulated daily revenue a player will obtain during a trading day d R , d exp , is the summation of the product between the player's hourly energy quantities and the corresponding day-ahead prices, DAP t d , and is calculated as: Although bids are considered as blocks of energy, it is required that participants upload their energy bids separately for matters of reproducibility and tractability.
represents the energy bid per source of the portfolio, that is, Ω = {Th,Wt,St} S . Pre-processing optimization step: A pre-processing step consisting of solving an optimization problem verifies the feasibility of the submitted bids and modifies all periods that are not aligned with the constraints. The optimization problem has been modelled using the Pyomo package and solved with Gurobi (see [25]). For each trading day d, the optimization model reads as in (4)- (13).
Imbalance or power mismatch: Depending on the characteristics of each player's portfolio, the actual energy production might not be as expected, for example, wind power may differ from forecasted values. In the EMGA, it has been assumed that the operation of controllable sources (thermal plants and energy storage) is perfectly deterministic. For a trading day d, the difference between the actual energy realization and the submitted bids is defined as Imbalance, E Δ t , as: and χ t the total constraints violations. Any slack variable whose value is different than 0 represents an infeasible bid; hence, a violation of the operational constraints. The bids with violation will be modified to the closest feasible operational limit and the values of the slack GIRALDO ET AL.
| 765 variables will be considered with imbalance market prices, resulting in revenue inefficiency. Market rules: The market rules implemented in the EMGA platform are simple and depend on the day ahead prices and the energy imbalance, as: Notice that E Δ t d , is the energy mismatch and it is positive if the player's portfolio is producing more energy than submitted in the bid file (surplus) or vice versa (deficit). The values for ≥ λ λ , 0 , are proportional to the day-ahead price, always respecting that . Flexibility redispatch: The imbalance cost might be decreased by considering the flexibility of the controllable sources. Thermal plants are assumed to be flexible, i.e., they are able to inject a different amount of energy than they were initially dispatched in the energy bid within a predefined range. This amount of used flexibility is represented by F t , and it is limited by a symmetric parameter proportional to the energy bid, that is, However, there is an operational cost since the units must deviate from their optimal operation to compensate for the imbalance. Hence, the EMGA platform solves internally an optimization problem at each time period, always selecting the best option for the player depending on the sign of Hence, the actual imbalance can be expressed as , , and the operational cost for flexibility is denoted by c t . Finally, the infeasibility penalty is computed as the total constraints violations times the imbalance price, that is, pen λ χ = t t t . Furthermore, it can be seen that the perfect forecast revenue is ⋅ Actual accumulated day revenue: The actual accumulated day revenue is expressed as: Notice that these market rules encourage the players to be as accurate as possible with their forecasts, that is, Leaderboard: There are three main components of the game to be aware of: Accumulated revenue or just Revenue, the number of played days or Days, and the Rate. Consider D Ω as the set of days a participant has played so far; hence, D ⊂ Ω Ω D . The Revenue is the sum of the accumulated day revenue during the days a participant has played so far. The Days represent the number of trading days a participant has played so far. However, the Rate is a measure of the performance of each player representing the quotient between the Revenue and the number of played Days, as:

| A player in the EMGA platform
The general flowchart of a player on the EMGA platform is shown in Figure 4.

| APPLICATION
The competitive simulation game was used in parallel to the module 'Data Driven Modelling' within the Master's course '5LED0-Smart Grid Operation Through ICT' at TU/e. Students' age range was between 21 and 27 years old, coming from different majors like Electrical Engineering, Mechanical Engineering, and Sustainable Innovation. Some of the course's goals related to electricity markets are: • Identify the main electricity markets and participants.
• Understand the wholesale markets of electricity and the (time/volume/price) dependencies between them. • Understand how different types of participants (producers, energy suppliers, traders) trade on the electricity wholesale markets on different time scales (futures, dayahead, intra-day, balancing, reserves). • Apply the above knowledge to specific real-world cases.
Thus, the EMGA platform is aligned in contributing positively towards achieving these goals. Some basic concepts on electricity markets, such as market dynamics and market structures, were introduced through traditional lectures. Well-known forecasting and decision-making techniques were also presented to the players, totalizing three hours of lecture to establish a common essential knowledge background. A Jupyter notebook (see [22]), was handed to the players with a prefilled code corresponding to the techniques introduced during the theoretical lecture (forecasting and decision-making) and recommendations about alternative algorithms to experiment with. Furthermore, a compendium of four short video tutorials (~4 min) was accessible through the platform regarding basic information on how to log in, download data and trade, read the output information, and use the client software.
The hands-on exercise was proposed in the middle of the module, so students had enough time to experiment  with the platform and provide the requested deliverable. Students were introduced to the game instructions and the grading scheme of the experience, as explained in the following subsections. One additional hour was dedicated to a Questions & Answers (Q&A) section on the functionalities of the EMGA and the Jupyter notebook.

| Game instructions
Each team is provided with a kit containing a username of their preference, a randomized six-digit alphanumeric password, and the Jupyter notebook client. Participants are then guided to access the platform at [16], watch the tutorial video, read the game manual, and log in using the login page in Figure 2a. If it is the first time the team logs in, then they would face the Portfolio selection page in Figure 2b, where they must decide the best energy mix of their portfolio considering particular characteristics. The portfolio can be conformed by energy from thermal plants, wind farms, and energy storage, with different unitary costs and some qualitative characteristics, such as variability, capacity factor, and emissions. The desired portfolio must be within a predefined budget range (98%-100%) to continue.
Once the team has defined its desired portfolio, they can continue to the Trading page. The trading page is the main component of the platform since it allows teams to download the most current historical information for uncertain parameters, that is, wind power and energy prices. It also provides an information section with the team's performance, a toolbox section, and a section to upload the decided bids.
Students were instructed to follow the steps below: 1. (EMGA) Register in the platform and access using your username and password.
-If it is your first time playing, you will be redirected to select your energy portfolio. After you have selected your portfolio, you will be redirected to step 2. -If it is not your first time, you will be immediately redirected to step 2. 2. (EMGA) In the main page, you will be able to download the updated historical wind data and the day-ahead prices (updated for each trading day). Also, you will be able to download the operational limits of your portfolio.

| Grading
The grade of this group assignment contributed to the course grade by 10% and was based both on the group game performance (GPG) and a submitted report (R). The final grade of the exercise is Final GPG R = 0.5( + ).

Game performance:
The game performance is evaluated in terms of 1) the position of each group in the leaderboard and, 2) the number of rounds (trading days) in which each group has played. The first component of the game performance part of the grade is G 1 and depends on the total number of groups playing ( Groups # ) and the position of each group in the Leaderboard, such as ∈ Pos Groups [1, # ]: Notice that ≤ ≤ G 5.0 10.0 1 . The total grade depends on the game performance and is identified as GPG. GPG depends on the total number of trading days (rounds) available for trading ( PG # ), the actual number of rounds currently played by each team ( ∈ Days PG [0, # ]), as well as the grade component G 1 . GPG is calculated as follows: It can be seen that the grading strategy penalizes any available rounds not executed by the team. Thus, the grading strategy incentives both to try to improve the group's position in the leaderboard and to execute as many trades as possible. 2. Submitted report: Students were asked for a team report with no page requirements (2-3 pages suggested) in which the most important experimentation efforts and creative ideas are discussed. Furthermore, all the code files necessary to support the discussions included in the report were also required to have a fully functional client. The report was assessed in terms of its readability, the team's demonstration of creative elements, and experimentation efforts. The team's choices were required to be backed by sufficient explanations and/or literature references. The grade of the reporting component was ≤ ≤ R 0.0 10.0. Note that the grade of the report was independent of the team's performance in the game.

| EVALUATION
A survey was applied at the end of the experience. It should be clarified that the survey was anonymous, and students' final grade of the exercise was conditional to its filling. The survey was divided into two main sections, one focused on the target components of the platform, that is, affective evaluation (Q x 1 ), conceptual understanding (Q x 2 ), skills development (Q x 3 ) and experience generation (Q x 4 ). The other section comprehends additional questions of interest for further development of the game (Q x 5 and Q x 6 ), where x denotes the question number. Most questions were evaluated using a 5-points Likert scale, so students can relate their level of agreement to each statement in five points: (1) Strongly disagree to (5) Strongly agree. As a remark, only question Q 61 a polar question. The full survey can be found in Appendix A.

| RESULTS
The 27 students enrolled in the course completed the hands-on exercise and were asked to answer a short survey regarding their experience with the EMGA platform. Let a be a real-valued random variable, that is, results from the Likert scale of one question. The mean (μ a ) and standard deviation (σ a ) obtained for each question was calculated as: where ⋅ [ ] represents the expectation. The mean and standard deviation obtained for each question are depicted in Figure 5a. Within the target components' questions, it can be seen that the lowest mean rating was 2.3 for question Q 34 , corresponding to the skills development evaluation. Specifically, this question enquired students if they were able to try different decisionmaking techniques in the game. The low rating is coherent with some students' comments about lacking programming experience as they were required to experiment with a prefilled Python script in a Jupyter notebook. Conversely, the highest rating was obtained for the affective evaluation in question Q 11 with 3.9, when students were asked about their preference between a classical lecture or a dynamic one (as the one deployed in the current exercise.) In general terms, it can be seen that the mean rating obtained was higher than 3.0 in all tested components, as can be seen in Figure 5b. It can also be noticed that the median rating was 3.0 for all the components but for conceptual understanding, where the median was 4.0. These results indicate that at least 50% of the students had a positive perception (greater than or equal to 3.0) of the tested components, with conceptual understanding being the highest valued component, while skills development being the lowest.
The Pearson correlation coefficient (PCC) between two obtained answers has been calculated as in [4]. The PCC between a and b is defined as: . The PCC of the obtained answers is shown in Figure 6. The correlation among all answers is presented in the centre of the figure, while subfigures identified with 1-5 represent the different question divisions, for example, 1 embraces questions for affective evaluation, 2 for conceptual understanding and so forth.
As expected, the diagonal presents the highest value as it is the correlation of each question to itself.
Identifying high correlation is the first step to forming a hypothesis. Notice that most answers to questions in group 2 (Q 21 -Q 27 ) show a high correlation. For instance, it can be said that students that found the platform helpful to understand the importance of uncertainty might have also understood the process of submitting and profiting in the electricity market, as can be inferred from the correlation of ∕ Q Q 21 23 which is approximately 21 23 . An example of positive correlation between two questions is shown in Figure 7, regarding the experience generation allowed by using the platform and creativity motivation ( ). It can be seen that most students (92.5%) found that the platform allowed them to experiment with situations that would not be possible in real trading scenarios, and at the same time, that the platform is open for experimentation and motivates creativity (81.5%).
Notice that an analysis of the opposite scenario is also possible, in this case, if the correlation is negative, it is an indicator of an inverse relationship. Take for example the affective evaluation regarding the frustration of math/programming expectations and the reduction of pressure for obtaining a result ( 13 17 ). Particularly, these results indicate that most students that felt frustrated due to math/programming expectations did not feel a reduction of the pressure to obtain a result by using the platform. An example of negative correlation is shown in Figure 8 ). It can be seen that most students (78%) found that the workload was spread equitably, while only 15% felt like they had invested more in the game assignment compared to their teammates.
The answers to the questions of each target component were merged to obtain a general representation of the students' perceptions. The number of times the students rated a specific value in the Likert scale is shown in Figure 9 for each tested component. It can be seen that results for experience generation and conceptual understanding present negative skewness, with more than 70% of the rating equal to or higher than 3.0. The affective evaluation component presents a more uniform pattern; however, it can be seen that more than 67% of the answers were equal to or higher than 3.0. Finally, the skills generation component presents a bimodal distribution with one mean at 2.0 (24.4%) and another one at 4.0 (31.1%). These results indicate that the feeling of students was in general positive in three of the components, but it was not defined regarding the skills generation component.
However, when asked about future game implementations, students were asked about their perception of the platform when developed further in Q 59 . Results in Figure 10a show that almost 90% of the students consider that the students can learn more from using the platform when it is developed further. Additional questions were asked to students to contextualize the obtained answers. Students were asked about their general perception of the platform in Q 53 . The results to that specific question are shown in Figure 11a, where it can be seen that the general perception of the platform is positive, with more than 50% of the students rating it with at least 4.0, while less than 30% rated it with 2.0 or less. The perception of the prefilled Jupyter notebook was also enquired to students in Q 54 . Obtained results to that question are depicted in Figure 11b, where it can be seen that over 70% of the students found it as a helpful resource.
An important factor aimed at the EMGA platform is the teamwork among the group members. Students were asked in Q 56 about the spreading of the workload over the team members. The results to that question are shown in Figure 8a, where it can be seen that more than 50% of the students think the work was evenly distributed, whereas almost 23% of the students felt that it was not evenly spread, rating that question with 2.0 or lower. This result is corroborated by Q 56 , where at least 40% of the students do not have a strong belief that they have invested more than their teammates while playing the game.
Students were also asked about the possible inclusion of gamification objects to increase their motivation and engagement, such as badges or game rewards in Q 58 . Results in Figure 12a indicate that including such gamification objects would not be particularly an improvement for at least 48% of the students, whereas it could be something positive for 51%. Although the sample size is not representative, this uniform distribution is recurrent in literature considering students' diversity. Finally, students were asked if they would participate in the game if it was not part of a graded activity in Q 60 , for which 66.7% answered negatively, as can be seen in Figure 12b. According to the feedback from students, these results are related to the mathematical/programming expectations involving the simulation game and the tools required to play it.

| DISCUSSIONS AND LIMITATIONS
One of the main limitations of the current research is the number of students. It must be pointed out that the Master's course where the EMGA is being tested has a yearly frequency with a mean of 25 students. However, even though the sample size is not large enough yet to validate the principal hypothesis, 'the subject of dayahead pool trading for energy portfolios can be taught effectively using game-based learning as a complement to traditional lecture-based learning', it must be pointed out that the results from this small test, along with previously obtained results in [17], are encouraging and might be of potential use for further developing the EMGA. When compared to similar approaches in the literature, it can be seen that the obtained results in this paper are aligned with previous findings regarding knowledge transfer and students' acceptance. For instance, in [38], where authors also used a small sample, it was found that most students found the game as a good way of understanding the basics of electric economics. Moreover, when compared to [19], it was found that teamwork can be exploited using GBL in the context of electricity markets and economics, which could also be seen in the obtained results in this paper. When compared to [8], it can be seen that students' interest, motivation, and creativity was also stimulated by using EMGA. In fact, after the experience, two students selected electricity markets as the topic for their graduation project, one of which continued developing the market structure which will be implemented in a future version of the EMGA. Finally, notice that the percentage of students that answered that they would participate in a nongraded experience (~33%) is coherent with the actual number of students participating in [17]. The aforementioned evidence strengthens the motivation to further the development of the EMGA platform.
It must be pointed out that attempting to compare the obtained results with previous research in the field of electricity markets is not straightforward. The difficulty stems from the lack of reported assessment methods since most papers were focused primarily on the technical implementation of the simulation tools rather than quantifying their educational value. Hence, a recommendation for future researchers would be to implement and report the assessment methods.
Some technical limitations were identified regarding the platform's user interoperability, such as file downloading and loading times. These issues can be solved with further development of the user interface. From the students' feedback, we could also identify that the main complaints were the lack of programming experience and the allocated time for the exercise during the course. Some students also suggested a more cooperative framework, where teams could collaborate sharing ideas or procedures. Future work will be focused on improving the user's game experience, such as developing a modular interface to eliminate client software and introducing the interaction between teams via the EMGA platform (e.g., allowing trading forecast information and bilateral contracts). Allocating time to explain programming basic knowledge could also be an important factor to consider in future experiences.

| CONCLUSIONS
This paper presented the development, application, and evaluation of an asynchronous-competitive simulation game for teaching electricity markets. A brief description of electricity markets was presented for contextualization. Furthermore, the front-end and back-end of the game were described in detail, including the market structure and balancing mechanism, as well as the asynchronous characteristics of the game. The simulation game was applied in a Master's course with 27 students as a complementary tool to traditional lectures. Four target components were assessed, that is, experience generation, conceptual understanding, skills generation, and affective evaluation, using a 5-points Likert scale survey. Obtained results were positive, with a general perception of the platform over 3.0 from more than 70% of the students. It was found that using the EMGA as a complement to a traditional lecture was effective to teach electricity markets in terms of experience generation (72%), conceptual understanding (72%) and affective evaluation (68%). Results for skills generation were not conclusive since 43% of the students had a positive perception, against 39.2% who did not. According to the students' feedback, this perception might be related to the lack of programming experience of some students, the time reserved for the experience, and the required workload. However, students felt optimistic about the platform's potential use when developed further (96%), and most of them think the EMGA should be part of next year's course (89%). Two interesting results were found about students' motivation and engagement. First, that using gamification tools such as badges and rewards might not be an effective direct incentive. Second, that most students (67%) would only play the game within a graded experience in a particular course. These final results might be related to the mathematical/programming expectations involving the simulation game.
Future development of the EMGA will be focused on obtaining a modular user-friendly student interface containing the necessary utilities to test and produce the bid files under the hood without the need for the Jupyter notebook. Furthermore, a new version of the EMGA will introduce the teams' interaction via the EMGA platform, for example, allowing trading information and bilateral contracts.
Postdoctoral Fellow with the Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands. He is currently a Researcher with the Mathematics of Operations Research Group, University of Twente, Enschede, The Netherlands. His current research interests include the optimization, planning, and control of electrical power systems, electricity markets, and machine learning applications to power systems.
Koen Kok received the B.Sc. degrees in electrical engineering and in technical informatics, the M.Sc. degree in computer science from the University of Groningen, Groningen, The Netherlands, and the Ph.D. degree in computer science from the Vrije Universiteit Amsterdam, Amsterdam, The Netherlands, for his thesis on smart grid coordination mechanisms based on distributed software technology. He is currently a Full Professor with the Electrical Energy Systems Group, Eindhoven University of Technology, Eindhoven, The Netherlands. Formerly, he was with TNO, the Energy Research Center of the Netherlands, VU University Amsterdam, and the Technical University of Denmark. He has extensive research experience in the fields of market-based control of power systems, smart-grid ICT architectures, and integration of distributed energy resources and demand response in the electricity system. The key results of his research have been field deployed, commercialized, and/or made available in open-source.