Keywords

1 Introduction

Computing education is widely known that necessity and importance. It is believed to leads gain Computational Thinking [20, 23]. In recent years, also, many countries include subjects in elementary schools which related to computing education. In particular, it seems that programming education is the core of those.

Meanwhile, it becomes revealed that there are many difficulties in these programming education. In programming education, it is basic difficulties that understanding concepts such as variables, loops, recursions, arrays, and so on [17]. Moreover, there are advanced difficulties that achieving complex activities such as tracing, reading, writing, abstraction, and decomposition [7, 14, 15].

In research fields of computing education, many studies engaged in reducing these basic and advanced difficulties by tools or educational systems [16, 19]. Notably, educational systems that able to adaptively pedagogical behavior and feedback by artificial intelligence technology is called Intelligent Tutoring Systems (ITSs) [18, 22]. ITSs aims to supply adaptive tutoring to learner’s profile by alternate to human tutors. And, ITSs which focusing on the domain of programming is also known as Intelligent Programming Tutors (IPTs) [3]. IPTs helps novice’s overcome not only basic difficulties but also advanced difficulties.

Despite there are much supports to programming education by IPTs, there no attempts for organizing knowledge in programming. From this perspective, we aim to reveal an effective way to supports knowledge organization.

The purpose of knowledge organization is to produce well-organized knowledge. Well-organized knowledge is widely known as chunks in fields of cognitive psychology [2]. Also, it was indicated that chunks help to solve problems in programming [1]. Because, well-organized knowledge, chunks, is able to reuse existing knowledge to the same or similar problems that solved once by scaling knowledge. That is, organizing knowledge leads to acquiring systematized knowledge and its scalability. Therefore, we considered that organizing knowledge is fosters problem-solving skills, and thus it gains Computational Thinking eventually.

In this paper, we proposed and developed Compogram: an ITS for organizing knowledge by visualizing behavior in programming. Furthermore, for identifying learning gains, we conducted an evaluation compared to our conventional systems. Results were suggested that Compogram was fostering knowledge organizing skills that can apply to out of learning ranges.

2 Objective and Research Questions

To solve problems, it is important that learners understand component elements of problems, and acquire it as knowledge. However, there is a problem that difficulties to adequately reuse the same or similar solutions which have already been used by learners. For solving that, one of the ways is to organize the existing knowledge. That is, by organizing the existing knowledge, it is expected that existing knowledge can reuse the same or similar solutions by scaling knowledge as adapting to the grain size of problems.

The knowledge in programming involves many concepts such as grammar of specific languages and the object-oriented approach. Thus, it is necessary that select knowledge types for organizing accordingly the objective. In this research, we focused on supporting the process of solving problems by combining a bit of program. Then we selected the knowledge of the “function” and the “source program that achieves function” as knowledge to be organized. And, we defined a pair of knowledge as a component.

Therefore, we addressed the following research questions:

  • RQ1 How to design/realize learning tasks/activities for bonding the “function” and the “source program that achieves function”, acquiring as a component, and organizing as the hierarchy of components?

  • RQ2 Does provide such designed learning tasks/activities lead to learning gains?

  • RQ3 How to design/realize effective feedbacks/interventions in such learning tasks/activities?

For this objective, we have been engaged in support of systematizing and organizing learners’ existing knowledge as a component and have aimed at improving reusability [10,11,12].

First, we proposed a knowledge organization method called BROCs (Building method that Realizes Organizing Components) [10, 11]. BROCs support to understand the relationship between components by presents problems stepwise and bottom-up. Next, we developed and evaluated a system that realized BROCs . We addressed RQ1 by designing BROCs as a learning method and developing a system of BROCs. Meanwhile, we addressed RQ2 by conducting comparison evaluations between the BROCs system with and without feedbacks, and textbook.

In [12], we addressed RQ3 by proposing a model of the problem-solving process in BROCs. Through discussion based on the model, it was pointed out that there is the “behavior” in the bonding process between the “function” and “source program that achieves function”. Also, we pointed out that insufficient supports for understanding the “behavior”. In the conventional system as stated above, however, the feedback only presented the excess or deficiency compared to the correct answer when learners mistake. Therefore, we conducted an evaluation to investigate the effectiveness of feedback included it related to the behavior, that produced by the constructed problem-solving process model. Results indicated the explanation based on the model seems to deepen the understanding was obtained.

In this paper, we developed a system that realizes support for organizing knowledge by visualizing program behavior, it called Compogram. Accordingly, we evaluated the system and addressed the following research question:

  • RQ4 Does provide such designed feedbacks/interventions lead to more learning gains?

3 System Design

3.1 Theory

BROCs: Acquiring Components Method for Knowledge Organization. In our research, as described in Sect. 2, we focused on acquiring the knowledge of “function” and “source program that achieves function”, and defined a pair of knowledge as a component.

BROCs (Building method that Realizes Organizing Components) is a method of knowledge organization in programming that our proposed. In BROCs (Fig. 1), presenting tasks to learners according to follows two steps: (1) make small functions (such as swap) by combining primitive components (e.g.. assignment, if statement), (2) make larger functions (such as sort) by combining components that include already made it.

Fig. 1.
figure 1

An example of BROCs.

Full size image

By repeat these steps, learners can structuralize and accumulate components stepwise. In fact, some studies claimed a learning approach is effective that makes small functions firstly, and then, combines these small functions to make larger functions [8, 21]. Therefore, BROCs support learners’ knowledge organization through repeat stepwise presenting these tasks. Due to this, BROCs situate as an important pedagogy in the proposed ITS.

Function, Behavior, and Structure in Problem-Solving Process. We have been explained the problem-solving process in activities of acquiring components accordingly Function-Behavior-Structure (FBS) Framework [12]. FBS Framework is a way of enabling clearly divide features of artifacts into the function, the behavior, and the structure. It helps the comprehension and utilization of artifacts by not only humans also computers. FBS Framework has been defined and utilized in fields of knowledge representation (e.g.. qualitative physics, applied ontology) [4,5,6, 9].

We considered that understanding components from such a perspective provide more effective feedback to previously unclear features. As such, we mapped and considered FBS Framework into activities of acquiring components as follows (Fig. 2):

  • Function: a function to be achieved, as a result of interpreting particular behavior,

  • Behavior: a sequence of data states between initial-state (given) to end-state (result),

  • Structure: a set of source program as operators that achieves a function by producing behavior.

In a similar way, Kruchten’s study has the same perspectives as ours in the domain of software design [13].

In light of such problem-solving, we considered that learners need to think of relationships not only between function and structure but also among these and behavior, for acquiring components.

Fig. 2.
figure 2

The problem-solving process in programming.

Full size image

3.2 Implementation

Based System Using BROCs. Heretofore, we developed an ITS using BROCs as a pedagogy [10, 11]. This system is available in Windows as a desktop application and developed using C#. Figure 3 shows an user-interface (UI) of this system with BROCs. On the middle-top of UI shows the target component that learners should be acquired and a problem and constraints for acquiring it. To construct the target component, on the left of UI shows the component list for selecting components to add to the workspace. The workspace shows on the lower-right of UI. In the workspace, learners can be edit to the textbox, order, and level in the hierarchy, of components.

In addition, this system have following feedback functions for scaffolding learners’ activities in BROCs:

  • Simple Hint: give a hint that pointed out an error of textbox/order/level of the uppermost in the procedure,

  • Detail Hint: give a correct answer to such error,

  • Check Answer: tell learners whether their answers are correct.

However, these feedback functions only presented the excess or deficiency compared to the correct answer when learners mistake. In other words, learners were only given feedbacks related to the structure for achieving the function. Accordingly, there was no support for learners to understand the behavior that is considered to be important factors for acquiring components.

Table 1. Functional comparison of the conventional system and Compogram.
Full size table
Fig. 3.
figure 3

An user-interface of our previous system.

Full size image

Proposed System: Compogram. Here, we developed an ITS based on the conventional system, called Compogram. Compogram have a function of visualizing behavior by extended the conventional system. Due to this, as we stated the importance in Sect. 3.1, Compogram can promote that awareness of relationships not only between function and structure but also among these and behavior, in learners’ activities of acquiring components. Like the conventional system, Compogram is available in Windows as a desktop application and developed using C#.

It is a mostly comprehensive extension compared to the conventional system’s functionality (see Table 1). However, in order for learners to notice the errors by themselves, the simple hint function has been removed. On the other hand, the detail hint function was left in place to help learners clear up any impasse by themselves.

In Compogram, as stated above, we extended a function of visualizing behavior. Figure 4 shows an UI of Compogram. The added function in the right of UI presents the behavior and differences between learners’ constructs and correct answers. And, the initial-state as input is changing per click the check button. Accordingly, if including control-flow effected by initial-state in constructs, also changing behavior.

Fig. 4.
figure 4

An user-interface of Compogram.

Full size image

4 Comparative Evaluation

4.1 Method

Fig. 5.
figure 5

Design of comparative evaluation.

Full size image

For addressing RQ4 (Does provide our designed feedbacks/interventions lead to more learning gains?), we conducted a comparative evaluation between the conventional system with/without feedback and Compogram.

Participants were 24 undergraduate and graduate students who have studied programming for at least three years in programming lectures. All participants had already learned basic concepts such as “for” and “if” statements, sorting algorithms, and functions. And, we divided participants into three groups: (CS) a group using the conventional system (8 participants), (CS-NF) a group using the conventional system without feedback (7 participants), and (CMPG) a group using Compogram (9 participants). As annotation, CS and CS-NF groups were already evaluated in [11]. In other words, CS and CS-NF evaluation data were collected from this another study.

Our design of comparative evaluation shows in Fig. 5. As a procedure, firstly, we conducted a pre-test in 25 min that contained three problem sets that can be developed stepwise to measure fundamental programming and design skills. These problem sets contain the following problems in stated time limits:

  • Basic problems (10min.): it included 11 problems in the range of learning in systems,

  • Learning-transfer problems (10 min.): it included 9 problems out of the scope of learning tasks, but that can be solved by transferring learning in systems,

  • Other-domain problems (5 min.): it included simple 4 problems in the other domain, that are almost independent of learning in systems, for measuring the gain of skills.

Also, in the pre-tests (and post-tests), we instructed to do structuring such as functionalization or componentization in each problem to the extent possible, and reuse earlier answers in other problems. These evaluations made two types of evaluations: (1) a procedural score that evaluated whether the procedure is correct, and (2) a structuralized score that evaluated whether learners trying to functionalize and reuse in other problems, and it excluded the correctness of procedures. After that, we conducted learning tasks in divided groups with each system, for 30 min. Finally, we conducted a 25 min post-tests with the same contents and evaluation as the pre-tests.

4.2 Result and Analysis

Figure 6 and Fig. 7 show the results of means of procedural/structuralized scores in pre- and post-tests. In these figures, we normalized each score to percentages divided means by maximums which different per each problem set and evaluation method. To analyze learning gains, we conducted a 3 (CS, CS-NF, or CMPG) x 2 (pre- or post-test scores) ANOVA for each problem set per scoring.

Fig. 6.
figure 6

Results of basic problems and learning-transfer problems.

Full size image
Fig. 7.
figure 7

Results of other-domain problems.

Full size image

In procedural scores of basic problems from Fig. 6, while results showed no significance among groups [\(F(2, 21) = 1.87, p > .10, \eta _p^2 = .15\)]; it showed significant learning effects of each system within groups [\(F(1, 21) = 28.45, p < .01, \eta _p^2 = .58\)]. Results of learning-transfer problems also showed no significance among groups [\(F(2, 21) = 0.16, p > .10, \eta _p^2 = .02\)] and significant learning effects of each system within groups [\(F(1, 21) = 13.78, p < .01, \eta _p^2 = .40\)].

In structuralized scores from Fig. 6, the results are similar to the results of procedural scores. In detail, the results of basic problems showed no significance among groups [\(F(2, 21) = 1.51, p > .10, \eta _p^2 = .13\)] and significant learning effects of each system within groups [\(F(1, 21) = 31.40, p < .01, \eta _p^2 = .60\)]. Results of learning-transfer problems also showed no significance among groups [\(F(2, 21) = 0.03, p > .10, \eta _p^2 = .00\)] and significant learning effects of each system within groups [\(F(1, 21) = 13.86, p < .01, \eta _p^2 = .40\)].

In procedural scores of other-domain problems from Fig. 7, similar to the above, the results showed no significance among groups [\(F(2, 21) = 1.05, p > .10, \eta _p^2 = .09\)] and significant learning effects of each system within groups [\(F(1, 21) = 13.09, p < .01, \eta _p^2 = .38\)]. On the other hand, the results in structuralized scores were different from the above. while the results showed no significance among groups [\(F(2, 21) = 1.28, p > .10, \eta _p^2 = .11\)]; it showed significant learning effects of each system within groups [\(F(1, 21) = 3.51, p < .10, \eta _p^2 = .14\)]. Furthermore, it showed significantly AxB interaction [\(F(2, 21) = 3.51, p < .05, \eta _p^2 = .25\)]. In an analysis of AxB interaction, the results within groups were showed significant learning effects in only CMPG [\(F(1, 21) = 10.52, p < .01\)].

In consideration of the above results and analysis, we considered that the learning effects of Compogram are almost the same as the conventional system with feedback. On the other hand, as a distinguished point, Compogram demonstrated its ability for other-domain problems that almost not related to the learning range of systems. That is, results were suggested that Compogram was fostering knowledge organizing skills that can apply to out of learning ranges. As stated above, we addressed RQ4 (Does provide our designed feedbacks/interventions lead to more learning gains?).

5 Discussion and Future Work

In this paper, we pointed out that insufficient support for organizing knowledge in programming education. Therefore, based on our previous studies, we considered a new way of support for organizing knowledge in ITS. In this way, we developed and evaluated Compogram. Furthermore, evaluation results suggested that Compogram was fostering knowledge organizing skills that can apply to out of learning ranges.

Our future works are to conduct more large-scale experiments in classrooms, verify the learning effects, and realize more adaptive feedback in the system.