Printed from https://www.writing.com/main/view_item/item_id/1974157-Evaluation-of-a-Project

No ratings.
An attempt to enhance the computer skills of incoming (ITDE) students. |

https://www.amazon.com/Scholastic-Educational-Research-Technology-Mathematics/dp... *** Evaluation of Project Get Going. Introduction. Project “Get Going” (GG) has gained much attention because it is one of several research projects being conducted at a prominent university, designed to enhance the computer skills of incoming Instructional Technology and Distance Education (ITDE) students. It involved the use of a multimedia package incorporated popular music and culturally appropriate vignettes and exercises. GG was initiated with 21 incoming graduate students who were divided into three teams: the red team, the blue team, and the green team. Each team attended daily hour-long, lab-based instruction using the multimedia package for 25 days before they began their graduate studies. Project GG was designed to answer the following questions: • Did participation in the instructional program result in an increase in students’ comprehension and mastery of computer concepts and skills? • Is there a significant relationship between the pre and posttest scores for the entire sample group? For the males? For the females? For each of the teams? • Is there a significant difference between pre and posttest scores for the sample, as a whole? For the males? For the females? For each of the teams? From the research questions, if the results showed that Project GG has significantly increased the computer skills of the students involved, The university has promised to replicate it in at least five other graduate programs at the university. The evaluation was conducted wherefore all participants were given a pre and a posttest to measure their level of comprehension and mastery of computer concepts and skills prior to, and upon completion of Project GG. The data as set out in Appendix A reflect the pre and posttest scores of the 21 students. The analysis is as follows: Findings. Descriptive Analysis. Table1 Percentage Distribution of Students by Teams and Gender Teams n Male Female Alla ________________________________________________________________________ Red 7 28.6 71.4 33.3 Blue 7 71.4 28.6 33.3 Green 7 57.1 42.9 33.3 All 21 52.4 47.6 100.0 Note. aTeam as percentage of group. As shown in Table 1, there is a greater percentage of males overall. The three teams have the same percentage of participants; however, there is a greater percentage of males in the Green team and also in the Blue team but for the Red team, there is a greater percentage of females (see Appendix F for crosstabs). Table 2 Statistical Analysis of Students by Gender, Team, and Group Group n Mean Median Mode Variance SD Range Gender Male Pretest 11 13.82 14.0 13a 7.36 2.71 8 Posttest 11 18.36 19.0 20 10.65 3.26 11 Female Pretest 10 15.30 14.5 14a 21.79 4.67 14 Posttest 10 21.70 22.5 15a 32.68 5.72 14 Teams Red Pretest 7 15.29 15.0 10a 13.90 3.73 11 Posttest 7 22.71 22.0 20 19.24 4.39 12 Blue Pretest 7 14.71 14.0 9a 20.90 4.57 14 Posttest 7 18.57 16.0 16 27.62 5.26 16 Green Pretest 7 13.57 14.0 9a 10.29 3.21 9 Posttest 7 18.57 19.0 15 16.29 4.04 10 All Pretest 21 14.52 14.0 14 14.06 3.75 14 Posttest 21 19.95 20.0 16a 22.95 4.79 16 Note. aMultiple modes exist. The smallest value is shown. As shown in Table 2, there is a distinct difference between pretest and posttest after treatment in all categories; viz. Gender, Team, and Group. For Group: The means for pre and posttest showed a difference after treatment. The standard deviation showed a difference in variation in scores on pretest and posttest after treatment (see Appendix B). For Gender: Female performed better in both pretest and posttest with a difference in means of 1.48 and a difference in standard deviation of 3.34 for pretest and posttest respectively. The standard deviation for pretest and posttest showed that female scores had a greater spread (see Appendix D). For Team: The means for pretest and posttest for all the teams showed a difference after treatment, in that the mean increased in each case. However, it appears that the treatment has a greater effect on the Red team. The standard deviation for the pretest and posttest increased in each case for the respective teams but the Green team seemed to have had the greatest spread of scores (see Appendix C). ________________________________________________________________________ Table 3 Percentage Distribution of Students by Range of Scores Score Pretest Posttest 4-9 9.5 - 10-15 52.4 19.0 16-21 33.3 47.7 Over 21 4.8 33.3 Note. Scores for entire group. Dash indicates that there was no score for the category. From Table 3, it can be seen that for the pretest, more than 60% of the students scored 15 and below and less than 40% of the students scored 16 and above. After the treatment however, more than 80% of the students scored 16 and over. This showed that there were some amount of improvement with a range of scores of 14, that is, (9 to 23) for the pretest and 16, that is, (13 to 29) for the posttest (see Appendix B). Correlation Analysis. __________________________________________________________ Table 4 Correlation of Student scores n df r r2 Variance Sig. Value Sig. Group 21 20 .825** .68 68 .000 .01 Gender Male 11 10 .607* .36 36 .048 .05 Female 10 9 .903** .81 81 .000 .01 Teams Blue 7 6 .910** .82 82 .004 .01 Green 7 6 .756* .57 57 .049 .05 Red 7 6 .862* .74 74 .013 .05 Note. **Correlation is significant at .01 level (two-tailed). *Correlation is significant at .05 level (two-tailed). For the Group (Table 4) a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A strong positive correlation was found (see Appendix E). Thus (r (20) = .825, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Male Gender, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A moderately positive correlation was found (see Appendix E). Thus (r (10) = .607, p < .05) level, thus indicating a significant linear relationship between the two variables. For the Female Gender, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A very strong positive correlation was found (see Appendix E). Thus (r (9) = .903, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Blue Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A very strong positive correlation was found (see Appendix E). Thus (r (6) = .910, p < .01) level, thus indicating a significant linear relationship between the two variables. For the Green Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A moderately positive correlation was found (see Appendix E). Thus (r (6) = .756, p < .05) level, thus indicating a significant linear relationship between the two variables. For the Red Team, a Pearson correlation coefficient was calculated for the relationship between pretest and posttest. A strong positive correlation was found (see Appendix E). Thus (r (6) = .862, p < .05) level, thus indicating a significant linear relationship between the two variables. From Table 4, however, the female scores are more predictable than the male scores and the blue team scores are more predictable than the green team and red team scores. Paired T-test Analysis. ________________________________________________________________ Table 5 Comparison of Population of Pretest and Posttest means Variable Pre-t M SD Post-t M SD T-value Sig. Lev df Group 14.52 3.75 19.95 4.79 9.172 .000 20 Male 13.81 2.71 18.36 3.26 5.590 .000 10 Female 15.30 4.66 21.70 5.71 8.085 .000 9 Red 15.28 3.72 22.71 4.38 8.832 .000 6 Blue 14.71 4.57 18.57 5.27 4.653 .003 6 Green 13.57 3.20 18.57 4.03 5.000 .002 6 Note. P < .05 (two-tailed test) A paired sample t-test was calculated to compare the mean pretest score and the mean posttest score for the group, for males, for females, and for each team. For the Group, the mean on the pretest was 14.52, (SD = 3.75) and the mean on the posttest was 19.95, (SD = 4.79). A significant increase from pretest and posttest was found. Thus (t (20) =9.172, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Male Gender, the mean on the pretest was 13.81, (SD = 2.71) and the mean on the posttest was 18.36, (SD = 3.26). An increase from pretest and posttest was found. Thus (t (10) = 5.590, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Female Gender, the mean on the pretest was 15.30, (SD = 4.66) and the mean on the posttest was 21.70, (SD = 5.71). A significant increase from pretest and posttest was found. Thus (t (9) = 8.085, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Red Team, the mean on the pretest was 15.28, (SD = 3.72) and the mean on the posttest was 22.71, (SD = 4.38). A significant increase from pretest and posttest was found. Thus (t (6) = 8.832, p < .01); thus indicating a difference in means with at least 1% chance of errors. For the Blue Team, the mean on the pretest was 14.71, (SD = 4.57) and the mean on the posttest was 18.57, (SD = 5.27). An increase from pretest and posttest was found. Thus (t (6) = 4.653, p < .01); thus indicating a difference in means with at least a 1% chance of errors. For the Green Team, the mean on the pretest was 13.57, (SD = 3.20) and the mean on the posttest was 18.57, (SD = 4.03). An increase from pretest and posttest was found. Thus (t (6) = 5.00, p < .01); thus indicating a difference in means with at least a 1% chance of errors. Gender Comparison. ________________________________________________________________________ Table 6 Group T-test on Pretest and Posttest by Gender N df Pre-M SD Post M SD Pre-t-val Post t-val Pre t-sig Post t-sig Pre f-val Post f-val Mal 11 9 13.8 2.71 18.4 3.26 .900 .878 1.66 1.62 .379 .395 .113 .127 3.23 4.66 Fem 10 8 15.3 4.67 21.7 5.72 .900 .878 1.66 1.62 .379 .395 .113 .127 .088 .045 Note. Val - represents value. Mal- represents male. Fem- represents female. The Levene’s Test for equality of variances showed that for the pretest, F = 3.228 and the Significant Level = .088. Thus, indicating that there is not a significant difference between the means either at p < .05 or p < .01 levels. For the posttest F = 4.611 and significant level = .045; indicating that there is not a significant difference between the means at p < .01, only at p < .05 level. Hence, assuming equal variances at p < .01, then the pooled variance estimate would be employed for this purpose. It was seen that there was no significant difference for t-value although the means and standard deviations on the pretest and posttest were higher for females. (see Appendix D). Team Comparison. ________________________________________________________________________ Table 7 Comparison of Teams on Pretest and Posttest. ANOVA showing Means for each Team. Team n PretestM SD Post M SD F-Pretest F-Post - test Sig. Pretest Sig. Post Red 7 15.29 3.73 22.7 4.39 0.355 1.903 .706 .178 Blue 7 14.71 4.57 18.57 5.26 Green 7 13.57 3.21 18.57 4.04 Differences in performance for the teams on both tests were not significant. Both pretest and posttest means were compared using a one-way ANOVA. No significant difference was found. For Pretest ( F (2,18) = (.355), p > .05). For Posttest ( F (2,18) = (1.903), p> .05). Hence, the three teams did not differ significantly with respect to their performance on pre and post test. (Appendix B) Summary The scores for pretest and posttest were analyzed using measures of central tendency, measures of variability, correlation, and analysis of variance. A statistical analysis of students was done to find out whether or not there is any difference in performance by Gender or Team. The analysis was done for the entire group, gender, and team. Further analysis was performed so as to pinpoint whether or not the difference was significant. A correlation analysis was performed in each case for the pretest and posttest measures. The correlation was graphed in each case for the two variables to indicate whether or not the scattergraph shows a positive correlation (an increase in one variable is associated with an increase in the other variable) or a negative correlation (an increase in one variable is associated with a decrease in the other variable) (Appendix E). This will tell us about the strength/weakness of the relationship between the variables. The Pearson product moment (Pearson r) was chosen because: 1. The data used were measured on interval scale, and 2. The variables (pretest and posttest) had a linear relationship in all cases. The t-test was used to compare the pre and posttest means for the Group, Gender, and Teams. The paired t-test was chosen to compare the pre and posttest in each case. An independent t-test was chosen for the Gender. The Levene’s Test was done to test for equality of variances, thus, to test for the significance of the difference between means. The F-test was done for Team Comparison to check if there were a significant difference between means. For the entire group, there was an increase in means from pretest to posttest scores. For Gender, there was an increase in means from pretest to posttest scores for males and for females respectively. However, when the t-test was used for gender comparison there was not a significant difference in means for males as compared to females. For Teams, there was an increase in means from pretest to posttest scores within each team. However, when the ANOVA, F-test was used for team comparison, no significant difference was found when pre and posttest means were compared. For research question 1. Did participation in Project GG increase students comprehension and mastery of computer concepts and skills? Yes, because in all cases: for the entire group, for gender, and for the teams there is a marked increase and difference of pretest and posttest scores there is a significant difference in each case for at least at the .05 level, that is p < .05. This is to say that after the treatment, the posttest revealed an improvement in scores. For research question 2. Is there a significant relationship between pre and posttest Scores for the entire group? For males? For females? and for each of the teams? The scatter graph (Appendix E) showed that there is a significant relationship between pre and posttest for : the entire group, which showed a strong positive correlation (r = .825) the Male Gender which showed a moderately positive correlation (r = 0.607); the Female Gender, which showed a very strong positive correlation (r = .903);the Blue Team which showed a very strong positive correlation (r = .910). The Green Team, which showed a moderately positive correlation (r = .756) and the Red Team, which showed a strong positive correlation (r = .862). From all indications the treatment made a positive difference in enhancing the comprehension and mastery of computer concepts and skills of the participants. Interpretation and Recommendations The findings of the study might have been affected by the sample size. The writer needs to know the likely size of the population and how the sample of 21 was selected, that is, what sampling procedure was used to select the 21 participants. The multimedia package that was used during the treatment might have been culturally biased. The method of instruction for the treatment might have been a crucial factor in affecting one’s performance. There is nothing to say whether or not the team worked cooperatively or individually within teams. To determine whether or not the treatment made a difference, the writer will look at the statistical power of the tests. Power is the ability of the test to show a statistical difference. Statistical power, in fact, is the probability of avoiding a Type 11 error ( i.e. the probability of retaining the null hypothesis when it should be rejected). For Project GG the question is whether or not the analysis had enough POWER to show a significant difference when the treatment (viz. Lab-based instruction using multimedia package to enhance the computer skills of incoming ITDE students) was concluded. The power of a significance test depends on the four interrelated factors: 1. Sample size, 2. Alpha level, 3. Directionality, and 4. The effect size. The Sample size for Project GG (21) could affect the statistical power. For example, power increases automatically with an increase in sample size. Thus, virtually any difference can be made significant if the sample is large enough. A standard sample size of a minimum of 30 subjects per group is recommended in most studies. However, in well – designed experimental studies, a smaller sample size is acceptable. The Alpha level selected for Project GG (p < .05 or p < .01) might have affected the POWER of the treatment depending on choice. Increasing the significance level – say, from .05 to .01 thus increases the statistical power. So, the statistical significance tests might have been more powerful if there were more significance at the .01 level. The Directionality refers to one-tailed or two-tailed. For Project GG, a non-directional (2-tailed) was chosen. This choice however, although opt as the more conservative, usually lessen the statistical power. The Effect Size = (mean of Post T – mean of Pre T) Avg. SD of Pre T and Post T =19.9524-14.5238 (3.7499+4.7904)/2 =5.4286/4.2702 =1.2713 (1.2713 means a big difference, i.e. 1.2713 SD). The Effect Size for Project GG is calculated to be about 1.2713, which is considered high or a big difference. This is a desired Effect Size. A close examination of the statistical power of Project GG would suggest that if power were to be increased then the writer would recommend: 1. an increase in sample size, 2. An increase in significance level, and 3. assume a research hypothesis that is directional, using the one-tailed test. The writer would recommend that more demographic information be collected about the participants such as age group, occupation, and previous knowledge on computer skills. For expansion, Project GG could also experiment separately on single gender. |

Printed from https://www.writing.com/main/view_item/item_id/1974157-Evaluation-of-a-Project