The George Washington University- GSEHD
Ed.D. in Curriculum and Instruction (C&I) & Educational Technology Leadership (ETL)
Paradigms of Instruction and Assessment  - TRED 330
Instructor: Dr. Natalie Milman
Fall 2010

"Nada's.Island"

Critiqued Study:
“Effects of Technology Immersion on Teachers’ Growth in
Technology Competency, Ideology, and Practices”
By Kelly Shapley (Shapley Research Associates, Austin, Texas)
& Daniel Sheehan, Catherine Maloney, Fanny Caranikas-Walker
(Texas Center for Educational Research, Austin)
2010

Critique (draft)- pdf

 
Abstract
This paper is an academic critique of an article written by Kelly Shapley, Daniel Sheehan, Catherine Maloney, and Fanny Caranikas-Walker (2010) and titled, “Effects of Technology Immersion on Teachers’ Growth in Technology Competency, Ideology, and Practices.” It examines the purpose, rationale, method, results, and strengths and weaknesses of the study of the effect of the Technology Immersion model including the school environment and support system on teachers’ growth rate for technology knowledge and skills, ideological affiliations with technology integration and learner-centered instruction, and the frequency of class activities involving technology in high-need middle schools. The schools were “immersed” in technology by providing laptops for each teacher and student, instructional and learning resources, professional development, and technical and pedagogical support. The study reports third-year findings for the teacher component of the theory-driven evaluation and uses Hierarchical Linear Modeling (HLM) to analyze longitudinal survey data. The results of this study revealed that Technology Immersion and the school support system have statistically significant effects on teachers’ growth in technology competency, ideology, and practices. Although this study was very comprehensive and well conducted, it did have a few limitations, namely in its external and internal validation, in the lack of information consent (or at least none was reported), and in the study design.

Purpose of the Study
The purpose of the study is a brief description of what the study examined. The research questions or hypotheses often provide a concise statement of the purpose of the study. Please identify and comment on the paradigmatic lens or worldview through which the research is conducted.
Clear and succinct description of the study examined, including all of the following (if applicable):
1. research questions or hypotheses
2. description of paradigmatic lens or worldview through which the research was conducted.

The purpose of this study was twofold: first, to examine the effect of Technology Immersion on teachers’ technology knowledge and skills, ideological beliefs about technology integration and learner-centered instruction, and classroom practices; second, to examine the relationship between the school support mechanisms for technology immersion and core subject teachers' ideologies, classroom practices, and personal technology use.

This study addressed two distinct research questions:

1. Do teachers who are immersed in technology-rich school and classroom environments increase their technical expertise, adopt supportive ideologies, and change their practices at faster rates than teachers in control schools with more conventional levels of technology, and
2. What factors, both personal and contextual, help to explain teacher change (or lack thereof)? (Shapley et al., 2010, p. 4)
Shapley et al. (2010) conducted the research in various middle schools which were assigned to either treatment or control groups. The research design was longitudinal and quasi-experimental since randomization, one of the three elements of true experimental research, was lacking. As for the paradigmatic lens, it was not mentioned in the study, but we can safely infer that it is post-positivist since quantitative data were used to predict, to test an existing theory, and not to develop a theory, as is the case in positivism (Willis, 2007). Besides, the study did explicitly mention threats to its external validity, which is definitely what a study conducted through a positivist lens would not envisage.

Rationale for the Study
The rationale for the study involves a brief review of the theory and research which form the foundation for the study and a logical argument about why the study is needed.
Provide a clear yet brief rationale that includes all of the following:
1. a logical argument about why the study is needed
2. the theory and research which form the foundation for the study

According to the literature review Shapley et al. (2010) conducted, although research has proved that it is crucial to incorporate technology in education in order to improve student learning in the 21st century, technology is not being used in schools nowadays the way it should be, despite the large increase of the availability of instructional computers and Internet access. Technology is still being used “on the surface;” students use it very little everyday, mainly in technology-related classes, and much less in core-content classrooms; and when they do, “students use computers for word processing, skill-based instructional software programs, and Internet searches rather than for more demanding activities such as multi-media authoring, spreadsheets/ databases, and electronic communication” (p. 2). Besides, whereas a few teachers use technology to innovate, most only use it to “automate activities previously completed with paper and pencil” (p. 2). After reviewing the literature, the authors concluded that there is a strong relationship between teachers’ proper use of technology in the classroom and both personal and contextual factors. The personal factors include teachers’ “technical proficiency, beliefs about technology’s value and costs, teaching philosophy, and historically understood conceptions of teaching” (p. 3). As for the contextual factors, they include “subject-area and grade-level assignments, access to classroom computers, the quality of professional development, assistance from colleagues, the presence of competing innovations, and organizational pressures for change” (p. 3). Therefore, the researchers considered it crucial to conduct research in order to investigate “whether school reform efforts that recognize teachers’ personal needs and provide supportive contextual conditions” (p. 3) can actually lead a much larger number of teachers to properly incorporate technology in their teaching, in order to better prepare students for the innovations of the 21st century.

The Theoretical Model of Technology Immersion, which drove this study, proposes a “linear sequence of causal relationships” (p. 4), and assumes that, in order for technology to be used effectively, treatment schools need to adopt a comprehensive approach. Those schools should be first “immersed” in technology by providing for treatment teachers and/or students immersion components such as: wireless computing devices, technology-based learning resources, professional development, and technical and pedagogical support for effective technology use. As a result of proper execution of the model, the school technology environment will improve, teachers will become more proficient in technology use, their professional productivity and technology integration will advance, their instruction will be more learning-centered, and the student activities they use will incorporate more technology. Consequently, students will learn and achieve better academically.

Method
The method refers to how the study was conducted. In this section you should briefly describe what was done in the study.
Provide a clear yet succinct review of the methods applied in this study, including the following (if applicable):
1. Variables studied
2. Sample size and selection procedures
3. Researcher’s role
4. Data collection procedures
5. Data analyses procedures
 

1. Variables Studied
The study covered two different sets of variables: teacher and implementation variables. The independent teacher variable is technology immersion (that includes computing devices, digital resources, professional development, and technical and pedagogical support), for which the dependent variables are teachers’ technology knowledge and skills (technology proficiency and use of technology for professional productivity), ideological beliefs about technology integration and learner-centered instruction, and classroom practices that include technology.

The independent implementation variable is school support mechanisms which include leadership, innovative culture, parent and community support, technical support, and professional development (including contact hours, classroom support, coherence, and content-focus). The dependent variables for the second independent variable are core subject teachers' classroom immersion which includes ideology (technology integration and learner-centered instruction), classroom practices that incorporate technology, and personal technology use (communication and productivity). The researchers focused on core-subject teachers here “because of their direct association with student academic outcomes measured by state assessments” (p. 20).

2. Sample Size and Selection Procedures
The sample size consisted of 42 high-need middle schools (grades 6 to 8), twenty-one of which were considered treatment (technology immersion) schools, and the remaining 21 were considered control schools. The treatment schools were chosen through competitive Technology Immersion Pilot (TIP) grants that the Texas Education Agency (TEA) dedicated to schools regarded as high-need because of poverty or a great lack of technological resources. The factors that played a big role in school selection include proposal ratings and school size, location, student diversity, and academic achievement. However, the determining factors were the need for geographical distribution and the availability of comparable control schools.

The control schools, which also received federal funds for participation in the study ($25,000 per year, 25% of which were dedicated to professional development), had to be carefully selected for issues of eligibility and treatment group matching characteristics. The researchers did address the possibility of selection bias, but they stated that, in order for the sample of schools to have good internal validity, with “no large, statistically significant treatment-control group differences” (p. 11), they worked hard to match the two groups with regard to school size; school districts; teacher gender, ethnicity, advanced degrees, and average teaching experience; student characteristics (economically disadvantaged, minority, English as a second language (ESL), and special education; and, finally, students’ enrollment, mobility, and TAKS (Texas Assessment of Knowledge and Skills) passing rates. Nevertheless, in both treatment and control groups, most teachers turned out to be female, White or Hispanic (very few African American), with an average of 11 years of teaching experience. As for the students, although the researchers claim that they were ethnically diverse, their sample consisted mainly of Hispanic (58%) and White (36%), and also very few African American (7%) students. Besides, because the TIP grant was aimed at high-need schools, “nearly 70% of students in the study came from economically disadvantaged backgrounds” (p. 11).

3. Researchers' role
After adapting survey items and scales from instruments that were previously validated and which include the Technology Proficiency Self-Assessment, Levels of Technology Implementation (LoTi) Questionnaire, and Observation Tools for School Observers, the researchers asked teachers to complete surveys over a three-year period. Those surveys consisted of items and scales measuring the teacher outcome variables hypothesized, and those included the following 3 categories: Technology Proficiency (27 items), Technology Integration (10 items), Learner-Centered Instruction (4 items), all of which involved a 7-point scale ranging from 1 (not true of me now) to 7 (very true of me now). The surveys also included 2 additional categories, Professional Productivity (17 items) and Student Classroom Activities (17 items), both of which involved a 5-point scale to rate the frequency of activities or interactions: 1 (never); 2 (rarely—e.g., a few times a year); 3 (sometimes—e.g., once or twice a month); 4 (often—e.g., once or twice a week); and 5 (almost daily). Cronbach’s alpha reliability coefficients for the scale scores ranged from 0.70 to 0.99.

As for the implementation outcome variables, the researchers used the survey items and scales they gave treatment teachers only in the third year in order to derive immersion standard scores relative to the relationship between:

(a) Immersion Support for implementation (measures of school Leadership, Innovative Culture, Parent and Community Support, Technical Support, and Professional Development); and
(b) Classroom Immersion (a composite measure of each teacher’s Technology Integration, Learner-Centered Instruction, Student Activities, Communication, and Professional Productivity) (p. 13).


4. Data Collection Procedures
The researchers administered, both to treatment and control teachers, online surveys that covered items and scales that measured the aforementioned teacher outcome variables hypothesized. The numbers of teachers that completed the surveys are as follows: 1,271 teachers in fall 2004 (97% of treatment and 98% of control), 1,144 teachers in spring 2005 (87% of treatment and 88% of control), 1,175 teachers in spring 2006 (92% of treatment and 95% of control), and 1,208 teachers in spring 2007 (94% of treatment and 93% of control). In order to measure implementation outcome variables, the researchers used the online surveys that treatment teachers completed the third year, in spring 2007 (N = 619, including 371 core-subject teachers).

5. Data Analyses Procedures
The researchers used two analytic approaches: the Hierarchical Linear Modeling (HLM) approach and the correlation and regression analysis approach. They used the first one to evaluate the effects of Technology Immersion on teachers’ 5 outcome variables. This approach involved a series of multi-level analyses (three-level hierarchical linear growth models) which contrasted “treatment and control teachers’ individual growth trajectories” (p. 14) for each of the variables. The researchers used the second analytic approach to investigate associations between school support mechanisms and treatment teachers’ classroom immersion which includes ideology, classroom practices, and personal technology use.
 

Results
The results section describes what was found in the study and the conclusions the researcher drew from these results.
Provide a clear yet succinct examination of the results of the study, including the following (if applicable):
1. the study’s findings
2. the conclusions the researcher drew from these results
 
The Study’s Findings
In order to cover the study’s findings, it is imperative to revisit the different variables studied. The independent teacher and implementation variables are technology immersion and school support mechanisms, with all their break down components: computing devices, digital resources, professional development, and technical and pedagogical support for the independent teacher variables; and leadership, innovative culture, parent and community support, technical support, and professional development (including contact hours, classroom support, coherence, and content-focus) for the independent immersion variables.

As for the dependent teacher and implementation variables, they are: technology knowledge and skills, and teachers' classroom immersion, also with all their break down components: technology proficiency and use of technology for professional productivity, ideological beliefs about technology integration and learner-centered instruction, and classroom practices that include technology, for the dependent teacher variable; and ideology (technology integration and learner-centered instruction), classroom practices that incorporate technology, and personal technology use (communication and productivity), for the dependent implementation variable.

The results stated that, on the one hand, Technology Immersion had a statistically significant positive effect on teachers’ rates of growth for each of the teacher variables. And on the other hand, there were significant relationships between school support mechanisms and implementation variables. It is noteworthy that, in the first two years of the study, teachers in treatment schools witnessed much higher growth trends than those in control schools. However, the last year of the research, control teachers also experienced highly positive growth trends for the technology integration variables; and that was due to the fact that, at the end of the second year, the TEA also offered grants to 16 of the 21 control schools “to begin planning for Technology Immersion” (p. 10).

As a result of Technology Immersion (computing devices, digital resources, professional development, and technical and pedagogical support), treatment teachers improved in technology proficiency more than twice as much as control teachers: “0.31 scale-score point per year on a 7-point scale versus 0.13 point, respectively” (p. 17); they reached significantly higher growth rates than control teachers in technology use for professional productivity; they viewed technology integration in the curriculum and in instruction, including innovative practices, much more positively; they “changed at a significantly faster rate [than those in control schools] in their affiliations with principles of Learner-Centered Instruction” (p. 19); finally, their Student Classroom Activities grew at a significantly faster rate: “0.23 and 0.03 scale-score points per year, respectively, for treatment and control teachers in average poverty schools” (p. 19). It is worth mentioning that the study found that the growth rates of teachers who taught both at treatment and control schools with higher levels of poverty were not as high as the others’ with regards to all the teacher outcome variables.

As for the relationship between school support mechanisms (leadership, innovative culture, parent and community support, technical support, and professional development [including contact hours, classroom support, coherence, and content-focus]) on the one hand, and teachers' classroom immersion, on the other, partial correlation coefficients demonstrated that it is positive and statistically significant in general, mainly with regards to technology integration, learner-centered instruction, and classroom practices that incorporate technology. The study showed, however, that school support mechanisms had a weaker relationship with teachers’ personal technology use for communication and productivity. The researchers concluded here that teachers do not need “extensive school and classroom support” (p. 20) in order to increase their use of technology in their own professional lives. Besides, the study also confirmed that the relationship between technical support and teachers’ classroom immersion was the weakest, and that what teachers need mainly, especially in order to incorporate technology in student activities, is “in-class support from instructional specialists” (p. 22).

Finally, the study showed that, among all the school support mechanisms, it is the school’s culture of innovation that is the strongest predictor of teachers’ technology integration in the classroom. Parent and community support, contact hours through technology-related professional development during the school year, in addition to the aforementioned classroom support, were also statistically significant predictors of treatment teachers’ classroom immersion. In the end, the researchers stated that, as the teachers’ Technology Proficiency increases over the years, their classroom immersion also increases.

Researchers’ Conclusions
The researchers concluded that the results that they reached in the study were not surprising; they were expected and were consistent with former findings. The Technology Immersion Model “has merit, given the statistically significant growth across 3 years for treatment teachers and their teaching practices (yearly growth rate gamma coefficients favoring treatment teachers over control for five dependent variables, 0.31, 0.20, 0.59, 0.38, 0.23 for teachers in schools with average poverty, p < .001)” (p. 24). Indeed, teachers who are immersed in technology-rich school and classroom environments do increase their technical expertise, adopt supportive ideologies, and change their practices at faster rates than teachers in control schools with more conventional levels of technology; and this serves to answer the first research question. As for the second research question about the personal and contextual factors that help to explain teacher change, the researchers advanced with regards to the personal factors that teachers’ characteristics influenced their beliefs and practices, and it is their technological proficiency that is “the strongest predictor of teachers’ Classroom Immersion” (p. 25). Two factors, the study showed, play a negative role in teachers’ proficiency in technology and hence, in their classroom immersion: teaching experience and poverty. The more experienced the teacher is, the less likely he or she is to be technologically proficient. Similarly, the higher the concentration of student poverty is in a school, the slower the rates at which teachers grow in technology proficiency are.

Vis-à-vis the contextual factors that help to explain teacher change, the researchers affirmed that some of the school support mechanisms are “critically important” (p. 25). Among those support mechanisms are (a) a school culture where colleagues are generally open to innovations in instruction and value the role technology plays in student learning; (b) long-term technology-related professional development; (c) numerous “classroom mentoring or coaching opportunities provided by either local pedagogical support staff or external consultants” (p. 25); and (d) support of community and, especially, that of parents, namely for one-to-one student laptop access, including the related financial and disciplinary responsibilities.

Instead of just providing computers in computer labs and media centers, it is important, Shapley et al. contend, to also provide laptops to each student: this would definitely promote teachers’ technology use in the classroom. Based on their findings, the researchers highly recommend that, in order to successfully improve technology incorporation in classrooms, it is better for a school to adopt a comprehensive school-wide model for the technology integration.
 

Critical Commentary
The critical commentary is one of the most important sections of the paper. In this section, you have the opportunity to comment on the value of the research as conceptualized, conducted and reported. You should comment on the value of the research to teachers, students and schools. Although both strengths and weaknesses are included in this section, it is not sufficient to discuss only the strengths of a study. Every study has weaknesses.
Provides a clear, succinct critical commentary that examines the strengths and weaknesses of the study, including the value of the research:
1. as a) conceptualized, b) conducted and c) reported
2. for a) teachers, b) students and c) schools

**Focus on:
• the purpose,
• my overall reaction,
• salient methodological issues,
• noteworthy weaknesses and strengths, and
• an overall recommendation (like whether anyone should pay attention to it or whether it should have even been published).
• should people attend to the research and possibly change their thinking or practice as a result?
http://www.sagepub.com/eic/14/Chapter14_Critique2.pdf

**Evaluate all aspects of the study: its substantive, theoretical, methodological, ethical, interpretive, and presentational dimensions.

I was very impressed with this study, including the way it was conceptualized, conducted, and reported (except for some repetitions that could have been avoided when writing the article). The article adequately reports the major features of the Theoretical Model of Technology Immersion, which is the theoretical framework used in this study. Although this framework is consistent with the post-positivist research paradigm that was adopted, it would have been much more effective to adopt a pragmatic/mixed methods paradigm which includes interviews and observations: the results would have been more comprehensive and accurate. The research questions, on which the whole study is based, naturally flow from the framework. The purpose and rationale of the study are legitimate and easy to identify with. The variables studied are constantly repeated throughout the paper, without explicitly pinpointing the independent and dependent ones. The sample size and selection were clear and exhaustive. The data collection and analyses procedures were well reported. At the end of the article, the researchers do tie the findings of the study back to the framework and the research questions; those findings do support the framework and are interpreted within its context.

On the other hand, I do believe this study has quite a few limitations, some of which the researchers mentioned explicitly. Those limitations are:
 

Internal Validity
The first limitation involves the study’s internal validity: there might have been some extraneous variables that interfered with the study’s internal validity. Those include the following:
 
Treatment group characteristics. Peer and/or administrative pressure, in addition to fear of humiliation within the treatment groups are two factors that were not addressed at all and posed limitations to this study. How confident were the teachers that the administration would not have access to their responses; that their jobs would not be threatened accordingly? How much extra time were the teachers actually given in order to learn more about technology? There is no such mention in the article. Besides, when were the teachers given the surveys? Did they fill them out hastily or did they take their time to do so? Did they just check the answers that were expected of them in order not to lose funding or even their jobs? None of the aforementioned questions were tackled.

Control group characteristics. In the third year of the project, the control group did not maintain the characteristics of a control group, namely after 76% of the schools that constitute this group are offered grants to start planning for Technology Immersion. Hence, teachers received laptops, instructional resources, and professional development (p. 10). It is true that those teachers only started being initiated to technology in the third year, versus the treatment teachers who had started 2 years earlier, but still, this is a factor to take into consideration.


External Validity
The third limitation involves the study’s external validity: to what extent can the results be generalized? The researchers do admit that because they had funding restrictions, “the treatment group was not representative of the average middle school in Texas” (p. 11) since it included a larger ratio of Hispanic teachers and students and a smaller ratio of White and (mainly) African-American teachers and students. Besides, most students came from poor rural areas. So, the results of this study can only generalize to schools with more Hispanic and economically disadvantaged teachers and students.

Ethical Concerns
There was no mention in the article of any independent ethics review by a board or a committee and no indication either that participants gave voluntary, informed consent. Did the subjects know they were being studied? This lack makes us seriously question the researchers’ ethical performance.

Study Design
The whole study relied on self-reported data from surveys given to teachers; so, the results relied entirely on those teachers’ perceptions; on their willingness to actually tell the truth; on the amount of time they were willing to dedicate to those surveys which might have been considered as additional chores. How reliable were those answers? Did the researchers conduct any debriefing? Providing debriefing would have allowed the researchers to gain valuable information through in-depth interviews.


Recommendations
I recommend that this study be repeated using a more representative sample, with more attention given to the characteristics of both treatment and control groups. Ethical concerns need to be addressed prior to the start of the study. Finally, by adopting a pragmatic paradigm based on mixed methods that include both surveys and interviews/observations, the results of the study would be more accurate.
 

---

References
 
Willis, J.W. (2007). Foundations of qualitative research: Interpretive and critical approaches. Thousand Oaks, CA: Sage.

Shapley, K., Sheehan, D., Maloney, C., & Caranikas-Walker, F. (2010). Effects of technology immersion on teachers' growth in technology competency, ideology, and practices. Journal of Educational Computing Research, 42(1), 1-33.

Related Reference
Texas Education Agency, & Texas Center for Educational Research (2009). Evaluation of the Texas Technology Immersion Pilot; Final Outcomes for a Four-Year Study (2004-05 to 2007-08). Retrieved October 5, 2010, from http://www.tcer.org/research/etxtip/documents/y4_etxtip_final.pdf




---

Relevant Links:
• Guidelines for Critiquing Research Articles
• Sage Publications- Chapter 14: Research Analysis and Critique
• How to Critique and Analyze a Quantitative Research Report- Author: EMS Village Staff
• Quantitative Research Article Critique, by Corey J. Ivany- Education 6100- Memorial University of Newfoundland
• Qualitative Research Article Critique, by Corey J. Ivany- Education 6100- Memorial University of Newfoundland
• Qualitative research critique
• Article Critique: Quantitative Research
• Research Critique: “Factors That Influence Self-Efficacy of Counseling Students: An Exploratory Study”- Kerry Heavy Runner and Neil Hokanson- University of Wyoming
• Example of a scholary research critique
• Foundational Issues: Postpositivist and Critical Perspectives
• Comparative Research Paradigms
• EXPERIMENTAL AND QUASI-EXPERIMENTAL TECHNIQUES
• Threats to validity of Research Design
• Guidelines for Critical Review of Qualitative Studies
• QUALITATIVE LITERATURE CRITIQUE WORKSHEET


Theoretical Model of Technology Immersion
In Red => What this study covers.

Technology Immersion Components:

1. a wireless mobile computing device for each educator and student;
2. productivity, communication, and presentation software for use as learning tools;
3. online instructional resources that support the state’s curriculum;
4. online formative assessment tools;
5. professional development for teachers supporting technology integration; and
6. initial and ongoing technical support to maintain an immersed campus (Shapley et al., 2010, pp. 4, 6).

---


Back to "Nada's ESL Island" || Back to Doctoral Classes at GWU
Back to Curriculum Theory
Foundations of Curriculum Theory: Notes & Reflections
Analysis of the Role of Teachers as Articulated in Kliebard, Tyler, and Apple Texts
The ESL Reading Curriculum: New Lenses
Education, One of the Impossible Professions
Emotional Intelligence (EQ) in Curriculum Theory: Why EQ Skills should be Incorporated in Teacher Education/Training
Integrating Technology in the Classroom- TPACK
Handbook of Technological Pedagogical Content Knowledge (TPCK) for Educators
Instructional Systems Design - ISD

What are the Skills that a College Dean Needs to Have
in order to be a State-of-the-Art Dean?
Click below for an answer....


Group for Discussions on Facebook: "Nada's ESL Island".
Join us there! Post/answer questions

Page Created on Oct. 5, 2010  ||  Last updated on Oct. 12, 2010
   Copyright © 2010 Nada Salem Abisamra
https://www.nadasisland.com/


(Whole Site)