Turning Technology Plans into to Usable Data

This is part of a series about analyzing how the use of Online Learning Platforms was affected by Covid. You can read the first post here.

In my last post, I wrote about finding data to help me answer questions about the use of digital learning platforms in schools. Much of this data led to more general questions about learning and technology use. One of the questions was about the impact that a state’s technology plans had on the use of digital platforms. I wanted to investigate if any particular focuses were correlated with more engagement with Edtech. I also wanted to see if there was any correlation between the content of these plans and student achievement.

For most states, the technology plans I found consisted of documents outlining how they would improve technology use over time. Unfortunately, I could not locate identical information for every state, so I chose to use the most relevant document I could find. When selecting which text to use for each state, I tried to find the most recent one that outlined a plan for improving technology use in the state. If this was not available, I chose the most recent document that included the technology curriculum (as these also tended to talk about what was necessary for students to learn). For Indiana, I used a PowerPoint outlining progress with technology development as this was the only thing that could be found. If this were an official study, I would have contacted each state’s education department. However, in this case, I decided to use what I could find.

After I had obtained the documents, I needed to change them from unstructured to structured data. To do this, I decided to create a spreadsheet that recorded which aspects of learning each state focused on in their documents. The assumption was that if something was included in their technology plan or curriculum, it was likely to be seen as important by the state’s educators and educational administrators. First, I skimmed each item and wrote down keywords linked to features that seemed like they might have an impact on student achievement.

These were the original key words I found:

• flexible
• for_assess
• goals
• home_access
• investment
• one_to_one
• professional_development
• privacy
• problem_solving
• learning_communities
• shared_resources
• personalized_learning
• standards
• student_centered
• tech_Leader
• integration tech_lit
• state_wide_goals
• as_learning_tool

I also included the year created as it indicated how long the state had been working on those particular goals. I suspected that this could impact student use of digital platforms.

Then I used a text rank algorithm with bag of words to find other vital words.

Before using the algorithm, I cleaned the text by removing stop words and all entities. I included terms like district, educational and grade as stop words. These were very common and not likely to be specifically related to a concept that would affect online use. I removed all entities to avoid finding keywords that were names of people or states.

Then I ensured that all of the words to be ranked were at least 30 characters and ranked above 0.003.

Next, I used topic modeling to look for topics in each text.

I chose the most relevant phrases from each topic based on my experience as an educator.

After that, I used Python to check for the keywords in each state’s documents. Their presence, or lack of, in a technology document was indicated by using 1 for the term was present, and 0 for it was not.

I then condensed some of the key phrases into the following concepts:

• 21st_century: talks about preparing students for 21st century/developing 21st century skills
• address_gaps: specific plan to address gaps in school provision of technology
• authentic: recommends using technology in authentic contexts
• collaboration: one goal of technology use is collaboration
• comp_sci_ed: specific focus on computer science skills such as programming
• data_driven: specifically talks about using data to drive/plan instruction
• data_protection: mentions protecting student data
• digital_citizenship: mentions digital citizenship or teaching skills that encourage good digital citizenship
• local_control: encourages or requires schools or districts to develop their own technology use plans
• equity: goal of increasing equity related to technology use
• flexible: mentions the need for flexibility in teaching to allow for the use of technology
• for_assess: encourages/requires technology to be used for assessment
• goals: has specific goals around technology use (including developing infrastructure, pd)
• home_access: goals related to increasing home access to technology
• integration: states use of technology should be integrated into other instruction
• investment: goal of more investment into technology resources or infrastructure
• one_to_one: goal of one device per student
• pd: goal of providing professional development for technology integration or teaching
• personalized_learning: one goal of technology use is to provide more personalized learning experiences
• privacy: mentions the importance of protecting student privacy
• problem_solving: encourages/requires technology to be used to promote problem-solving skills
• learning_communities: encourages/provides for the development of professional learning communities
• shared_resources: goal/provision of state-wide shared resources to promote digital learning
• standards: has or developing technology skills standards
• student_centered: promotes student-centered teaching
• tech_Leader: provides for a technology leader at the school/state or district level
• tech_lit: mentions the importance of developing technology literacy skills
• state_wide_goals: has state-wide technology development goals
• as_learning_tool: primary focus is the use of technology as a tool for learning, not a subject to teach
• year: year document used was created

Finally, I carefully read the documents to see if they talked about the concepts while not directly using any keywords. For example, a teacher would understand all of the phrases in the picture below to mean “authentic learning”, even though those words are very explicitly stated.

I updated the spreadsheet as appropriate. Now that the data was structured, I would finally be able to use it to determine if there was a correlation between technology documents and student learning.