Usage data represents the extent to which a user (e.g., student) participated or engaged with the edtech tool. Examples of usage data (also sometimes referred to as usage metrics) include minutes on system (MOS), days of use, total number of logins, total lessons attempted, number of lessons completed, and number of lessons mastered.

Edtech vendors often allow administrators to export usage data. Depending on the edtech tool that you choose, you may need to contact your district administrator in charge or the edtech company directly to get usage data.

Usage Data Checklist

❏ Is my data user-level?

❏ Does the data include usage from the population of interest?

❏ Does my data have user IDs that match my other files?

❏ Is my data only within 2 weeks of the end of my testing window? (if running Outcomes analysis)

❏ Does my data have at least one clear metric? (ie. total time in minutes, lessons completed)

Notes:

• This data must be available at the user level. District or school-level data is not appropriate for use in an RCE.
• Only one usage metric is needed to run an RCE.
• Our team can help you determine whether a vendor's usage data is appropriate for an RCE.

Student Information Systems (SIS) data consists of student identifiers and demographic data. Example data include student ID, grade level, school, race/ethnicity, gender, special education status, and free/reduced lunch status.. 

RCE uses SIS data in two ways:

• To account for differences in the main analysis: Rapid Cycle Evaluation RCE uses SIS data as a covariate in the main analysis because SIS differences across students, classrooms or schools may influence the study outcome. For example, the comparison group might have more students receiving special education services. By including an indicator for special education services, we can account for this variation between the groups so that it does not influence the results.
• To examine differences in results by subgroups: Rapid Cycle Evaluation RCE allows you to examine overall results and results for specific subgroups. Including SIS data allows you to drill down into usage and performance within SIS subgroups (e.g., grade level, gender, school). For example, did using the edtech product result in better learning outcomes based on grade level?

To gather SIS data, you may need to contact your district's SIS Administrator or Data Manager for access to the data or to include them in the data discovery process.

Outcome Data Checklist

❏ Is my data user-level?

❏ In the population of interest?

❏ Does my data have user IDs that match my other files?

❏ Are my results aligned with my testing window?

❏ Is there a clear column of scores?

Demographic Data Checklist

❏ Is my data user-level?

❏ In the population of interest?

❏ Does my data have user IDs that match my other files?

❏ Does my SIS data contain demographic identifiers of interest?

Note: While SIS data is optional when running an RCE, regardless of your research goals, including as much SIS data as is appropriate can rule out the effects of other variables in order to gain confidence that effects can be attributed to the intervention. Your team can work with LearnPlatform to determine the best use of SIS data.

Outcome data measures educational outcomes that the edtech product claims to impact. To run an outcomes analysis with LearnPlatform (i.e., an analysis that links edtech product use to student learning), our research team will ask for a standardized, reliable, and valid outcome measure. To examine how edtech impacts student learning, we recommend using an outcome measure to represent student achievement.

Examples of outcome data include standardized assessment scores, GPA, or attendance. There are two main types of outcome data:

• Pre-intervention: Performance or scores prior to using the edtech product. We use this data as a covariate in RCEs to account for differences in student performance before the intervention (i.e., edtech product) was available to students.
• Post-intervention: Performance or scores after using the edtech product. We use this data to examine student performance and impact after receiving the intervention.

There are a variety of research questions that an Outcomes Analysis can help your team answer. Two common types of Outcomes Analyses are described below:

• Without a Comparison Group: examine the relationship between edtech product usage and your post-intervention outcome.
• With a Comparison Group: examine whether students who use the edtech product outperform other students, based on post-intervention outcome performance.

To locate outcome data, most assessment systems provide easy access to user-level reporting. You may need to contact your district's assessment director for access to these reports. Your RCE team member can also assist you by providing example data.

Pricing data includes the cost of the edtech product subscription or license, priced per student or site. Pricing data may also include other direct and indirect costs associated with owning the product (e.g., professional development, staff hours). Pricing data can be included in Usage and Outcomes Analyses, but is not required.

Pricing is usually entered into your product library during technical implementation. Alternatively, cost information can be submitted while setting up the RCE if it has not already been shared with LearnPlatform . If you are taking this approach, you may need to contact someone within your organization who has purchasing information for the edtech tool of interest. LearnPlatform can also help you determine the approximate cost per user if that is not clear from the provider’s invoicing.

There are many reasons why data might not be available. Sometimes, data is inaccessible because the vendor did not grant access or is not currently collecting data. Other times, the data is presented in an unusable way. (e.g., an aggregated district count of activities completed vs user-level data.)

If the required data is not available, the research question may not be answerable, which is why it is important to check what data is available as you formulate your research goal. Your RCE team can support you throughout this entire process or provide alternative options to get you the most information possible.