Advancing the Future of Research on Student Learning by Leveraging Standards
Hi, everybody. I'm Ben. Don't call me Benjamin unless I'm in trouble. I'm, assistant professor of departmental sorry, of psychological and brain sciences at Indian University. I'm kinda interested to know who you are. Your weird that you came to the one of the only sessions that has research in the title.
Are any of you researchers yourselves? Way to go. Anybody work with the researchers or need to support them or something? That's cool too. I don't know why anybody else would be here. You're just curious or The start date of time. Awesome.
Okay. I I feel really strongly that, we could be doing more with research. So I, like, I don't know why else other people might be attracted here, but I hope it's useful. I'm a pretty conversational guy, so I'll be giving a presentation, but I'd welcome you to stop me if you have any questions. My plan is basically to talk for maybe twenty or thirty minutes, and then to open it up to Q and A.
And I have stickers. I haven't seen all of you at my little booth, so I'm happy to hand out stickers here if you want. And I have more stuff if you wanna come visit me in the expo I wanna start with an anecdote. Have any of you heard of Percy Spencer before? It's an amazing person. So I'm gonna start by introducing you to Percy Spencer.
That's him right there. He was born in eighteen ninety four. And he was orphaned at age two. So his father died, and his mother put him up basically for adoption. She he was adopted by his aunt and uncle.
He's living with his aunt and uncle from age two. And then at age seven, his uncle dies. So he's orphaned again. And, this time, because he's old enough to work at the ripe old age of seven. And this is, again, around like nineteen hundred.
He gets a job at the spool factory in his town. So this is in rural Ohio. There's a factory where they're making spools out of wood. So these are like giant spools to make like cable and stuff. Works in the spool factory eventually joins the army while he's in the army.
The titanic disaster happens, and he becomes fascinated with radar. That it would have been helpful if there was like a communication method between boats at the time so that people could more people could have been saved becomes fascinated with radar and then ultimately goes and becomes a scientist at Raytheon. So Raytheon is the company that's doing a lot of research on radar during World War II. And what Percy Spencer is working on specifically is, this thing. It's called a magnetron.
I've learned a lot about magnetrons just for building this presentation. It's an amazing little device. It's a device that radiates an electromagnetic field by sending an electron beam over holes that are born in bored into an electromagnet. And it's like an electromagnetic version of a flute. So, like, you know, how you play a flute by blowing air over a flute.
This is exactly the same thing, but with electrons going over bored holes in this magnet. And when you do that, just like a flute makes, you know, sound waves. When you do that with a magnetron, it makes microwaves. So one day Spencers working on the magnetron standing next to it, and he notices that a candy bar that's in his pocket has started to melt. Now I don't know about you.
I don't often carry candy bars in my pocket, but when I do, I imagine they're already at risk of melting just by the fact that they're in your pocket. Right? So anyway, he's, notices that this thing is melting. And it's worth saying everybody who's working on magnetrons at the time knows if you stand close to them, you'll get hot that they radiate heat. So what Percy Spencer didn't do is discover that the magnetrons produce heat. Instead, what he did was he had the idea from the melting chocolate that magnetrons could potentially be used to heat food.
So I really respect Percy Spencer for the next step that he took instead of just being like, okay, we're gonna heat food with this. He decided to do experiments. He's decided to do research. And, so his first research, idea is because he's a Midwestern boy. He takes, popcorns and puts them next to the magnetron and waits and sure enough they pop.
And then, later on, he takes an egg. Have any of you microwaved an egg before? Tell us what happens when you microwave an egg. It explodes. Yeah. Exactly.
And sure enough the egg, this is the first example of a microwaved egg exploding all over his colleagues. It was apparently a great success. And then he decides that he should file a patent. So he files a patent in nineteen forty five, for what's called, at the time, a method of treating food stuffs, and then there's more patents in nineteen forty six, and ultimately it becomes the first microwave made by Raytheon. They call it the radar range.
So why am I telling you about the invention of the microwave oven? I'm trying to make a point that sometimes work in one area can have massive benefits to a totally different application, completely separate from its original intention. So military radars had nothing to do with heating popcorn, but nevertheless military radars led to the development of the microwave. And there's so many more awesome examples of this kind of in the history of entrepreneurship. Have you heard of bubble wrap? Bubble wrap, it was initially invented to be a textured wallpaper. So, like, they thought like, oh, we'll put texture on the wallpaper with the bubbles.
And that was not popular. So they decided, oh, maybe we could use it for packaging material. And then Viagra is a fun example of this too. So viagra was initially developed by Pfizer as a medication to treat cardiovascular diseases. And then during clinical trials, they just kinda governed that it had this effect on male erectile function, that would have been a wild experimental treatment group to be a part of.
Okay. So sometimes you're working on one thing and it turns out it has a benefit to something different. And here we are at instructurecon twenty twenty three. We're celebrating how things like standards and open lead tech ecosystem. LTI Advantage in particular can all expand the range of what's possible with student learning.
So you can create a collaborative ed tech ecosystem with lots of different vendors making a very rich educational technology space. Like here in Canvas, you could just click the blue plus app button and then dramatically instantly unlock new possibilities of what students could do. That's cool. Now again, I'm a faculty member. I'm not an EdTech entrepreneur, but I've tried to listen really closely to how people describe the purpose of what's going on here.
And from the best I can gather, it's that the intended goal is to advance the industry. So give legs to new tools by providing a secure and responsible means for connecting them to students in canvas, and by making this sort of like free market, then maybe the things that are most effective will become things that get adopted broadly. And just like Percy and Percy Spencer in the microwave of, and I want to suggest that I've stumbled on an additional purpose of all the stuff that's been built to make this at Tech ecosystem, specifically advancing research on student learning and student success, basically, by using the standards that are currently there for vended tools. So here's my rough outline of what I wanna do today. The purpose of my talk, while we're talking about intended purposes, is to share with you how work here can have benefits for research on student learning.
Specifically, I'll start by talking about how standards help overcome the challenges of research on student learning and success. The sort of thing that I do, then I'll go on to introduce to terracotta, an open source LTI tool that, I'm presenting here and sponsoring terracotta I'm sorry, sponsoring and structure con with. And then when I'm done, I'll share a little bit about the vision that I have of future educational research infrastructure as enabled by edtech standards, and we can have Q and A. It'll be fun. So far so good? Just stop if you have questions.
Okay. Here's something that you should all know. Education research is a science. It's not just me making this bold claim. This is a consensus view of the American Education Research Association, the US Department of Education, the Science Science Foundation.
And to do science, you really need two different things. You need data, and you need control. And these two things are uniquely challenging in education settings. Like it's as if we pick the two most difficult things in education to want data and control. They're like third rail issues, like so charged that if you start talking with an administrator about, hey, I need data.
They don't even wanna go near them. So everyone agrees I don't think that anybody here would disagree that improving research on student learning is a really critically important thing. So we should base our decisions when we're adopting tools or when we're developing pedagogies On research, we need an evidence base to what we are doing so that we have an a better understanding of what works in practice rather than just our own hypotheses about what should be useful. However, despite the fact that everybody agreeing that this is important, doing research on authentic education settings is just really, really difficult. Sometimes prohibitively difficult.
And it's difficult for lots of reasons. The primary challenge is just getting permission from the school. And for what it's worth, schools have really good reasons to create barriers to people like me just going in and messing with stuff. Right? Like, and they make a lot of barriers that prevent researchers from examining things in practice. Think about it from the school's perspective.
I'm asking for permission to observe, and sometimes not even just to observe to make detailed notes recording private information that's usually protected by by, federal law. So just giving researchers access to observe and record data about students as problematic. Teachers are already overburdened. Is anybody here a teacher? Okay. In a K twelve district too, I have relatives and collaborators who, I collaborate within K twelve.
And yeah, it's funny how they see when they're approached by researchers. It's kind of like a drag like, oh, this is gonna take a while. So it's it's the case that teachers are overburdened. And usually a study will require teachers to do something beyond what they would normally do. And I might be also asking for parental permission talking with parents about what's going on in classrooms right now is hard.
You tend to get lots of really negative reactions. Getting parents involved is like another third rail issue in, education. It's harder than ever before. And finally, if I'm running an intervention study, What I'm usually doing is taking instructional time away from what's normally being done in the classroom, and instructional time is a precious thing. It's especially precious as we're focused on things like standard.
So, yeah, it's possible that students might not have any benefit from an assigned intervention, and that's a problem if we've separated them from what would otherwise be normal instruction. Okay. So there's many issues. And all of them have plagued me in my career. However, somehow the digital transformation of educations just made it possible to break through these barriers.
Making it more open than ever before. So when students are working in the LMS and with standards for data and interoperability, the challenges that I've been talking about so far, they can almost be entirely I wanna tell you a little bit about what I mean by that specifically. Let me give you an example, like the big hard one is information security and data privacy. For example, even though ferpa law contains a clear exception for research on student learning, schools still want to avoid exposing themselves to any sort of legal issues, by sharing any student data, additional concerns related to, like, data ownership and data management, data access also present problems for folks like me who just wanna conduct research on student learning. But vended learning tools have busted a big hole through this wall.
So even though I as a researcher have issues with ferpa and with data ownership and data management and data access, somehow vendors sign papers that just kind of make it through. So we're able to say, okay, we're gonna ignore ferpa in this case. You're being treated as a school official. So FERC is a non issue. And that's something that I, as a researcher, would love to capitalize on.
Again, this process that we have for being able to get through issues like information security and data privacy with vendors. So I like to, give a case in point for this in saying that in twenty eighteen, I ran a crazy study that we called many classes I don't know if any of you heard about it, but many classes was an effort to run an experiment, not just in one class, but in many classes so that we could explore what the boundary conditions are of an experimental intervention. So this was like dozens of classes ranging in ages and, institutions. And we ran the same experiment across multiple institutions. So we had to deal with different institutions, processes for getting through this brick wall.
At one university whose name I won't mention, and they're not even here, so it really doesn't matter. They had no process for sharing student data. It wasn't like there was a research data manager, data Stewart involved, until their chief privacy officer, it made it all the way up to their CPO recommended, why don't you just behave like vendors? So the research team registered as a vendor, like we actually like vendor paperwork. We called ourselves a vendor. We got access to their purchasing platform through the vendor thing.
We just submitted an invoice for zero dollars Like, okay. We'll do the vendor thing. And only then once we did all those steps, did they feel comfortable sharing data with us? So was as if, like, we took off our researchers' hats and put on the vendor hat, and then all of a sudden, the challenges that have been facing education science for a while just kind of magically disappeared. So it's kind of cool that we've built processes around things that are otherwise barriers. So vended tools have busted through the data wall, the other requirement of scientific research is control.
And Vented Tools have also managed to negotiate new models of how control and delivery happen in education settings. It wasn't so long ago that teachers had complete control over the full student experience. Like it was just the case is a chalkboard, and the person who writes on the chalkboard is in control of what happens. So for that reason, if somebody like me wanted to gain control, I would have needed to gain control of the chalkboard. So a common way to run an experiment in these old days was to get a bunch of teachers together, random sample of teachers into a training program, and instruct them on some new things, some way of changing things from normal practice.
So, like, we'll train teachers on a new way of delivering biology education or something. And then they go off, and we hope that they teach students in the way that they were trained. And we have really no way of controlling or knowing whether they did the thing that they were supposed to In fact, there's this whole, subfield in education research called Fidelity of implementation. I don't know if you've heard of that. It's a fallout of the fact that when you'd run an intervention study, because teachers were the ones delivering the intervention, you had no idea whether they implemented it the way that you would have wanted them to do.
But technology has fixed all this. It's made it totally possible for teachers to share control, not to relinquish control, but to share control with, technology tools with, like, services and providers. So you can assign students to work in a learning management system in a platform that has control over what the questions are and what the things are. And it's delivered entirely online without the teacher needing to do much. So here, the teacher might behave more like a guide, like selecting what it is that the student might get assigned to And there are amazing tools that give students new opportunities to do things that would have been well outside of what a teacher would have had in their normal toolkits, and that would have been in just totally impossible fifteen years ago.
So what I'm advocating for here is that we can also be building research services using the tools and online platforms the teachers are already sharing control with. So either within these tools or is entirely new research focused tools. So we can literally build controlled experiments inside learning experiences either at the student level or the class level, taking out all the hassle and guesswork of whether the teacher is doing the thing that we thought that this teacher would do. So literally, we can design like big robust large scale studies that randomly assign students to different treatments so that we're not just guessing about whether they're, actually making an impact in education outcomes. And we're also rigorously evaluating their benefits.
So that's the background. I think that there's a lot of opportunity using like standards and vendor processes to be able to do better research. So now I wanna show you an example of what that might look like, specifically with terracotta. So terracotta is an open source LTI tool, that turns an entire LMS course site into a really cool research laboratory. I'm kind of switching the background, so you know, this is what terracotta looks like.
It's a we call it a portmanteau or maybe an acronym or maybe an acromantau for, tool for education research with randomized control trials. I'll start out by saying it's totally free. We're supported by a variety of funders, prominently the Institute of Education Sciences, IS is the nonpartisan research arm of the US Department of Education. We've also gotten generous funding from Schmidt futures, and all of our work is in collaboration with instructure. So the grant from IES is actually in collaboration with Instructure.
And what it is is it's an experiment builder that just sits within an LMS course site. So currently Taricata's primary feature is to be able to experimentally manipulate, assignments. So you can make two different versions of an assignment, version a in version b, and then say you want half the class randomly to get version a and half the class randomly get version b, and then you'll be able to compare outcomes between a and b on downstream course assignments and assessments. And that's a simple example, like just doing AB testing on an assignment is maybe the simplest thing you could possibly do. What does way more complicated things? And in a moment, I'll walk you through some of the additional things that it does.
Okay. And, it's not a startup. Again, it's a free service. If you were interested in adopting terracotta, you'd actually be adopting it from Indian University. And we provide hosting for anybody who wants to collaborate on research.
We're always looking for new partners. If you have any interest to partner and collaborate, then please like, let me know. It would be amazing. So, let's walk you through a little bit of what goes into it. If you click that blue button, create your first experiment, then, you're taken to an interactive guide that slowly walks you through the process of designing an experiment.
One of the first screens that you'll see in that guide asks you to identify or name the conditions in your experiment. Any condition I'm sorry, any experiment involves multiple conditions that people or subjects might be exposed to. So here you just said, okay, label them. And we can have AB tests if you just wanted to compare between two different conditions, but we can also support up to sixteen different conditions. So a two by two by two by two by two, that you can really test not just, is this better than that, but why something might be better.
So you could do actually, like, multiple control conditions. And then terracotta will later on randomly assign students to whatever conditions you include here. So what are the different conditions? Like, what are the things that somebody could do in terracotta? It's a pretty robust feature set. So, you build assignments in terracotta, and then you randomly assign which assignments or which versions of assignments students will experience. And these assignments could have multiple question formats, so it could be multiple choice, and they could be randomized or multiple correct.
You could also have short answer and file upload. You also can, adjust the display format. So for example, if you were interested to do a research study on whether presenting questions one at a time, or all in one block. That's something that you could just implement natively in terracotta. In fact, all of these can be differentiated between treatments.
Also submission option options. So if you wanna have some students only submit once, but other students can submit multiple times. Question content, we also support YouTube media. So if you wanted to have students watch videos for an assignment. That's something you could totally do.
And also, it's worth saying that all these things, all these interactions, we have robust data collection four. And YouTube media, I'll call out in particular. So we've instrumented terracotta with the, Caliper media profile. So anytime a student clicks in that media player, you're able to see what, they did. And you could even reconstruct exactly what portions of a video students have watched.
It also does interesting stuff with feedback, so you could have auto graded. If you have auto grading, it can take the most recent or the highest or the average, grade. We also implemented this really cool thing that I've never heard of before that turns out to be super efficacious. Cumulative grading, it doesn't happen in canvas. So you can tell students I want you to do this quiz four times.
And every time you do the quiz, you'll get an additional twenty five percent of your grade from that quiz. So it could be like that you give them full credit and they just have to do it multiple to give themselves practice. And if you've got that set up, you can also schedule it so that they have to have like a day in between each attempt. So we've really tried to make terracotta as rich with the sorts of things that you could put conceivably want to do research on. And you know, given this every possible combination, there's probably like thousands of different types of experiments you could do just on formatting alone, ignoring content.
At this point in presentations about terracotta, you might be thinking something like, how is this ethical? Like, what the heck You're gonna take students and give some of them, like, immediate feedback, and other students get delayed feedback, and you think that that's okay. And the answer is, Yes. But I acknowledge that it's, sensitive. So I'll say first that we we use and we strongly encourage anybody who's using terracotta to use the informed consent feature so that students who are about to be participants in experiment agree to be a participant next that experiment. So, the way that this works in terracotta is that we actually create an assignment that is an informed consent assignment.
It can have a due date. You can give students credit for responding to the informed consent assignment. There's no right answer, so they'll get credit whether they say yes or no. But this is the fundamental branch that determines whether students are randomly assigned. If they do agree to to participate, or if they don't agree to participate, they'll get like a business as usual default condition, and we won't do any data collection on that what you were gonna ask? Yeah.
Yeah. So, it's worth saying that if somebody says no, I don't want to participate in the experiment, it would be crappy if they just didn't get any educational experience. Everybody gets an educational experience, whether they agree or don't agree. It's just if they don't agree, they're not randomly assigned. They get default.
Whatever the teacher says is default. Any questions about informed consent? It's this is also, an interesting feature of terracotta that Unlike everything else in Canvas, the teacher can't see how students respond. So we also make it so that informed consent responses specifically are never exposed to anybody so that there's no risk of like a teacher, I don't know, in unfairly incentivizing students to participate in a research study. If you didn't wanna do informed consent, like, for example, if your your principal said, don't even worry about it. You can do the experiment on everybody, then that's totally okay.
We just have like a few warning screens. Like, are you you want to do that, before that's something that gets implemented. Okay. Simple study would be like compare a to b, but, more interesting study from my perspective would be to do within subject crossover design. So one other concern about experimentation is like maybe a is better than b.
What would happen with the students who get b? Like, does that mean that they're just kind of out of luck they get the worst educational experience? And one way to get around that is to say all students will get all treatments. So either a student will get a then b or a student would get B, then A. That's called the within subject crossover design. And in that situation, everybody gets everything. There's no bias due to experimental treatment.
It's just a matter of when they experience it. So you can measure student outcomes at the student level when they got a, and then when they got b, and then you can pick compare not between students, but within students What an individual student's experience of a, how how that changed things compared to b. So it's also really good for, like, statistical power and stuff. So terracotta does crossover designs easily. And, it's also the case that you don't need to have just one crossover.
You can go back and forth and back and forth back and forth throughout the whole semester. So you're able to do some really cool things with, terracotta as far as, like, scheduling treatments. Okay. Also, a lot of the details of what students are doing in their classes are not gonna be experimentally manipulated, I'm talking about a tool that makes it possible to randomly assign students to different treatments, but downstream, like after you do your experiment, you're probably interested in how that affects I don't know their performance on the next test or on the midterm or the final or this, you know, the statewide assessment. And what you can do in terracotta is you can link the experiment that you've created to some sort of other outcome that's already in the Canvas Gradebook, or if you just have some other data that's not in the data in the Gradebook, but that you want to include, like I don't know, students' attendance or something.
You can just manually add those, those numbers to the roster just the same. So the idea is that we're allowing teachers and researchers to have the scope of their experimentation be beyond just what they're manipulating in terracotta. You can also look to see how it affects stream outcomes that are authentic to whatever the teacher's really interested to improve. Yeah. So you just select it from the grade book.
Okay. And, like this is my favorite and also least favorite screenshot. Imagine yourself like scrolling up on this screen. So this is the experiment status screen. This is what you'd see while you're running an experiment.
At the very top of this page, there's this blue button that says export data. And imagine yourself clicking the export data button, and it'll download zip archive that contains a variety of CSV files that have all the data from your experiment that you could ever want. It includes like what the conditions were, what the assignments were, the questions in the assignments, the participants, when they provided consent, it's all rigorously de identified. So no student name will ever appear in this And it also makes it possible for you to do analyses on the experiment that you ran without needing to make a request of like the Canvas data admins or something. So you have control over the experimental research data.
And, yeah, you as a teacher, because you're already being exposed to, like, what students grades are. There's no, like, new exposure of, outcomes that, you know, you wouldn't otherwise be able to see it. It just makes it so that you could analyze research data without worrying about identities. Yeah. We've got a robust data dictionary.
It describes all the details of things. And also, I've provided some, like, short analysis scripts to help people if they're just interested in getting started with this format. We had to we had to invent, format. So there's no, like, common or standard for how you represent an educational experiment in data form. So, yeah, we did our best to put it together, and we also have a mapping to SEDs interested in like comparing your outcomes with what goes on in common educational data standards.
Okay. That's what terracotta does. I also wanna be very clear about what terracotta doesn't do. So first, terracotta doesn't intrude. It's not the case that I am going to run an experiment in your school.
So researchers would still need to negotiate with teachers and schools and get study approval. It's like set up like an LMS is normally set up where the teacher is the one who needs to create the stuff. So if I wanted to run an experiment in somebody's class, I talk with them how to create this stuff. And I work with them, and I might have like a Zoom meeting where we actually build it together. So it's not the case that anybody's running experiments from behind the curtain wizard of Oz style.
Also, it doesn't provide any access to private data. So, terracotta is not a company. We don't go and use the, We don't go and use like our access privileges to go and scrape data from outside of terracotta about what students are doing in Canvas. We also don't provide anybody with backend access the terracotta database. So it's not like we're gonna be doing research on the research that you're doing.
You can you maintain ownership of all of your data as kind of the institutions and teachers and really our our ideas to get the data to you. If you're an institution though, so like if you're an LMS admin, We do allow you to subscribe to an identifiable Caliper event service so that you can see what's going on in terracotta so that you know working, and you can verify the integrity of your academic assessments. So yeah, that's the rundown of terracotta. I wanna now kind of shift to mentioning what I think about when when it comes to the future of education research. And really, I I think Tericot is amazing.
It's made my wildest dreams come true. I've done some amazingly rapid rigorous and responsible large scale experimentation. We're, because everybody's talking about AI, I should mention that, like, we were able to whip up an amazing experiment that we're gonna be deploying in the fall, in all of the elementary composition courses at IU. So, one thing that you could do with chat GPT is you could give it an a student work and then have it give feedback. So we're gonna compare, when chat CPT provides feedback to peer feedback.
So in this elementary composition, students already do peer feedback, and rather than just having them type the feedback to their peers, we'll have them process their peers work through chat JPT, get the feedback, before delivering it directly, they'll, like, look at it, reflect on whether it's the feedback that they would give, and then they'll deliver that chat Jpt feedback. So we'll be able to see whether the efficacy feedback from Chachapiti is any better or worse than when peers actually do it to each other. Yeah. We're excited about that, and we're gonna be do doing it. Again, at a very large scale, so we're talking about like eight thousand students just in the fall.
Okay. In spite of how excited I am about it and all the cool things we could do, I acknowledge that Terracot is just one tool. So, it does some cool things enabling one particular kind of experimental research, but that's not the full range of all the research that people could possibly do. And it's my guess that of the people who raise their hands when I asked who does research, that some of the things that you're doing is very different from what currently is being done with terracotta. Okay.
So education in general, but education technologies, specifically and in particular, have just chronically under invested in research and development. It's the case that we're much quicker to kind of like run to market than to evaluate whether the thing actually works. So if we're gonna rely on research for making evidence based decisions in education, we just need much more expansive research infrastructure. We need to make it easier for vended tools and for teachers and for researchers like me to be able to do stuff, like evaluate what works, it doesn't make sense to claim that we're evidence based if people like me need to, like, struggle for years to be able to conduct a single research study. So I have a vision that someday terracotta might be one tool in something like a national research infrastructure toolkit, which would include additional for doing additional things like manipulating personalized messages that might be deployed by some AI system or communicating with parents or doing survey data collection or any of the range of other things that might be possible.
So such a toolkit would need standards for implementation and interoperability, and that's really why I'm here to advocate that in the same sense that LTI and Caliper analytics and all these things have enabled an ed tech ecosystem that thrives. It'd be cool if we also used those processes and standards to be able to make it so that other research services might become integrated with our educational systems. So, the good news about this is not just me who has this vision. There are other people who have this vision too. And one of those, organizations is the National Science Foundation.
So I'm, excited to share the news that we've recently received some super generous funding from the National Science Foundation to imagine what this toolkit might look like So if you've got a toolbox, terracotta might be one thing. What are the other things that should be in it? So with colleagues at IU, we formed what we call the interact incubator. It's a collaboration between some of the most amazing brightest, education researchers in the country and also cognitive psychologists and social psychologists and technology stakeholders. Also district administrators and policy makers, we have the actual commissioner for higher education from the state of Texas involved. And our focus is really on improving equity in STEM education, specifically in K twelve.
And over the next two years, we're gonna be convening multiple times to conduct first a needs assessment. So let's just imagine what we would like to do if there were no barriers to research at all. And then we'll imagine what that infrastructure would look like. So we'll actually do a design process. So what do we need in terms of data and control? And the idea is that once we're done with that, we're gonna be submitting a giant figure grant proposal to NSF, just to make it a reality.
So if you're interested in any of the things that I've talked about, if you're like, oh, Terracat is cool, but I would really like to do something else, And especially if you're in a K twelve district, then we would really love to include you. We have applications to expand the incubator from what we currently have, and the applications are doing like four days. So, if you're interested, then please let me know. It's not a hard application. You just like write down your name and give us your resume.
Okay. So to bring it totally full circle, I want you to see me like a melting candy bar in Percy Spencer's pocket. So we're all trying to build an educational technology marketplace with tools and providers all constructively working together. And here I'm saying, Hey, you're also totally solving a big challenge that's plaguing education researchers for generations. So if you're interested, I'd be excited to to chat about how we can melt this other forms of food, and also to build the next generation of research infrastructure.
Thank you so much. Yeah. I'm interested in questions. Yeah, please. I have two two questions.
Okay. First, as a utility, as a surface. In the University of Wisconsin System, we have a rather rigorous intake process the learning tools. You know, you describe it. You would describe being a vendor and kind of having an open door in your end.
That's the exact opposite how it is, our our intake process to be a brick wall at the vendor chooses to not do something in the whole process because we met them very thoroughly. I was wondering about the challenges, if you've had any of those challenges, where the gaps are. The second one is more of a comment, and that is that the holy rail in the LMS right now is being able to use all the tools that Canvas offers while being largely anonymous. Because even if you take off the people and hide other things, the inbox is still there. So I'm just going to leave that with you as here's here's the pie in this guy.
Yeah. This is where our IRP group wants us to be able to do things that we just can't. So, okay. Those are good questions. The first I'll start with the first question, which was something like Okay.
Is the vended process for gaining access to a provider actually so simple? And you'll just have to take my word for it that despite the fact that, yeah, administrators and info and audits make things harder, it's still easier than if you just make a research data request. The other thing that's unique about the vented tool process is that it's process. So you can say, I need you from the perspective of Wisconsin. I need you to have a heck fat. I need you to have, vulnerability scanning certificate.
I need you to do this and this and this And that's a set of criteria that a researcher or a tool like terracotta could muster. It might take us a little bit, but once we've got that, then the process is relatively lubricated. So we've worked through much of this word, like, a week away from our vulnerability scanning certificate. So, yeah, it's the, it's the case that there's at least a transparent process. So it might not be like super smooth, but at least it's something that somebody could navigate.
Okay. The second thing about like an anonymity, I'm actually excited about the idea of anonymity, at least from the perspective of the student. What I would like is there to be nevertheless a lot of protections for what that, what that identifiable data are. I try to treat it, treat it as, like, rigorously as sensitive as possible, and as I can in terracotta. So the, the data that is identifiable is in separate tables that have tighter security protocols than the other data that's in terracotta.
Yeah. So we've tried to architect things so that we're being as cautious and as sparing with identifiable data as we can. I think that I think that you would share the idea that even if we've got this pie in the sky vision of anonymity, we still want there to be some identifiable integrity. Right? And so administrators should probably have that. And I'd like to believe that that is also potentially useful for understanding what works, like we should be able to assess our practices.
If not, then it's like why are we collecting the data in the first place? So, yeah, I'd I'd like to think that research could be one of those privileged few that should be able to play. Maybe not with, identifiable stuff though. Yeah. Yeah. So you mentioned you don't do anything with the data you collect.
Is there any options to just not collect like opt out and have it sent to the institution, like, choose a data store online institution. Totally. Yeah. I I'd it was something I glossed over super fast. You don't need to involve me at all.
Terracat is totally open source. So, We have instructions on our GitHub page if you wanted to spin up an instance and host it yourself on your own infrastructure. I'm not a developer, so I don't know how difficult it would be to, like, pull the branches down as things get updated on our side. But if if the fact that IU is hosting it or that it's being hosted on AWS as a problem or something, then I would be excited to explore the possibility of trying to help you do your own thing. Yeah.
We haven't done that yet. It's it turns out that people are much more happy to just let us take care of But if it's something that's sensitive and you'd like to do it yourself, that would be that'd be awesome. Yeah. Yes. Okay.
So our, we just finished a two year wacky research competition. I'll take it two steps back and tell you why it's wacky. So, normally when people like me do research studies, will, like, write a a grant, and somebody will review it, and then maybe they'll fund it, and then we'll do the project. For various reasons, funders are interested in shifting the risk to the researchers. So instead of giving them funding to do a research study and then they do it, they'll have them do the research study and then judge whether it's good and then give them funding after the fact as like a competition.
It's like a big grown up science fair with extremely high stakes. So, we were participating in the X Prize Digital Learning Challenge, and thankfully, we came out in second place, so we got the payout in the end. The study was awesome. It was, very large scale effectiveness study of pre questions? Have you heard of pre questions before? It's okay. So normally in education, you teach students something and then they answer questions about thing.
It goes like learning and then you assess. But for over a hundred years, there's been this hypothesis that maybe people would actually learn better if you ask the questions ahead of time. And we're starting to see some preliminary evidence of this, not just from like psychology research, but also from practical research when students are doing pre tests because teachers need pre test assessments to show lift in their classes. So, anyway, this was a very large scale, assessment of, like, what happens when you ask questions before students watch videos compared with asking it afterward. And we found sure enough that there's an amazing improvement in learning when the questions are asked before.
And it's weird from the student's perspective. I mean, imagine like sitting down to an assignment And before you've learned anything, you get questions on this stuff that you don't know the answer to. So for the most part, they're guessing, and most of them get it wrong, but you give them credit just for the fact that they answer And you find that that improves learning downstream better than asking the questions directly after the the video. So it's like a five percent improvement. It's like totally, practically significant, which is crazy to to consider again that they don't know the answers when you're asking the questions.
So that's not as sexy as chattyPT, but yeah, we'll be doing the chattyPT thing in the fall excited about that too. Please reach out to me. Yeah. So, it's it's me and a project manager, and we use, Unicon for development services. So we're a relatively small team.
That's why we're free. But yeah, yeah, we'd be excited to walk through the process with you. Yeah. Yeah. And also, Everybody, like, later tonight, come to my little little exhibit booth.
I I have chapstick. I have like so much chapstick. And it's a good chapstick. I just got, like, a fifteen hundred things of chapstick, and I have not given away fifteen. I don't wanna bring it all home.
Yeah, you should not need to buy chapstick for five years. You're serious. I'm so sorry. Yep. Okay.
In the back first. Yep. Yeah. I'm I'm one of the people that doesn't know why I'm here, really. Way to go.
Educational research at our university. And I thought it was more about the school and family. So I was really a little inappropriate question, but, like, I know whenever he's trying to do something. He just cut off from IRP. Yeah.
It's a good question. Like, what's our It's a great question. Every study I've done with terracotta so far. So, sorry. We've been like out of alpha, so we've actually been testing it for about a year.
So I've done maybe like four or five studies. Every study that we've done is with IRB approval. And I'm a big proponent of Open Science. I make all of my IRB protocols public, So I'd be happy to share with you. If you just want to see what an IRB protocol looks like with terracotta.
And the good news is that it's very structured, and everything is, like, kept What's the right way to say it is that I've anticipated what IRBs would mind. So I've made it so that this addresses their concern. So I'm happy to share the templates with you if you're interested. Yeah. This is not a way of circumstance.
Totally the opposite. It's a way of making IRB happy. Yeah. Yeah. Exactly.
Yeah. Yeah. So, every step of terracotta in a sense is exactly what the IRB would want. So it describes how, informed consent is hidden from the teacher so that nobody ever sees student's consent responses. It's just managed automatically, which is a major improvement over any manual study.
And, yeah, the fact that the data are de identified from the source is also a big deal. There's never any need for anybody to see anybody's identities. You could just imagine they're just being random codes to identify students So that and a variety of other things make it super easy. Yep. Yeah.
That's a good question. Okay. And do you had a question? Same. About the use of higher education. I'm in a learning analytics working group.
And, so to actually get some more more observational than And, are there other uses applications for total. Yeah. So I've presented it here as being like an experimental research platform, but that's because that's how I envisioned it. And the more that I talk about it with other people, the more I discover, like, oh, something new that somebody could do with it. So one of the things is just to do survey research.
So normally, if you wanted to run a survey, you're forced to go off and use qualtrics you could use Canvas' survey tool, and in either situation connecting those survey responses to student performance in Canvas, is really hard. It involves like knowing students identities and then mapping them. So terracotta solves that problem. If you're ever interested in a survey and also to have students grades, somehow connected to survey responses, it does that. So you could do anything in learning analytics that would otherwise be associated with a survey, like don't know, self determination theory or some sort of self regulation, self self regulated learning inventory.
I'm closely involved with learning analytics too. I'm I'm one of the leaders of the method sig for the society for learning analytics research. And I always imagined that, like, learning analytics is continuing its growth and that eventually it'll shift from being purely observational to starting to do things to nudge students in the right direction. And when that happens, I imagine that experimental research opportunities are gonna like, I don't know, what people start talking about and learning analytic spheres. And that'll be a good thing so that we actually start, like, feeding forward and closing the loop.
Yeah. Yeah. It's a good question. Yeah. One question.
I hope I didn't miss this part, but when the study has completed or that initial data is there, Is there any kind of, high level dashboard presentation, or is it all statistical analysis to determine results? I love that question. Okay. So I have a collaborator. So Mark McDaniel, he's a professor at Washington St. Louis, And, this has been like a debate between us for a long time.
Like, okay, so I'm giving people data because I'm a fiend for raw day I love data, but Mark has always been saying, like, you need to have some sort of, like, high level summary so that people could just see what went on. So, We've been designing it and going back and forth with Mark and going back. And it was only just like two weeks ago that we finalized our non functioning prototype for a results dashboard. And it'll be built probably by the end of this calendar year. So by the time anybody here is adopting it and completing an experiment, there should be a nice, results dashboard.
That'll do two things. It'll just kind of show you the numbers. So, like, you'll be able to see, like, this is the number of students that completed the informed consent assignment. This is the number of submissions you had to every assignment. This is the grades on those assignments, but then beneath that, it would actually show you the performance contrasted between conditions.
We would have an actual figure that shows dots for every individual student submission. And you could select different outcomes that might have been included in the experiment. And then, also, it would have, like, the descriptive statistics. So you could take those and do statistics if you wanted to, like, if you but it'll show you the means and standard deviations so that you know what what the differences are and what the scale is. Yeah.
Yeah. So how how soon will you have an AI generated paragraph copy and paste into that main screen. Not yet. That's a good question. And that's exactly the debate that I've been having with Mark is like, How how much do we want to spoon feed it to people? So, we made this strategic decision that we're not gonna do statistics.
So, that gets complicated. Like we can't just know that the statistical assumptions are met. So, yeah, that's a, that's a good place to draw a line. Yep.
Are any of you researchers yourselves? Way to go. Anybody work with the researchers or need to support them or something? That's cool too. I don't know why anybody else would be here. You're just curious or The start date of time. Awesome.
Okay. I I feel really strongly that, we could be doing more with research. So I, like, I don't know why else other people might be attracted here, but I hope it's useful. I'm a pretty conversational guy, so I'll be giving a presentation, but I'd welcome you to stop me if you have any questions. My plan is basically to talk for maybe twenty or thirty minutes, and then to open it up to Q and A.
And I have stickers. I haven't seen all of you at my little booth, so I'm happy to hand out stickers here if you want. And I have more stuff if you wanna come visit me in the expo I wanna start with an anecdote. Have any of you heard of Percy Spencer before? It's an amazing person. So I'm gonna start by introducing you to Percy Spencer.
That's him right there. He was born in eighteen ninety four. And he was orphaned at age two. So his father died, and his mother put him up basically for adoption. She he was adopted by his aunt and uncle.
He's living with his aunt and uncle from age two. And then at age seven, his uncle dies. So he's orphaned again. And, this time, because he's old enough to work at the ripe old age of seven. And this is, again, around like nineteen hundred.
He gets a job at the spool factory in his town. So this is in rural Ohio. There's a factory where they're making spools out of wood. So these are like giant spools to make like cable and stuff. Works in the spool factory eventually joins the army while he's in the army.
The titanic disaster happens, and he becomes fascinated with radar. That it would have been helpful if there was like a communication method between boats at the time so that people could more people could have been saved becomes fascinated with radar and then ultimately goes and becomes a scientist at Raytheon. So Raytheon is the company that's doing a lot of research on radar during World War II. And what Percy Spencer is working on specifically is, this thing. It's called a magnetron.
I've learned a lot about magnetrons just for building this presentation. It's an amazing little device. It's a device that radiates an electromagnetic field by sending an electron beam over holes that are born in bored into an electromagnet. And it's like an electromagnetic version of a flute. So, like, you know, how you play a flute by blowing air over a flute.
This is exactly the same thing, but with electrons going over bored holes in this magnet. And when you do that, just like a flute makes, you know, sound waves. When you do that with a magnetron, it makes microwaves. So one day Spencers working on the magnetron standing next to it, and he notices that a candy bar that's in his pocket has started to melt. Now I don't know about you.
I don't often carry candy bars in my pocket, but when I do, I imagine they're already at risk of melting just by the fact that they're in your pocket. Right? So anyway, he's, notices that this thing is melting. And it's worth saying everybody who's working on magnetrons at the time knows if you stand close to them, you'll get hot that they radiate heat. So what Percy Spencer didn't do is discover that the magnetrons produce heat. Instead, what he did was he had the idea from the melting chocolate that magnetrons could potentially be used to heat food.
So I really respect Percy Spencer for the next step that he took instead of just being like, okay, we're gonna heat food with this. He decided to do experiments. He's decided to do research. And, so his first research, idea is because he's a Midwestern boy. He takes, popcorns and puts them next to the magnetron and waits and sure enough they pop.
And then, later on, he takes an egg. Have any of you microwaved an egg before? Tell us what happens when you microwave an egg. It explodes. Yeah. Exactly.
And sure enough the egg, this is the first example of a microwaved egg exploding all over his colleagues. It was apparently a great success. And then he decides that he should file a patent. So he files a patent in nineteen forty five, for what's called, at the time, a method of treating food stuffs, and then there's more patents in nineteen forty six, and ultimately it becomes the first microwave made by Raytheon. They call it the radar range.
So why am I telling you about the invention of the microwave oven? I'm trying to make a point that sometimes work in one area can have massive benefits to a totally different application, completely separate from its original intention. So military radars had nothing to do with heating popcorn, but nevertheless military radars led to the development of the microwave. And there's so many more awesome examples of this kind of in the history of entrepreneurship. Have you heard of bubble wrap? Bubble wrap, it was initially invented to be a textured wallpaper. So, like, they thought like, oh, we'll put texture on the wallpaper with the bubbles.
And that was not popular. So they decided, oh, maybe we could use it for packaging material. And then Viagra is a fun example of this too. So viagra was initially developed by Pfizer as a medication to treat cardiovascular diseases. And then during clinical trials, they just kinda governed that it had this effect on male erectile function, that would have been a wild experimental treatment group to be a part of.
Okay. So sometimes you're working on one thing and it turns out it has a benefit to something different. And here we are at instructurecon twenty twenty three. We're celebrating how things like standards and open lead tech ecosystem. LTI Advantage in particular can all expand the range of what's possible with student learning.
So you can create a collaborative ed tech ecosystem with lots of different vendors making a very rich educational technology space. Like here in Canvas, you could just click the blue plus app button and then dramatically instantly unlock new possibilities of what students could do. That's cool. Now again, I'm a faculty member. I'm not an EdTech entrepreneur, but I've tried to listen really closely to how people describe the purpose of what's going on here.
And from the best I can gather, it's that the intended goal is to advance the industry. So give legs to new tools by providing a secure and responsible means for connecting them to students in canvas, and by making this sort of like free market, then maybe the things that are most effective will become things that get adopted broadly. And just like Percy and Percy Spencer in the microwave of, and I want to suggest that I've stumbled on an additional purpose of all the stuff that's been built to make this at Tech ecosystem, specifically advancing research on student learning and student success, basically, by using the standards that are currently there for vended tools. So here's my rough outline of what I wanna do today. The purpose of my talk, while we're talking about intended purposes, is to share with you how work here can have benefits for research on student learning.
Specifically, I'll start by talking about how standards help overcome the challenges of research on student learning and success. The sort of thing that I do, then I'll go on to introduce to terracotta, an open source LTI tool that, I'm presenting here and sponsoring terracotta I'm sorry, sponsoring and structure con with. And then when I'm done, I'll share a little bit about the vision that I have of future educational research infrastructure as enabled by edtech standards, and we can have Q and A. It'll be fun. So far so good? Just stop if you have questions.
Okay. Here's something that you should all know. Education research is a science. It's not just me making this bold claim. This is a consensus view of the American Education Research Association, the US Department of Education, the Science Science Foundation.
And to do science, you really need two different things. You need data, and you need control. And these two things are uniquely challenging in education settings. Like it's as if we pick the two most difficult things in education to want data and control. They're like third rail issues, like so charged that if you start talking with an administrator about, hey, I need data.
They don't even wanna go near them. So everyone agrees I don't think that anybody here would disagree that improving research on student learning is a really critically important thing. So we should base our decisions when we're adopting tools or when we're developing pedagogies On research, we need an evidence base to what we are doing so that we have an a better understanding of what works in practice rather than just our own hypotheses about what should be useful. However, despite the fact that everybody agreeing that this is important, doing research on authentic education settings is just really, really difficult. Sometimes prohibitively difficult.
And it's difficult for lots of reasons. The primary challenge is just getting permission from the school. And for what it's worth, schools have really good reasons to create barriers to people like me just going in and messing with stuff. Right? Like, and they make a lot of barriers that prevent researchers from examining things in practice. Think about it from the school's perspective.
I'm asking for permission to observe, and sometimes not even just to observe to make detailed notes recording private information that's usually protected by by, federal law. So just giving researchers access to observe and record data about students as problematic. Teachers are already overburdened. Is anybody here a teacher? Okay. In a K twelve district too, I have relatives and collaborators who, I collaborate within K twelve.
And yeah, it's funny how they see when they're approached by researchers. It's kind of like a drag like, oh, this is gonna take a while. So it's it's the case that teachers are overburdened. And usually a study will require teachers to do something beyond what they would normally do. And I might be also asking for parental permission talking with parents about what's going on in classrooms right now is hard.
You tend to get lots of really negative reactions. Getting parents involved is like another third rail issue in, education. It's harder than ever before. And finally, if I'm running an intervention study, What I'm usually doing is taking instructional time away from what's normally being done in the classroom, and instructional time is a precious thing. It's especially precious as we're focused on things like standard.
So, yeah, it's possible that students might not have any benefit from an assigned intervention, and that's a problem if we've separated them from what would otherwise be normal instruction. Okay. So there's many issues. And all of them have plagued me in my career. However, somehow the digital transformation of educations just made it possible to break through these barriers.
Making it more open than ever before. So when students are working in the LMS and with standards for data and interoperability, the challenges that I've been talking about so far, they can almost be entirely I wanna tell you a little bit about what I mean by that specifically. Let me give you an example, like the big hard one is information security and data privacy. For example, even though ferpa law contains a clear exception for research on student learning, schools still want to avoid exposing themselves to any sort of legal issues, by sharing any student data, additional concerns related to, like, data ownership and data management, data access also present problems for folks like me who just wanna conduct research on student learning. But vended learning tools have busted a big hole through this wall.
So even though I as a researcher have issues with ferpa and with data ownership and data management and data access, somehow vendors sign papers that just kind of make it through. So we're able to say, okay, we're gonna ignore ferpa in this case. You're being treated as a school official. So FERC is a non issue. And that's something that I, as a researcher, would love to capitalize on.
Again, this process that we have for being able to get through issues like information security and data privacy with vendors. So I like to, give a case in point for this in saying that in twenty eighteen, I ran a crazy study that we called many classes I don't know if any of you heard about it, but many classes was an effort to run an experiment, not just in one class, but in many classes so that we could explore what the boundary conditions are of an experimental intervention. So this was like dozens of classes ranging in ages and, institutions. And we ran the same experiment across multiple institutions. So we had to deal with different institutions, processes for getting through this brick wall.
At one university whose name I won't mention, and they're not even here, so it really doesn't matter. They had no process for sharing student data. It wasn't like there was a research data manager, data Stewart involved, until their chief privacy officer, it made it all the way up to their CPO recommended, why don't you just behave like vendors? So the research team registered as a vendor, like we actually like vendor paperwork. We called ourselves a vendor. We got access to their purchasing platform through the vendor thing.
We just submitted an invoice for zero dollars Like, okay. We'll do the vendor thing. And only then once we did all those steps, did they feel comfortable sharing data with us? So was as if, like, we took off our researchers' hats and put on the vendor hat, and then all of a sudden, the challenges that have been facing education science for a while just kind of magically disappeared. So it's kind of cool that we've built processes around things that are otherwise barriers. So vended tools have busted through the data wall, the other requirement of scientific research is control.
And Vented Tools have also managed to negotiate new models of how control and delivery happen in education settings. It wasn't so long ago that teachers had complete control over the full student experience. Like it was just the case is a chalkboard, and the person who writes on the chalkboard is in control of what happens. So for that reason, if somebody like me wanted to gain control, I would have needed to gain control of the chalkboard. So a common way to run an experiment in these old days was to get a bunch of teachers together, random sample of teachers into a training program, and instruct them on some new things, some way of changing things from normal practice.
So, like, we'll train teachers on a new way of delivering biology education or something. And then they go off, and we hope that they teach students in the way that they were trained. And we have really no way of controlling or knowing whether they did the thing that they were supposed to In fact, there's this whole, subfield in education research called Fidelity of implementation. I don't know if you've heard of that. It's a fallout of the fact that when you'd run an intervention study, because teachers were the ones delivering the intervention, you had no idea whether they implemented it the way that you would have wanted them to do.
But technology has fixed all this. It's made it totally possible for teachers to share control, not to relinquish control, but to share control with, technology tools with, like, services and providers. So you can assign students to work in a learning management system in a platform that has control over what the questions are and what the things are. And it's delivered entirely online without the teacher needing to do much. So here, the teacher might behave more like a guide, like selecting what it is that the student might get assigned to And there are amazing tools that give students new opportunities to do things that would have been well outside of what a teacher would have had in their normal toolkits, and that would have been in just totally impossible fifteen years ago.
So what I'm advocating for here is that we can also be building research services using the tools and online platforms the teachers are already sharing control with. So either within these tools or is entirely new research focused tools. So we can literally build controlled experiments inside learning experiences either at the student level or the class level, taking out all the hassle and guesswork of whether the teacher is doing the thing that we thought that this teacher would do. So literally, we can design like big robust large scale studies that randomly assign students to different treatments so that we're not just guessing about whether they're, actually making an impact in education outcomes. And we're also rigorously evaluating their benefits.
So that's the background. I think that there's a lot of opportunity using like standards and vendor processes to be able to do better research. So now I wanna show you an example of what that might look like, specifically with terracotta. So terracotta is an open source LTI tool, that turns an entire LMS course site into a really cool research laboratory. I'm kind of switching the background, so you know, this is what terracotta looks like.
It's a we call it a portmanteau or maybe an acronym or maybe an acromantau for, tool for education research with randomized control trials. I'll start out by saying it's totally free. We're supported by a variety of funders, prominently the Institute of Education Sciences, IS is the nonpartisan research arm of the US Department of Education. We've also gotten generous funding from Schmidt futures, and all of our work is in collaboration with instructure. So the grant from IES is actually in collaboration with Instructure.
And what it is is it's an experiment builder that just sits within an LMS course site. So currently Taricata's primary feature is to be able to experimentally manipulate, assignments. So you can make two different versions of an assignment, version a in version b, and then say you want half the class randomly to get version a and half the class randomly get version b, and then you'll be able to compare outcomes between a and b on downstream course assignments and assessments. And that's a simple example, like just doing AB testing on an assignment is maybe the simplest thing you could possibly do. What does way more complicated things? And in a moment, I'll walk you through some of the additional things that it does.
Okay. And, it's not a startup. Again, it's a free service. If you were interested in adopting terracotta, you'd actually be adopting it from Indian University. And we provide hosting for anybody who wants to collaborate on research.
We're always looking for new partners. If you have any interest to partner and collaborate, then please like, let me know. It would be amazing. So, let's walk you through a little bit of what goes into it. If you click that blue button, create your first experiment, then, you're taken to an interactive guide that slowly walks you through the process of designing an experiment.
One of the first screens that you'll see in that guide asks you to identify or name the conditions in your experiment. Any condition I'm sorry, any experiment involves multiple conditions that people or subjects might be exposed to. So here you just said, okay, label them. And we can have AB tests if you just wanted to compare between two different conditions, but we can also support up to sixteen different conditions. So a two by two by two by two by two, that you can really test not just, is this better than that, but why something might be better.
So you could do actually, like, multiple control conditions. And then terracotta will later on randomly assign students to whatever conditions you include here. So what are the different conditions? Like, what are the things that somebody could do in terracotta? It's a pretty robust feature set. So, you build assignments in terracotta, and then you randomly assign which assignments or which versions of assignments students will experience. And these assignments could have multiple question formats, so it could be multiple choice, and they could be randomized or multiple correct.
You could also have short answer and file upload. You also can, adjust the display format. So for example, if you were interested to do a research study on whether presenting questions one at a time, or all in one block. That's something that you could just implement natively in terracotta. In fact, all of these can be differentiated between treatments.
Also submission option options. So if you wanna have some students only submit once, but other students can submit multiple times. Question content, we also support YouTube media. So if you wanted to have students watch videos for an assignment. That's something you could totally do.
And also, it's worth saying that all these things, all these interactions, we have robust data collection four. And YouTube media, I'll call out in particular. So we've instrumented terracotta with the, Caliper media profile. So anytime a student clicks in that media player, you're able to see what, they did. And you could even reconstruct exactly what portions of a video students have watched.
It also does interesting stuff with feedback, so you could have auto graded. If you have auto grading, it can take the most recent or the highest or the average, grade. We also implemented this really cool thing that I've never heard of before that turns out to be super efficacious. Cumulative grading, it doesn't happen in canvas. So you can tell students I want you to do this quiz four times.
And every time you do the quiz, you'll get an additional twenty five percent of your grade from that quiz. So it could be like that you give them full credit and they just have to do it multiple to give themselves practice. And if you've got that set up, you can also schedule it so that they have to have like a day in between each attempt. So we've really tried to make terracotta as rich with the sorts of things that you could put conceivably want to do research on. And you know, given this every possible combination, there's probably like thousands of different types of experiments you could do just on formatting alone, ignoring content.
At this point in presentations about terracotta, you might be thinking something like, how is this ethical? Like, what the heck You're gonna take students and give some of them, like, immediate feedback, and other students get delayed feedback, and you think that that's okay. And the answer is, Yes. But I acknowledge that it's, sensitive. So I'll say first that we we use and we strongly encourage anybody who's using terracotta to use the informed consent feature so that students who are about to be participants in experiment agree to be a participant next that experiment. So, the way that this works in terracotta is that we actually create an assignment that is an informed consent assignment.
It can have a due date. You can give students credit for responding to the informed consent assignment. There's no right answer, so they'll get credit whether they say yes or no. But this is the fundamental branch that determines whether students are randomly assigned. If they do agree to to participate, or if they don't agree to participate, they'll get like a business as usual default condition, and we won't do any data collection on that what you were gonna ask? Yeah.
Yeah. So, it's worth saying that if somebody says no, I don't want to participate in the experiment, it would be crappy if they just didn't get any educational experience. Everybody gets an educational experience, whether they agree or don't agree. It's just if they don't agree, they're not randomly assigned. They get default.
Whatever the teacher says is default. Any questions about informed consent? It's this is also, an interesting feature of terracotta that Unlike everything else in Canvas, the teacher can't see how students respond. So we also make it so that informed consent responses specifically are never exposed to anybody so that there's no risk of like a teacher, I don't know, in unfairly incentivizing students to participate in a research study. If you didn't wanna do informed consent, like, for example, if your your principal said, don't even worry about it. You can do the experiment on everybody, then that's totally okay.
We just have like a few warning screens. Like, are you you want to do that, before that's something that gets implemented. Okay. Simple study would be like compare a to b, but, more interesting study from my perspective would be to do within subject crossover design. So one other concern about experimentation is like maybe a is better than b.
What would happen with the students who get b? Like, does that mean that they're just kind of out of luck they get the worst educational experience? And one way to get around that is to say all students will get all treatments. So either a student will get a then b or a student would get B, then A. That's called the within subject crossover design. And in that situation, everybody gets everything. There's no bias due to experimental treatment.
It's just a matter of when they experience it. So you can measure student outcomes at the student level when they got a, and then when they got b, and then you can pick compare not between students, but within students What an individual student's experience of a, how how that changed things compared to b. So it's also really good for, like, statistical power and stuff. So terracotta does crossover designs easily. And, it's also the case that you don't need to have just one crossover.
You can go back and forth and back and forth back and forth throughout the whole semester. So you're able to do some really cool things with, terracotta as far as, like, scheduling treatments. Okay. Also, a lot of the details of what students are doing in their classes are not gonna be experimentally manipulated, I'm talking about a tool that makes it possible to randomly assign students to different treatments, but downstream, like after you do your experiment, you're probably interested in how that affects I don't know their performance on the next test or on the midterm or the final or this, you know, the statewide assessment. And what you can do in terracotta is you can link the experiment that you've created to some sort of other outcome that's already in the Canvas Gradebook, or if you just have some other data that's not in the data in the Gradebook, but that you want to include, like I don't know, students' attendance or something.
You can just manually add those, those numbers to the roster just the same. So the idea is that we're allowing teachers and researchers to have the scope of their experimentation be beyond just what they're manipulating in terracotta. You can also look to see how it affects stream outcomes that are authentic to whatever the teacher's really interested to improve. Yeah. So you just select it from the grade book.
Okay. And, like this is my favorite and also least favorite screenshot. Imagine yourself like scrolling up on this screen. So this is the experiment status screen. This is what you'd see while you're running an experiment.
At the very top of this page, there's this blue button that says export data. And imagine yourself clicking the export data button, and it'll download zip archive that contains a variety of CSV files that have all the data from your experiment that you could ever want. It includes like what the conditions were, what the assignments were, the questions in the assignments, the participants, when they provided consent, it's all rigorously de identified. So no student name will ever appear in this And it also makes it possible for you to do analyses on the experiment that you ran without needing to make a request of like the Canvas data admins or something. So you have control over the experimental research data.
And, yeah, you as a teacher, because you're already being exposed to, like, what students grades are. There's no, like, new exposure of, outcomes that, you know, you wouldn't otherwise be able to see it. It just makes it so that you could analyze research data without worrying about identities. Yeah. We've got a robust data dictionary.
It describes all the details of things. And also, I've provided some, like, short analysis scripts to help people if they're just interested in getting started with this format. We had to we had to invent, format. So there's no, like, common or standard for how you represent an educational experiment in data form. So, yeah, we did our best to put it together, and we also have a mapping to SEDs interested in like comparing your outcomes with what goes on in common educational data standards.
Okay. That's what terracotta does. I also wanna be very clear about what terracotta doesn't do. So first, terracotta doesn't intrude. It's not the case that I am going to run an experiment in your school.
So researchers would still need to negotiate with teachers and schools and get study approval. It's like set up like an LMS is normally set up where the teacher is the one who needs to create the stuff. So if I wanted to run an experiment in somebody's class, I talk with them how to create this stuff. And I work with them, and I might have like a Zoom meeting where we actually build it together. So it's not the case that anybody's running experiments from behind the curtain wizard of Oz style.
Also, it doesn't provide any access to private data. So, terracotta is not a company. We don't go and use the, We don't go and use like our access privileges to go and scrape data from outside of terracotta about what students are doing in Canvas. We also don't provide anybody with backend access the terracotta database. So it's not like we're gonna be doing research on the research that you're doing.
You can you maintain ownership of all of your data as kind of the institutions and teachers and really our our ideas to get the data to you. If you're an institution though, so like if you're an LMS admin, We do allow you to subscribe to an identifiable Caliper event service so that you can see what's going on in terracotta so that you know working, and you can verify the integrity of your academic assessments. So yeah, that's the rundown of terracotta. I wanna now kind of shift to mentioning what I think about when when it comes to the future of education research. And really, I I think Tericot is amazing.
It's made my wildest dreams come true. I've done some amazingly rapid rigorous and responsible large scale experimentation. We're, because everybody's talking about AI, I should mention that, like, we were able to whip up an amazing experiment that we're gonna be deploying in the fall, in all of the elementary composition courses at IU. So, one thing that you could do with chat GPT is you could give it an a student work and then have it give feedback. So we're gonna compare, when chat CPT provides feedback to peer feedback.
So in this elementary composition, students already do peer feedback, and rather than just having them type the feedback to their peers, we'll have them process their peers work through chat JPT, get the feedback, before delivering it directly, they'll, like, look at it, reflect on whether it's the feedback that they would give, and then they'll deliver that chat Jpt feedback. So we'll be able to see whether the efficacy feedback from Chachapiti is any better or worse than when peers actually do it to each other. Yeah. We're excited about that, and we're gonna be do doing it. Again, at a very large scale, so we're talking about like eight thousand students just in the fall.
Okay. In spite of how excited I am about it and all the cool things we could do, I acknowledge that Terracot is just one tool. So, it does some cool things enabling one particular kind of experimental research, but that's not the full range of all the research that people could possibly do. And it's my guess that of the people who raise their hands when I asked who does research, that some of the things that you're doing is very different from what currently is being done with terracotta. Okay.
So education in general, but education technologies, specifically and in particular, have just chronically under invested in research and development. It's the case that we're much quicker to kind of like run to market than to evaluate whether the thing actually works. So if we're gonna rely on research for making evidence based decisions in education, we just need much more expansive research infrastructure. We need to make it easier for vended tools and for teachers and for researchers like me to be able to do stuff, like evaluate what works, it doesn't make sense to claim that we're evidence based if people like me need to, like, struggle for years to be able to conduct a single research study. So I have a vision that someday terracotta might be one tool in something like a national research infrastructure toolkit, which would include additional for doing additional things like manipulating personalized messages that might be deployed by some AI system or communicating with parents or doing survey data collection or any of the range of other things that might be possible.
So such a toolkit would need standards for implementation and interoperability, and that's really why I'm here to advocate that in the same sense that LTI and Caliper analytics and all these things have enabled an ed tech ecosystem that thrives. It'd be cool if we also used those processes and standards to be able to make it so that other research services might become integrated with our educational systems. So, the good news about this is not just me who has this vision. There are other people who have this vision too. And one of those, organizations is the National Science Foundation.
So I'm, excited to share the news that we've recently received some super generous funding from the National Science Foundation to imagine what this toolkit might look like So if you've got a toolbox, terracotta might be one thing. What are the other things that should be in it? So with colleagues at IU, we formed what we call the interact incubator. It's a collaboration between some of the most amazing brightest, education researchers in the country and also cognitive psychologists and social psychologists and technology stakeholders. Also district administrators and policy makers, we have the actual commissioner for higher education from the state of Texas involved. And our focus is really on improving equity in STEM education, specifically in K twelve.
And over the next two years, we're gonna be convening multiple times to conduct first a needs assessment. So let's just imagine what we would like to do if there were no barriers to research at all. And then we'll imagine what that infrastructure would look like. So we'll actually do a design process. So what do we need in terms of data and control? And the idea is that once we're done with that, we're gonna be submitting a giant figure grant proposal to NSF, just to make it a reality.
So if you're interested in any of the things that I've talked about, if you're like, oh, Terracat is cool, but I would really like to do something else, And especially if you're in a K twelve district, then we would really love to include you. We have applications to expand the incubator from what we currently have, and the applications are doing like four days. So, if you're interested, then please let me know. It's not a hard application. You just like write down your name and give us your resume.
Okay. So to bring it totally full circle, I want you to see me like a melting candy bar in Percy Spencer's pocket. So we're all trying to build an educational technology marketplace with tools and providers all constructively working together. And here I'm saying, Hey, you're also totally solving a big challenge that's plaguing education researchers for generations. So if you're interested, I'd be excited to to chat about how we can melt this other forms of food, and also to build the next generation of research infrastructure.
Thank you so much. Yeah. I'm interested in questions. Yeah, please. I have two two questions.
Okay. First, as a utility, as a surface. In the University of Wisconsin System, we have a rather rigorous intake process the learning tools. You know, you describe it. You would describe being a vendor and kind of having an open door in your end.
That's the exact opposite how it is, our our intake process to be a brick wall at the vendor chooses to not do something in the whole process because we met them very thoroughly. I was wondering about the challenges, if you've had any of those challenges, where the gaps are. The second one is more of a comment, and that is that the holy rail in the LMS right now is being able to use all the tools that Canvas offers while being largely anonymous. Because even if you take off the people and hide other things, the inbox is still there. So I'm just going to leave that with you as here's here's the pie in this guy.
Yeah. This is where our IRP group wants us to be able to do things that we just can't. So, okay. Those are good questions. The first I'll start with the first question, which was something like Okay.
Is the vended process for gaining access to a provider actually so simple? And you'll just have to take my word for it that despite the fact that, yeah, administrators and info and audits make things harder, it's still easier than if you just make a research data request. The other thing that's unique about the vented tool process is that it's process. So you can say, I need you from the perspective of Wisconsin. I need you to have a heck fat. I need you to have, vulnerability scanning certificate.
I need you to do this and this and this And that's a set of criteria that a researcher or a tool like terracotta could muster. It might take us a little bit, but once we've got that, then the process is relatively lubricated. So we've worked through much of this word, like, a week away from our vulnerability scanning certificate. So, yeah, it's the, it's the case that there's at least a transparent process. So it might not be like super smooth, but at least it's something that somebody could navigate.
Okay. The second thing about like an anonymity, I'm actually excited about the idea of anonymity, at least from the perspective of the student. What I would like is there to be nevertheless a lot of protections for what that, what that identifiable data are. I try to treat it, treat it as, like, rigorously as sensitive as possible, and as I can in terracotta. So the, the data that is identifiable is in separate tables that have tighter security protocols than the other data that's in terracotta.
Yeah. So we've tried to architect things so that we're being as cautious and as sparing with identifiable data as we can. I think that I think that you would share the idea that even if we've got this pie in the sky vision of anonymity, we still want there to be some identifiable integrity. Right? And so administrators should probably have that. And I'd like to believe that that is also potentially useful for understanding what works, like we should be able to assess our practices.
If not, then it's like why are we collecting the data in the first place? So, yeah, I'd I'd like to think that research could be one of those privileged few that should be able to play. Maybe not with, identifiable stuff though. Yeah. Yeah. So you mentioned you don't do anything with the data you collect.
Is there any options to just not collect like opt out and have it sent to the institution, like, choose a data store online institution. Totally. Yeah. I I'd it was something I glossed over super fast. You don't need to involve me at all.
Terracat is totally open source. So, We have instructions on our GitHub page if you wanted to spin up an instance and host it yourself on your own infrastructure. I'm not a developer, so I don't know how difficult it would be to, like, pull the branches down as things get updated on our side. But if if the fact that IU is hosting it or that it's being hosted on AWS as a problem or something, then I would be excited to explore the possibility of trying to help you do your own thing. Yeah.
We haven't done that yet. It's it turns out that people are much more happy to just let us take care of But if it's something that's sensitive and you'd like to do it yourself, that would be that'd be awesome. Yeah. Yes. Okay.
So our, we just finished a two year wacky research competition. I'll take it two steps back and tell you why it's wacky. So, normally when people like me do research studies, will, like, write a a grant, and somebody will review it, and then maybe they'll fund it, and then we'll do the project. For various reasons, funders are interested in shifting the risk to the researchers. So instead of giving them funding to do a research study and then they do it, they'll have them do the research study and then judge whether it's good and then give them funding after the fact as like a competition.
It's like a big grown up science fair with extremely high stakes. So, we were participating in the X Prize Digital Learning Challenge, and thankfully, we came out in second place, so we got the payout in the end. The study was awesome. It was, very large scale effectiveness study of pre questions? Have you heard of pre questions before? It's okay. So normally in education, you teach students something and then they answer questions about thing.
It goes like learning and then you assess. But for over a hundred years, there's been this hypothesis that maybe people would actually learn better if you ask the questions ahead of time. And we're starting to see some preliminary evidence of this, not just from like psychology research, but also from practical research when students are doing pre tests because teachers need pre test assessments to show lift in their classes. So, anyway, this was a very large scale, assessment of, like, what happens when you ask questions before students watch videos compared with asking it afterward. And we found sure enough that there's an amazing improvement in learning when the questions are asked before.
And it's weird from the student's perspective. I mean, imagine like sitting down to an assignment And before you've learned anything, you get questions on this stuff that you don't know the answer to. So for the most part, they're guessing, and most of them get it wrong, but you give them credit just for the fact that they answer And you find that that improves learning downstream better than asking the questions directly after the the video. So it's like a five percent improvement. It's like totally, practically significant, which is crazy to to consider again that they don't know the answers when you're asking the questions.
So that's not as sexy as chattyPT, but yeah, we'll be doing the chattyPT thing in the fall excited about that too. Please reach out to me. Yeah. So, it's it's me and a project manager, and we use, Unicon for development services. So we're a relatively small team.
That's why we're free. But yeah, yeah, we'd be excited to walk through the process with you. Yeah. Yeah. And also, Everybody, like, later tonight, come to my little little exhibit booth.
I I have chapstick. I have like so much chapstick. And it's a good chapstick. I just got, like, a fifteen hundred things of chapstick, and I have not given away fifteen. I don't wanna bring it all home.
Yeah, you should not need to buy chapstick for five years. You're serious. I'm so sorry. Yep. Okay.
In the back first. Yep. Yeah. I'm I'm one of the people that doesn't know why I'm here, really. Way to go.
Educational research at our university. And I thought it was more about the school and family. So I was really a little inappropriate question, but, like, I know whenever he's trying to do something. He just cut off from IRP. Yeah.
It's a good question. Like, what's our It's a great question. Every study I've done with terracotta so far. So, sorry. We've been like out of alpha, so we've actually been testing it for about a year.
So I've done maybe like four or five studies. Every study that we've done is with IRB approval. And I'm a big proponent of Open Science. I make all of my IRB protocols public, So I'd be happy to share with you. If you just want to see what an IRB protocol looks like with terracotta.
And the good news is that it's very structured, and everything is, like, kept What's the right way to say it is that I've anticipated what IRBs would mind. So I've made it so that this addresses their concern. So I'm happy to share the templates with you if you're interested. Yeah. This is not a way of circumstance.
Totally the opposite. It's a way of making IRB happy. Yeah. Yeah. Exactly.
Yeah. Yeah. So, every step of terracotta in a sense is exactly what the IRB would want. So it describes how, informed consent is hidden from the teacher so that nobody ever sees student's consent responses. It's just managed automatically, which is a major improvement over any manual study.
And, yeah, the fact that the data are de identified from the source is also a big deal. There's never any need for anybody to see anybody's identities. You could just imagine they're just being random codes to identify students So that and a variety of other things make it super easy. Yep. Yeah.
That's a good question. Okay. And do you had a question? Same. About the use of higher education. I'm in a learning analytics working group.
And, so to actually get some more more observational than And, are there other uses applications for total. Yeah. So I've presented it here as being like an experimental research platform, but that's because that's how I envisioned it. And the more that I talk about it with other people, the more I discover, like, oh, something new that somebody could do with it. So one of the things is just to do survey research.
So normally, if you wanted to run a survey, you're forced to go off and use qualtrics you could use Canvas' survey tool, and in either situation connecting those survey responses to student performance in Canvas, is really hard. It involves like knowing students identities and then mapping them. So terracotta solves that problem. If you're ever interested in a survey and also to have students grades, somehow connected to survey responses, it does that. So you could do anything in learning analytics that would otherwise be associated with a survey, like don't know, self determination theory or some sort of self regulation, self self regulated learning inventory.
I'm closely involved with learning analytics too. I'm I'm one of the leaders of the method sig for the society for learning analytics research. And I always imagined that, like, learning analytics is continuing its growth and that eventually it'll shift from being purely observational to starting to do things to nudge students in the right direction. And when that happens, I imagine that experimental research opportunities are gonna like, I don't know, what people start talking about and learning analytic spheres. And that'll be a good thing so that we actually start, like, feeding forward and closing the loop.
Yeah. Yeah. It's a good question. Yeah. One question.
I hope I didn't miss this part, but when the study has completed or that initial data is there, Is there any kind of, high level dashboard presentation, or is it all statistical analysis to determine results? I love that question. Okay. So I have a collaborator. So Mark McDaniel, he's a professor at Washington St. Louis, And, this has been like a debate between us for a long time.
Like, okay, so I'm giving people data because I'm a fiend for raw day I love data, but Mark has always been saying, like, you need to have some sort of, like, high level summary so that people could just see what went on. So, We've been designing it and going back and forth with Mark and going back. And it was only just like two weeks ago that we finalized our non functioning prototype for a results dashboard. And it'll be built probably by the end of this calendar year. So by the time anybody here is adopting it and completing an experiment, there should be a nice, results dashboard.
That'll do two things. It'll just kind of show you the numbers. So, like, you'll be able to see, like, this is the number of students that completed the informed consent assignment. This is the number of submissions you had to every assignment. This is the grades on those assignments, but then beneath that, it would actually show you the performance contrasted between conditions.
We would have an actual figure that shows dots for every individual student submission. And you could select different outcomes that might have been included in the experiment. And then, also, it would have, like, the descriptive statistics. So you could take those and do statistics if you wanted to, like, if you but it'll show you the means and standard deviations so that you know what what the differences are and what the scale is. Yeah.
Yeah. So how how soon will you have an AI generated paragraph copy and paste into that main screen. Not yet. That's a good question. And that's exactly the debate that I've been having with Mark is like, How how much do we want to spoon feed it to people? So, we made this strategic decision that we're not gonna do statistics.
So, that gets complicated. Like we can't just know that the statistical assumptions are met. So, yeah, that's a, that's a good place to draw a line. Yep.