Penn State: Leveraging Robust Analytics to Deliver Impactful Outcomes for the Modern Learner

Hello everybody. We're back Just a reminder, there are seats up front. So feel free to come and sit down. They're not actually reserved. That was just for the keynote. So, I've got notes for this one because it's been a long day, and so we're we're recovering.

So I wanna thank everybody for joining us. This is, panel discussion that we are so excited about. It's actually amazing, sponsored by Amazon Web Service who's our Diamond sponsor for this year's Instructure Con. I've said it before, but they allow us to do so much of the amazing work that we do. So we can't really thank them enough.

So today we're talking about, our educational institutions, needs to, use data and analytics to support learners across the board. And this is I you'll hear me say Venn diagram, I think, across the board, many times. But I think, Dean analytics is one of those areas that is, so intrinsic to supporting students. And as we talk about emerging, topics like AI and the metaverse and things like that, the underlying tracking of that data and the success of of students based off of their experiences is is incredibly important. So, joining me today are Doctor.

Jennifer Spiro, senior manager of Higher Education at AWS. Andy Fisher, manager of learner educate engagement, at Penn State. And Ben Heller, manager of Data empowered Learning at Penn State. So thank you for welcoming them all to the stage. You've seen me.

I'm boring. Alright. To get yourself Jen, tell us a little bit about your role at AWS. I know you came from the education space. You're uniquely qualified for what you do.

Tell us all about it. Sure. I'm gonna talk about that Venn diagram piece of it because it feels like so many parts my lives are overlapping. When I come up here on this stage, I came to AWS about twenty months ago, as a higher education senior manager really helping to bridge the gap between the work that we do at AWS and the most challenging issues and concerns and problems that higher education is face but before that, I was actually at Penn State. So I got the pleasure of working with my two colleagues here on stage.

And we were thinking about the entire student ecosystem in really interesting ways. And so this team has been a part of sort of pushing the envelope for Penn State and pushing the envelope for instructure and for AWS on how we can do that better. So, and, I teach at Penn State So, I get to actually, as somebody said, they're changing eat your own dog food to drink your own champagne, which I find is a much better metaphor. It is a much better metaphor. We like that one better.

Yep. Excellent. Well, you see you're in the trenches every day. We, you know, many people in the audience, and it's hard to see, so maybe cheer a little bit. How many people are are struggling to actually, update or modernize their data their approach to data and analytics.

Yeah. I see a lot. I see some ads as well. So It's kind of a common challenge we hear about a lot, which is of course why we're up here talking about it. But, Andy Ben, thanks for joining us.

Tell us a little bit about, Penn State and your role at Penn State. Yeah. Sure thing. So I can kick off the Penn State crew here. Go for it.

Alright. So I'm Andy Fisher on the manager for learner engagement at Penn State. It's a title that encompasses several different things. I manage our LMS canvas, which is why we're here, but also some other learning tools I think many other folks here in the crowd probably do as well. If you end up with your Canvas environment, you probably own forty or so integrations that are somewhat supported at your institution.

I also manage Zoom and Kaltura, and a little bit about Penn State. So Penn State is a, large State institution, it's a sole land grant institution in Pennsylvania. We have, twenty four campuses, about six four hundred faculty and about eighty nine thousand, ninety thousand, learners, students, varies by year, slightly, that that that we support. So I'll hand it off to Ben, and then we can dig in some more. Sure.

Thanks, Andy. Hi, everybody. My name is Ben Heller, and I am the manager for the data empowered learning team at Penn State. The data empowered learning team's mission is to leverage all the institutional data sources that we have available at Penn State, our SIS system, the learning management system data, the learning analytic data that's coming out of Canvas. And what are the ways that we can mine that be able to use that to empower student success to, improve pedagogy and teaching and learning.

I also just want to say, this is my first, instructor Khan. And I would've Excellent. Yes. He's one of us now. One of us.

Thank you for joining us. It's awesome here, and thanks for the background around Penn State. Tell us a little bit about how, Pennstate's leveraging data and analytics. Both within Canvas and then other aspects of student life. Sure.

I believe we have a slide. I I both I multitasking. Oh. No. It's it's right.

So I'll give a little context before we get into the slide. So, canvas is our LMS. So that's really the the the centerpiece of all of the learning platform kind of tool integrations that we have. So, we have been on Canvas since two thousand fifteen. I don't think I came to that instructure, Con.

I think I came to I might have been in a different conference that year. I think Jennifer may have been here, so for Penn State. But what I did find this is an aside, but the On the way into Denver, I've always noticed that there's always a tent involved somewhere in an Instructure Con, either for keynotes or the airport makes it look like a tent. So I think carrying that forward maybe Long Beach was the only one that didn't have a tent I was at. Yeah.

Yeah. Yeah. Is that a requirement from here? I think so. Yeah. K.

You have a move forward? Okay. So Canvas to centerpiece, when you came from Angel, so if anyone has run Angel, that was where that was how I came to Penn State. I started working on Angel. We, at when we departed Angel, we were running a hundred and sixty web servers at the time to support our twenty five thousand concurrent users. This was the handle failover.

They were all created manually installed. There were there were some scripts involved, but everyone was a bespoke web server, one hundred and sixty of them. Good sized team that supported that. So that's one of the reasons why I'm on stage with AWS right now to show and talk about how some of that has improved over those years with the commodity services and other things that are out there that can really make that that easier. And with Canvas, we don't have to worry about that load anymore.

I'm not building a hundred sixty web servers. Thank goodness. Yeah. That was that was, you know, during the pandemic. Those are the schools that really struggled.

The ones that had So many servers without that ability to scale really was a a massive challenge. Yeah. Yeah. So canvas scales up or down to what we need along with some of the things that Ben's gonna share as well. So we can have the right hardware, right cost, not have overcharges, those types of things.

And really not, as me, as a service owner, not have to worry about scale and scope of some of these applications. Yeah. So I I like starting the story with the pandemic because that was really the genesis for our innovation. Let let's rewind back to spring twenty twenty early March when the pandemic hits. Penn State is on spring break at time, and the president of the university says, students don't come back.

Everyone's going to get online. And our instructional designers across the university get a heroic effort to put every single course at Penn State online, such that every course at Penn State, and this is now even true to today, that every course at Penn State has an automatic Canvas course created for And this can make analytics challenging because you want to be able to figure out and one of the questions that would come to us as a result of the pandemic is how can we find the students that are no longer showing up? How can we, if all everyone is going online, how do we know whether or not, you know, a student is disengaged just from one course or all courses? Across their, you know, across their semester schedule. So what we did is we, I sat down. I have a background in user experience design that was my, that was where my PHD was and in human factors. And I sat down with our data scientists, and we created an analytic, and if you can just go to the next page there.

There we go. We created this analytic, for academic advisors. And what this analytic shows is we looked at all of the, activity that was taking place in Canvas, and we made a relative analytic that compares the student's activity in red to the, course average activity or the other, or the activity of other students that are, in that same course. And the power of this is that, you know, you know, when I said that every Canvas course and every every every course at Penn State had a Canvas section, That includes like every single sculpture or painting class that has probably very little, use of canvas, and it also includes every, like, stem course or every online course that is heavily using Canvas. So, we needed a way to reduce false positives and alerting such that we don't need, if a student is not doing much in the Canvas course, then we, in the rest of is not doing anything, then we don't need to, alert or let anyone know that, you know, that's just something that we can ignore.

However, if a student is engaged in a course like this one, and then all of the other students are doing something, and the student is not. Well, then chances are that this is something that an advisor would want to know about. And importantly, an advisor would be able to see this pattern across multiple courses. Is this a student that's just disengaged in one course because they're not interested anymore, or is this a student that is at home sick, taking care of a family member, and they're showing now patterns of disengagement across all of their core sections. And so we created this tool for academic advising, and we released that into the wild, in, twenty twenty ish, fall twenty twenty.

And then what we did is we took that same tool and we, showed it to academic, we showed it to faculty and we said, Hey, we have a tool that advisors are using to great success to help them find students who are disengaging from their online courses, and they're using that successfully to sort of engage and develop interventions around reaching out to them. Would you guys like to have access to? And the first thing the faculty told us was no. We don't want another tool, to use like we want another impacted wisdom tooth. So if you're going to give us a tool, you have to give it to us where we live, and that's canvas. So the next slide shows a, we created a whole new tool called course insights, and that's what I'm here mainly talking about today.

And so course insights is a version of this, analytic that is able to show the same thing that an advisor sees for an individual course roster. This is a diagnostic, level of data that allows people to see what's actually happening with a student's activity in a class. And that's also why we make this, it says a rolling seven day average because it's amortizing all of the activity that's taking place over time, such that it doesn't matter whether a student is studying on a Sunday or a Tuesday or a Wednesday. There is not a lot of volatility. We are just looking at what's taking place in the curve.

And to the degree that the, the, the distance between the red line, the student, and the blue line, we show we paint the background orange when the activity is flat to indicate this is a period of zero activity. And if it's in yellow, this is when we're showing you, hey, this is a a time where this student is doing fifty percent less than the average student in the course. And this is something that when we show this to faculty. You know, this is very interesting when they were trying to help students that said, oh, I've I just can't catch up with my work. There's I I keep struggling with my work.

And we heard from faculty that then used this tool and said, well, I hear you say that, but I also see that you're doing about half as much activity as the rest of the students, you know, in this course. So What is, you know, what sort of remediation, what sort of intervention can we help provide you to help you, you know, understand the study requirements and adapt your own, studying strategies to be able to be able to succeed in the course. And what other material can I give to support that? And so, course insights has this analytic data along with other factual data that we've taken from our SIS, and other tools along to paint a more holistic picture of a student experience. Yeah. One of the powerful aspects of course is the fact that you kind of put it right where your educators are.

Right? We see that with a lot of aspects. It's one of the cool things about Canvas is you can do that easily. And where they can find it, they can adopt it. It's pretty impressive. One of the things we also like to talk about, I always like to say, Canvas was born in the cloud.

Right? AWS has been our partner since the founding. It's the only way we host, Canvas and the the instructor learning platform. And so it's one of those things where, because we've partnered for so long, we're able to to scale and and support our institutions with this type of customizability. But, Jan, I know, you actually had some background working with Penn State for seven years before, moving to AWS. Tell us a little bit about, you know, the challenges you faced.

These are these are amazing results, but there's a lot of work behind that. Right? Sure. And I think, the folks in this room that are working on campuses understand that anytime we start to, expose some insights like this, there's often some, tension at the institution about how much they wanna know versus how much they don't wanna know. And so I love that the the data empowered learning team started off with some really basic questions in order to get access to the data because we were fighting with data silos across the institution and and data owners who said, you know what? That's not you don't have a legitimate reason to have access to that. So we started off by asking the registrar, what what would you like to know? And the registrar said, I want to know if eight AM classes are bad, because I continue to hear that all the time.

And Ben and team started to dig into that a little bit, and it turns out that that's actually the wrong question. The right question is, are, for whom our eight AM class is not good for. But for whom are they good for. And and that was a really interesting insight. And then we took what the the team had learned from that and gained access to the data to really roll that into, looking at underrepresented populations in our STEM programs at Penn State.

And this was really great because again, we worked closely with advisors on what information, what insights do you need in order to help these students who may be, historically have not stayed in the STEM programs. They may have stayed at Penn State, but they didn't stay in those STEM programs. And so we we had this really great that the team came up these these amazing insights. We worked closely with, the folks in, our our ethics in to. We worked closely with our privacy officer.

And I thought we had really excellent buy in. And then we started to roll this into it and even deeper insights into, day zero grade prediction, and the team got really good at this. They were looking at sort of how, the the day the moment a student registered classes, we could get in within about a half a letter grade, with this. And what was fascinating was the institution wasn't ready for that kind of insight. Right? They there was a lot of trepidation about what, what faculty might do with that information.

Are advisors appropriately trained to address that? What, what support did we have for students that maybe we we saw weren't going to be successful in those courses. And so, the team really took what they had learned in that space and pivoted into this course insights here. So, what I I think you're gonna get to this, Ben, but in addition to being able to view this as a faculty member to see, Hey, is there a student who's just dropped off, compared to the other students in the class? There's also a lot of background information and data that's provided to faculty. So as they're thinking about how they might change their curriculum, how they might adapt to it, it might be dependent upon all of the students that are enrolled in my course this semester have all had x courses before. Or they're co enrolled with those courses.

So is there an opportunity to do some crossover curriculum? And knowing things like GPA. I think those are really powerful tools that as we start to expose them to faculty through the things that they know, Pency is sort of on the, on the bleeding edge of this. Call it the bleeding edge, because I do feel like we kind of jumped over the precipice and had to come back a couple of times. Because as I see the landscape across all of higher ed and in my role at AWS. The things that folks are trying to do, number one, they're trying to have better insights from their data.

So turn that data into wisdom. And number two, they're trying to provide a twenty first century digital experience for our students. So you think about the things that students come to our institutions expecting their institutions to do. I have an n of three. So, Andy had to live through two of them.

I'm like, Hey, my kids are reporting that canvas is down, right? It was only once. Was only one time. But, you know, I I get those reports from my kids before we were getting any other insights. But, you know, my end of three and the things that they were saying, like, this this, whatever this is, technology should be able to do these things. And why can't you have a course recommendation engine like Netflix has a a TV show recommendation.

So that twenty first century digital experience that students are expecting when they come to their campuses I think we're starting to make some inroads with the data, and working with our partners on how we expose these kinds of things and make them available in the ways that folks are are using them already. Yeah. I get the feature requests for my children as well. It's also my fault that there are no more snow days. I don't know why I'm wayne personally, but, you know, I love that we're in this era where, you know, I went to college when we had weed out courses.

And they said, look to your left, look to your right. One of these people isn't gonna be here at the end of the semester, and they they took pride in that. Right? I love that we've evolved in the point where we're really at, creating a a holistic picture of a student so we can see how well they're doing and keeping them on track. It it's pretty amazing. But tell me a little bit about, you know, those analytics are only as good as the data they have access to underneath.

How have you figured out how to pull those data sources together? With lots of elbow grease. No. So, we actually have a slide here that talks a little bit about the data sources. So we get our data sources for our learning analytics, through three different means, that we all combine, together, which I'll I'll talk a little bit about next. So the first thing that we get is we get our learner activity data from Unison.

Unison is a nonprofit consortium, that Penn State is a part of, as a part of our, consortium with other universities. And this is, universities that are also sort of, you know, similar scale institutions, here in Denver, Colorado State is a is a member of Unison. Give you an idea in Michigan, Ohio State as well. And, the things that Unison provides is great. Whoa.

Wow. That would be a rocky mountain thunderstorm, everyone. Thunderous kinds of information you're providing. Understood. Wow.

That's your walk I'm using. Yeah. Trying to make sure everybody's staying awake. That's just, you know. Okay.

So Unison provides a mapping so that when we say, Hey, we have a student with, an ID number at Penn State, they do the cross walking and the data modeling, so we can we can find that same student in Canvas and then find out what they're what they're doing in top hat and then find out what they're doing in Kaltura. And any other, vendor tool that is supporting and and supporting that underlying caliper data standard. So we're taking all that data from Unison, combining that with the rest of our SIS data so that we know when students sort of drop or add different enrollments because enrollments can change almost any day during the semester as students late drop or withdraw. So we need to make sure we're only doing analytics on students that are actually enrolled in their courses. And then we have to combine that with our advising data, which is Starfish, which I see says by Hopson's, but has now been bought out by EAB.

But, Starfish gives us a very important role because that this gives us some sense of governance so that we can know which staff at Penn State has provenance to see which students. So does an advisor have access to be able see a specific college, a specific list. Are they responsible for, let's say, you know, a specific population or like, our athletic center uses our tools to be able to, to be able to track and monitor the the, athletic teams And so we need all of that data, and we need to be able to combine that. So we have all of the data available in order to produce, a seamless analytic. And we use that when we're taking this data, we're putting this into our Amazon, our our own Amazon instance, at Penn State specifically.

We're we're basically taking all the data from Unison, from Starfish, from LinePath. You can go ahead to the, to the next slide. And we're we're building out a pipeline. So Imagine imagine that every day, we're looking at all the activity, all the changes in enrollments, anything that's changing, in our advising as well. And we're getting what's happened in the previous twenty four hours.

We're taking all that data. And at Penn State on any given fall or spring day, we're talking about like two terabytes of just raw change data. And so in order to process that, we load that into an Amazon EC two instance. EC two is a la elastic cloud, I believe, or elastic computing. I always forget the acronyms.

Well, elastic cloud computing. The c is two. That's it. Yeah. So I knew it would come to me.

And we we're spinning that we're we're spinning up a a virtual machine, and we're loading it with all the data, and then we're using data science to process because the data is so vast, this isn't something we can connect with SQL Servers. This isn't something we can connect with microservices. We need to use R and Python and the and the toolkit of a data scientist to be able to to wrangle all of that data and then optimize it so that we can deliver insights you know, you know, at the, at the speed of which the university operates. And so we're able to can take that data, that two terabytes, compress that down to just a sort of fifty megabyte. Here's the daily facts that have changed, and then we store that in our own separate database so that we can keep track of that.

Out this semester. And so that's the secret sauce underneath how we're able to sort of deliver all of this insight, within like less than a second. Annie? Yeah. Yeah. So to tie this back to, to then, to the, the, what does this mean at Penn State, right? So there's a system here that's consolidating a whole bunch of data together, and we're using some bits and pieces from places, units in Amazon, it all together.

But there's a a a core, this guiding principle we have at Penn State. We have this whole idea of one university, physical campuses separated geographically with one world campus. And I forgot the last part is. Oh, and one single degree. Right? So at the end of it, there there there is no Penn State Altuna degree.

There's no Penn State online MBA degree. It is a penn state degree from everywhere is the same. So all of our students may not have the same, faculty. They might be at a different location. They may not even be in the, in the United States.

But what they have is that this system has to tie all of that together with all those different learning tools, all those different faculty, the six thousand four hundred faculty, whatever they choose to use in their course, get into the system so it shows the same graph to every faculty member, so they can then see at the back end, this is what it means to my students. This is how I'm engaging my students, and they can pick and choose the tools that have better datasets that we could start processing and so on. We're gonna throw that VIN diagram out again. You know, as we We talk about partners. We talk about LTI integrations.

It's a little Venn diagram that ultimately leads to more cohesive data underneath. Right? Exactly. Exactly. And then part of it, we we did a little bit of the auto scaling. So I think back to that hundred sixty web servers, with these systems, we can't be spinning up a new architecture because our data set has grown by two fold, or because CD two has rendered all the stuff we do obsolete, which it hasn't.

It just changes it. But we can still do that type of work because we can scale up and scale down. We don't have to migrate off of old hardware. Right? I I haven't had that conversea. I've every other project I have on migrating off of old hardware that I maintain, but Yeah.

No. And the other other importance here of of us being on on AWS is that it's also very cost efficient for us to do so. I mean, and he was talking about scale, but the the cost of our team doing this and doing like this level of data processing is like less than twenty five thousand or it's a very small amount. And a lot of this is also because, you know, while we're doing EC two, where there's this Amazon thing called reserved instances. And I can say this in front of this crowd where I couldn't say in front of my students, which is you remember a nineties calling card.

Where you had, you could pay for minutes ahead of time. And it's the same sort of idea with EasyTwo. A reserved instance means you're sort of paying for a cost at a reduced rate versus sort of paying for EC two during the sort of whatever the on demand rate is at a time. So we can we have a very affordable way of doing this through Amazon. Yeah.

One of the things that we also had to, overcome in terms of hurdles was, data security. So I'm married to a seesaw, right? I spend a lot of time trying to do Here's all the great things and tools you can bring, and he tells you the thousand reasons why we can't. So as we were starting to look at this, where the student data was being stored, and, how we were moving it back and forth, was a really serious question. And I think as, as we at Penn State, we're looking at AWS as the the housing of the data and the housing of our analytics pieces of this. It was a no brainer for, for the security office to say, yeah.

This this is actually as good as if not better than if it were stored in a server underneath your desk or in our data center. And so I think that, you know, that continues to be, I think, challenges as we talk about student data, we talk about privacy, we talk about how, these analytics, can be, misused. Right? And so I think for for our purposes, we were trying very hard from the beginning to think about what is the most ethical, moral way for us to be doing this in a way that is hearing student data. And Jen said that as a joke, but literally when we started, there was a server underneath someone's desk, and we had to make sure that the door was locked any, you know, every night and would panic if, you know, for whatever reason, it was unlocked or open. So switching it into the cloud helped, like, relieve a lot of stress.

Yeah. It helped it helped yeah, relieve the the concerns about who has access to this. Right? We just you couldn't guarantee that it wasn't, you know, and and how were we protecting it from malware and cyber cyber attacks. Right? So I think those are just important things that we're There are natural disasters like crazy thunderstorms there too. You know? There's there's always that angle.

One of the things I love, and you mentioned it, from the original conversation we had around Penn State. I love that is, you know, you said we we don't do an online degree. We don't do a degree by this campus. You go to Penn State, you go to Penn State. And it's it's one to be I I actually love that approach.

I think it's so important to leverage the brand that way. And I I all that I that stuck with me after our previous conversation. It's very difficult to do that, I think, from an IT perspective, because you're gonna have a class where you talk modalities, but within that one class section in Canvas, They may be different modalities already. Yep. You may have in person a in person student taking an online class, and that's the only online class they have that's interacting with forty five year old professionals who are only online and taking this at night.

So there's a lot of differences in these systems and having it be the same end product really brings people together. I can see that. Absolutely. You know, my daughter's headed off to college in two weeks. And, you know, she's looking at her calendar and her schedule saying, man, I'm gonna I'm gonna be taking some fully online courses because I can't find a time that fits with work and that.

So, yeah, just a powerful powerful approach. I remember standing in front of our, academic leadership council in the weeks prior to COVID, our shutdown. And I they said, you know, where where are we with our systems? And I I said, you know, canvas is is on, the cloud and our SIS is, you know, we're we're I think we're in really good shape, unless the entire, you know, the whole first eastern half of the United States goes remote at one time. At little did I know Ray? It was like being ridiculous. And I said, I'm I'm worried about that scaling piece, but it turns out we, everybody was, was sort of ready for that, and we were able to scale in interesting ways And then when we came back to campus in the fall of twenty twenty, and even the spring of twenty twenty one, that mixed modalities was within classes even.

So we we taught one hundred forty students, but we were only allowed to have seventy in the room. So half were online via Zoom and half in participating through Canvas and half were in person, and then they flip flopped. And then some students didn't come at all. And so it was a really interesting way to see not only how the systems approach that, but how faculty challenge themselves to think about their pedagogy differently, which, which I think is going to be a lasting impact for. Yeah.

And in a situation like that, how do you collect holistic student data? How do you collect data from all those different approaches in order to create a consistent picture for your educators. Right? Obviously there's a certain amount of data in the LMS. We've got lots of plugins and lots of different tools that help create that picture how do you make sure that you've got that data for every educator for every course? That's a great question. But before I answer that, I want handy to answer, how many LTIs do we have at Penn State currently? So we put another one in -- Oh, no. -- this week, two in last week.

And so when I put the slides together, I had thirty plus, then it was forty plus. I think we're closer to fifty, and we just keep on adding. Yeah. That that's actually reasonable. Are there, you know, for a lot of instances to stay in the tire? We make every LTI tool sign a a a addendum -- Yeah.

-- contract with us. So slows it down considerably. Yeah. But it's still a good good size amount. Yeah.

So I mentioned this because the activity data that we generate our, our analytics from is based on the Caliper standard. Caliper is a one ed tech, formerly IMS Global. I think that's how you have to say it now. Is a is a standard for, you know, It's a it's a standard beyond sort of on, basic click stream. It sort of represents, as specific, types, the different types of actions that students can take, whether they're a quiz or, posting on a discussion forum or stopping and rewinding a video.

All these types of actions that students are doing in various tools can be mapped to a specific type of event in Caliper. And this can then be used by any sort of vendor to be able to explain what students are doing in their tool. And one of the cool things that we do with, course insights is we aggregate the data not just from Canvas, but from the other supported learning tools because we're relying on that underlying caliper standard. So here's a graph that is actually, telling a similar story to the graph that I shared previously where a student was you know, started engaged at the beginning of the semester and then slowly disengaged throughout while the rest of the the course, kept on going. But what we see here is we engagement in this course was largely, through Canvas and top hat.

For those who don't know what top hat is, top hat is a sort of an active engagement platform so that students, that the faculty can ask questions in the course and then students can use and use their phone or use the LTI to sort of, to answer questions It's a very common sort of, engagement strategy with faculty, especially in, our STEM courses at Penn State. And so what we see here is activity that's sort of split between Canvas and Top Hat And then somewhere around, oh, I can't see the graph, but it's somewhere around, let's say March. The the top hat data stops before the Canvas data. And so why is that? Just kidding, I'm not going to make an answer. So the reason for that is that the students stop coming to course.

If there if top hat data was being was a student doing engagements in the classroom. This is a student that stopped being engaged in the classroom, but it still sort of kept along with some of the canvas assignments at home, and then a few weeks later stopped. This adds, you know, dare I say another level of insight to understanding what a student has and that's sort of the power of caliber data is so that we can have a sort of transparent, and this can enable us to enable new types of alerting in the future of because faculty are always in want to know, like, well, what what what tools are my students actually using? I have content in Kaltura for videos. I have content in top hat. I have content in I don't want to go to each one of their own dashboards and look, I want some place to just tell me what is this student not using that the other that everyone else's or, and this allows us to be able to get to this type of, of insights when we can be able to look at all of these tools in a holistic way to understand the the student experience.

So it's one thing to collect the data. Right? Anybody that's been in this room has you've done all the work to pull it all together, get it into one place, do some analysis, experiment. Then it's how do you make it actionable? Right? Do you automate any of this approach or is it all kind of left to the educator to to the right outreach? So that's a that's a good question. You know, there's a reason that we're, we're called the data empowered learning team and not the data driven learning team. You know, the whole thesis of our team is what data can we put in in front of a faculty or, an instructor to be able to, understand what's actually happening with the student at another layer.

In fact, something that's not shown in the slides is that every time that, you know, through our course insights pilot, every instructor that we add in goes through a training session. And so when we are training, instructors or advisors, how to use our tool, We always say that, you know, this is just another data point while you're doing your triage, while you're doing your understanding of where a student, student is. You know, you know, in their, in their career. So, you know, a student with low activity does not always correlate to low performance. Sometimes a you know, for example, I teach first year students how to program.

And in my course, you know, sometimes I have students that come in from comp sci. And so they have very little Canvas engagement and very little, you know, and sometimes very little attendance. But, you know, their grades are fine because they already know the content. Other cases, I can see another similar student who is struggling in their grades and is also not, not you know, not attending and also doing low. And those are the students that I that I that I have that extra knowledge and that I reach out to them.

So we always train advisors, and, faculty to use what else you know about this student to inform the decision. You have many different data points available to help you understand This is just a new one, and this is how you can use this to help. Ben, I would add that, within teaching and learning with technology, which is the group that, data empowered learning team is a part of, they have something called faculty fellows, and those are a group of faculty that have, done sort of a deep dive, one year engagement with TLC. And we joke once a fellow, always a fellow. And so we bring those faculty in to give feedback on and provide insights into how might this help, how how else might I do this? So that feedback cycle isn't just about the training piece and how folks are using it, but how to, how does their use of it inform, what they're doing? And we love that because they become champions for this in their own colleges.

Right? So they might say, gosh, did you know we have access to this information, and we can do interesting things with it. So, it's part of a greater team that also has a group of instructional designers that with faculty, Ben mentioned that they they carried, the watershed of what we needed to do going into, our remote learning for COVID, and so they're always championing, championing, being champions for. The, you know, the additional tools, the additional in sites and how not just being able to look at this, but then how does that actually inform and change how we're teaching in our classes and how we're engaging our students. That kinda gets us to the punch line, which is, what has been the impact of insights, on Penn State. Well, I think of this in a couple of ways.

So I know we have a slide that puts a bunch of numbers on the screen, big numbers always impressive, but I actually don't think it's the numbers that are here that are are are are some of the takeaways. In fact, I think of it in two levels. One is, you know, the old adage of, you know, it can take a long time to turn a big ship. And Penn State is a very, very big ship. And so, you know, as, you know, as you were alluding to, you know, we have faculty partners and one of the challenges, you know, with changing and empowering a data culture is is getting this in front of faculty that have the ability to make the change.

And so we started this pilot with only about ten or twelve of those faculty fellows, and we naturally grew this to a hundred and fifty six. That was just out of word-of-mouth. That was out of a desire to say, we would show this to one faculty, and they say, oh, I want to show this to all the rest of the, the faculty in my math department or my English department. And I want all of my cohort to use this and then we, you know, so that we can build our own faculty learning community where we can figure out how we can use the data that you've made available to improve, you know, situations for their students. In fact, just, just last week, I heard from a faculty group in Penn State Harrisburg where they're trying to use our course insights, and our advising to elevate to, figure out how to support first generation students at that campus and how they can use that to help monitor students in, you know, in real time.

And I think that real time element is also the sort of, big, you know, Denu ma moment here because you know, when we think of institutional research, institutional research has been around for decades and they've, you know, they've helped student success by looking at trends and courses over time, seeing DFW rates, AB rates, etcetera. But the problem with those sort of metrics is that they're always lagging in Right? By the time you have a DFW rate, the semester's over. You can't help the student anymore. But with learning analytics, you can devise new strategies and interventions to figure out Okay. We know the students that are struggling right now at Penn State.

And so the challenge for leadership oh, the big bold challenge for leadership is to lose my thought. No, the big challenge for, for leadership is to figure out how they can develop those strategies and support that in real time. And that's That's where this work is going in the futures. Leadership that we have at Penn State that's like, how can we support this? Because, you know, in institutional research, there's best practices. But in learning analytics, no one quite knows how to use this data yet.

We've and so it it takes people willing to take a risk and figure out how we can we can do that within the safeties and confines of following the advice from the registrar, and the privacy office and our ethics office, and our student affairs office following all those policies while still sort of slowly pushing as you'd say the bleeding edge. Yeah. And it's funny because we, you know, there's we see headlines about, oh, increase federal regulation, increase reporting, things like that. But if you're already actually creating the data and using it for stuff like this, makes that you know, you're already on the front foot when it comes to, those kind of emerging, challenges. Right? Yeah.

Yeah. And that that's part of it too, because this data is data the faculty already have. It's just not aggregated and consumable. Right? They can get activity reports out of Canvas, they can look to see what grades they're putting in, look at each other LTI tools and see how the students are interacting. It's just canvas and getting the data over and using things like CD two makes it easier so we can aggregate and change the view.

So it works for Penn State faculty. It may not work for another school. Their faculty may need different support processes, but for us, it works. And I think one thing that, as we think about the advising process as well, at Penn State Advisors may have five or six hundred students. Right? And so this provides a way to provide those insights in a way that's easily digestible and actionable.

So they can say, gosh, these are the twenty students that need my attention the most as opposed to having that scatter shot looking at this entire five hundred. Yeah. Just repeat that one advisor, five hundred students. That's that's not uncommon. Right? Yeah.

That's one of those areas that AI can help It was an AI plug. One more. We we'd love we'd love to talk with you about that. But but it is one of those things where prioritizing that many students and really identifying as really as possible at risk students, it could make a huge difference of whether a student's gonna be successful or not. Right? Yeah.

One last slide we are we would be remiss if we did knowledge that you, received an award recently? Yes. So, as a result of us, you know, that impact of how we're trying to, understand a more holistic student experience based on the Caliper data. We recently won, the platinum award from from one ed tech formally IMS Global, for our, course insights application. And so that was a big honor that we received in June. Yeah.

Amazing. Oh, and if you, afterwards, if you go to my LinkedIn page, we have a very short five minute video that gives a summary of course insights, where I impersonated Mr. Mister wizard, and instead I was called mister analytics. Ended a little five minute, little talk with instructor Sally, talk about how course course insights can be used in the classroom. Yeah.

And that that's one of those things as a follow-up. If you have additional questions, if you want access to to demos and videos like that, You know, reach out off to, to Brandon Andy, track him down the hallway, you know, reach out to Jen and and experience, understand her background and and experience there. These are why we're here. That's why, you know, being back in person is the opportunity to have those side follow-up conversations and and, you know, continue that conversation for all of us. So, parting words, we've got a couple of minutes left, from a wrap up, Jen.

Do you wanna do you wanna wrap up final thoughts? Yeah. I'll I'll start with this. I'm gonna come back to your Venn diagram piece of this. Right? So, I'm gonna also give kudos to Andy. They they finally shut down the Angel servers in December from our transition over to Canvas.

Right? It was very exciting. Very, very exciting. I I wasn't invited to the party. I'm I'm not sure. The funny part was you gave me advice to never tell We were still running those because they did not wanna hear that we're still spending money on Angel.

Yeah. But we had to because of our retention requirements. Right. Right. So I would say like this, this idea that we've got, we've got really interesting, questions around the data.

We've got nice, curious people who, on the team, who are thinking, how could we look at this data in new and interesting ways? What are the probing questions we could ask that both of the teams, work extensively with faculty, with advisors. I don't know, Ben, how many times have you presented to leadership even since I've been gone, it's it's been numerous, right, to reassure them that we're we were we were good and well intentioned. And now coming to AWS and seeing how AWS is working with folks like Penn State and other institutions to do more with the data, to get better insights. And again, it comes back to what does a good faculty student experience look like? What does that look like in the twenty first So I just want to thank you both. It was such a pleasure to get to work on that team with you, but it's also such a great pleasure to be, supporting that work in my new as well.

And thanks for letting me moderate the panel. I appreciate that. And I do wanna say thanks again to AWS for being a Diamond sponsor. And and allowing this to all happen. We couldn't do it without you.

And so I also wanna remind you that here are, some some LinkedIn links you're able to click on those and, connect with any of us. I'll leave that up there for a minute. And I think I've got a slide somewhere. I don't know if I see it on this one. Let me see.

Can I go back? Oh, there's some, yeah, there's some extra content slides that show some more perspectives, of course, insights. So Let me skip through those really quick. Yep. So this is more just a lot of the other things that you just blur through were some of the demographic data that showed facts about the populations of students in the course so that students so that faculty could learn about where students were coming from and their learning pathways or what the basic demographics look like. Yep.

And of course, add us on LinkedIn, but most importantly, scan this QR code, so you can get your badge. This is one of our our highlight sessions, spotlight sessions, that let you, continue along this pathway of, credential success. Thank you again for joining us. I know it's been a bit of a long afternoon, and there's, thunderstorms going on outside. But, we can't wait to continue sharing information with you, you know, this evening and then into tomorrow. So have a great day. Thank you.