What We Learned About Using AI to Review a Course

Courses change.

Deadlines move. Assignments get revised. New technologies are introduced. Announcements accumulate. Modules are copied from one semester to the next. Each individual change may make perfect sense, but over time, even a thoughtfully designed course can drift from its original intent.

The problem is that the person who knows the course best may also be the person who has the hardest time seeing that drift.

That practical challenge drove a collaborative exploration of AI-assisted course review involving Instructure, institutional partners, and members of the Instructure Community. We wanted to understand whether AI could help educators review course content against established design criteria.

What we learned was more interesting than whether AI could perform the review.

A second set of eyes

Course-quality frameworks such as Quality Matters (QM) and the Online Course Scorecard for Quality Review (OSCQR) give educators established criteria for examining course design. They help reviewers look systematically at elements such as learning objectives, assessment alignment, organization, accessibility, communication, and student engagement.

But having criteria and applying them thoroughly are different.

Reviewing an entire course takes time and attention. Familiarity introduces another challenge. When you know why an assignment exists, what an announcement was supposed to communicate, or how two course elements are intended to connect, it can be difficult to experience those elements as a student would.

AI offered an interesting possibility: Could it provide another set of eyes?

Using established course-design criteria, we developed and tested a prototype that could review Canvas course content, surface potential issues and positive findings, and explain the reasoning behind its observations.

Then we put those observations in front of educators and instructional designers.

The useful part wasn't the judgment

Initially, one possible use case was institutional course review at scale. If AI could consistently apply established criteria across courses, perhaps it could make large-scale review more efficient.

Testing pointed us toward a different opportunity.

Educators found value in using the reports to revisit familiar courses, reconsider design decisions, and discuss potential issues. The observations could highlight outdated announcements, inconsistencies across modules, or structural changes that had accumulated over time.

But educators didn't simply accept those observations and make whatever change the AI suggested.

They interpreted them.

A finding might reveal something that genuinely needed attention. Another might prompt a conversation with an instructional designer. And sometimes, examining a flagged item might reinforce that the existing design was intentional and appropriate.

That distinction matters.

Course design is contextual. Institutional goals, disciplinary norms, instructional approaches, student populations, and an educator's intentions all shape whether a particular design decision makes sense. Those factors cannot always be inferred from the course content itself.

The AI could surface something worth looking at. The educator determined what it meant.

Explainability makes that possible

That also changed what mattered in the tool's design.

A simple judgment wasn't particularly useful. Educators wanted to understand why something had been surfaced and how the observation related to the course-design criteria being applied.

That visibility made the output something educators could interrogate rather than simply accept.

And that is especially important when working with large language models. LLMs are probabilistic, context-dependent systems. Their ability to synthesize information and recognize patterns makes them useful for examining complex course environments, but they aren't deterministic rule-enforcement systems.

Treating an AI-generated observation as a prompt for review rather than an authoritative evaluation better reflects both the technology's capabilities and the nature of educational expertise.

AI can make expertise easier to apply

The collaboration ultimately shifted our question to something more useful:

How can AI help educators review their own courses more effectively?

The strongest use cases we observed involved instructor self-review, conversations between faculty and instructional designers, and structured opportunities to revisit courses that had evolved over time.

In these situations, AI was helping people apply expertise across an environment that had become difficult to see clearly. This is a useful way to think about AI more broadly. Rather than handing judgment over to it—make thoughtful human judgment easier to exercise. 

When time, scale, complexity, or familiarity get in the way, AI can help.

Check out the research write-up here.