Generative AI in Education

My name is Bill Bales. I'm with Microsoft. And I'm the chief strategist for our education technology companies in our education vertical. And it's so great to, to meet so many of you at our booth and about some of the great things we're doing with our team's integration with Canvas, and we're also, proud to be here with, in structure for all all things Canvas. Today, we're gonna talk about generative AI in education and is such a, hot topic as everyone knows. And, we're excited to be part of the forefront of the thought leadership around what that looks like for you as educators and and technologists.

And today we're gonna listen to, Doctor. Michael let me try that again. Can I try that again? Okay. Doctor. Michael Jabor.

And he'll give you a little bit of background of of where he's come from. But we're gonna dig right in. And just so you know, we don't probably won't have time for questions. So if you do have questions, We have our Microsoft staff over here on this corner, and you can scan our badge or come talk to us, and we'd be glad to do so. So with no further ado, MJ, Thank you, Bill.

Alright, folks. Thank you so much for having me today. Just TJ, just raise your hand, LTI, TJ? Okay. In case you guys have questions about that. After as well.

Alright. So there's a lot of content here. We definitely don't need a presentation in order to have this conversation but it's a very important one. I'm the chief innovation officer for Microsoft EDU. I cover k two twelve higher ed graduate programs, including academic medical and, have been integrating with worldwide probably more than I should be recently but, this conversation has, takes very, very informs in each of these organizations, in each of the roles in these organizations.

So I'm gonna start you here. Just raise a hand you've seen this image. Hopefully, most people have at least seen it once. It's rubens based. It was an illusion built in the nineteen fifteen to demonstrate, serial information processing.

So when you look at it, you will either see two faces or a vase. You will not be able to see both of them at the same time. The brain searches for clarity when there is ambiguity. You cannot have both perceptions. You can try but your your mind will flip when you're looking at these images.

Not non, AI, non cloud thinking, and AI thinking, cloud thinking are very different. If there's a broad range of applications for these tools, but they're different. When I'm speaking across the country, one thing that I find is that developers are extremely difficult to train communications people. Perfect. Executives.

Great. Marketing people the best. But developers, the worst. They have a really hard time making this transition. So I'm starting here.

Now where do we fit in all of this? It's certainly questioning our humanity. There's been some studies recently even put out on the medical side where they had patients that opted in to receive difficult news via chat. And they didn't know whether it was a human or a chatbot behind. It was a blind several blinded studies and these are we're talking about cancer diagnosis and other types of difficult, news. The, humans that were receiving that news were given a survey after these sessions and they persistently ranked the chatbot as more useful and more human.

Right? Empathy was one of the words they used. So when when when we If we think that our intelligence, our ability to feel pain, differentiates us, I think we should think again. Now this is a hallucination. It's a, the heart and the brain coming together vivid and alive. And what you can see here in the, upper right hand quadrant, and if this this laser will work on the screen, but you can see here the little brain attached to a heart.

It's sort of in a Valentine's heart. You've got a tree with a little red leaf symbolizing a blood vessel, I'm assuming. You have got moss growing, which I'm, again, I'm guessing it means, some form of life. Right? Now I typically run new exercises before some of my talks. And I ran that in March.

I ran it without adjusting any of the training parameters, and this is just two months later. It approximates an actual human heart. You can see that they're actually using, heart tissues to create brain structures. It's now weaving in elements like the lung. Right? So very, very interesting, without training in just two months This is going to go fast, and it's going to have us question what makes us human.

It'll have your kids question what makes them human. And so I think it's important that we have an answer to that or at least a feeling about it. Now we're gonna talk about hallucinations. In in a bit. But hallucinations can also be used in very practical form.

So this is a photo realistic, modern New York City classroom. Where, it has drawn out the the actual classroom with desks and smart board. It it's fully populated the the New York City builds three hundred schools a year, and they design three hundred schools a year. Looking at something like this, the amount of money that spent on design. This will be an incredible game changer for New York City School System, and this is again without any training, without plugging in Cads or any other files.

So where are we in all of this? I I get texted constantly from CIOs and CTOs, I'm typically known as the phone a friend at Microsoft because I came from the New York City Department of Education. I I'm the former, chief technology officer. And, when I get tech these texts and calls, they are typically with, questions that are frustrating them, but they don't have, good answers to. So I got this question, more recently and I said, this is a great question for chat GPT. As AI systems are getting more intricate, they become harder to understand.

So how can I easily explain this idea to others and create a visualization? Sounds like familiar conversation familiar? So, chatty BT came up with a great response. It says imagine that you have a an orchestra with only a few musicians in it. It's very easy to understand each musician's role and the difference that each musician makes. When you have a larger orchestra with many musicians, it's very difficult to pinpoint what one musician is doing. Let alone the changes of that which one person will make in the overall, piece.

Now I asked it to explain this, visually And, it had me draw this in Excel. So, on the columns, you'll see complexity going from ten to a hundred and explicability from a hundred to ten on rows. And what it was trying to do was draw a complexity, explicability, trade off map. Where it showed the amount of connections. One of the reasons why is because I constantly prompt it until I'm a neuroscientist and I was asking in the context of connectomes.

So it was showing me the artificial neural networks and how at a certain point they hit mathematical complexity to the point where it's no longer worth understanding. Right? I said, this is a great grid, but I Can you explain it? And it went ahead and provided an excellent explanation. And of its own diagram. And, it was describing how it was trying to show going from the upper left hand corner of low complexity, high explicability, to the opposite in the lower right hand corner. So now taking a look at this, this will change everything.

It's gonna change the way that we work. It's gonna change the way that where we, operate in schools and also the way we are at home. Planning a vacation even. Right? Now, I was just recently, actually, on vacation. And, when I come back, my meetings are recorded now through Copilot.

I get summaries instantly. And even Outlook, Outlook behavior is gonna change for each of us you're here, you may be here for three or four days. You're gonna go back to your desktops. You're gonna ask outlook to summarize all the emails that you got over the past week. And then itemize the things that are assigned to you.

And then the things that could be delegated to delegate them out to the correct people And then the things that are assigned to you that you can't delegate to have it start that work. That is not the future. That is here. Right? So I'm gonna show you a little demo of Copilot just so you get a sense of what it looks like for those of you who have not seen this. So you have this crying inside watching as it's okay.

Sleep wasn't activated. Excuse me. So that's Microsoft Office. If you are interested in seeing the Windows version of that embedded into Windows, you can also find that online. We're going to pick up the pace just a bit.

And, I break its conversation that I have with folks into two parts. And just as a level setting, everything I'm giving to you, I'm I'm giving it to you in a way that you can give to others. So you can speak to your programs. You can speak to your cabinets and feel confident about I believe you'll have a copy of the the slide deck also. That's helpful.

The two parts of this conversation are what I call vegetables and dessert. Right? Desert is AI in practice and vegetables are the things that you need to level set on. There are many terms that you hear very frequently, and we're gonna do some quick level setting now. So AI and, large language models. AI has been around since the fifties, if not earlier, probably even since the late forties.

Large language models are the higher order cognitive systems that are trained on internet scale data. GPT stands for generative pre trained transformer. Transformer is the type of AI that's used. It's a natural language processor. Pretrained P on internet scale data g in order to generate.

Right? Now when we look at a brief history, again, starting in the late forties fifties, you had just training computers to do tasks that humans would otherwise do. Shortly afterwards, all the way up until nineteen ninety seven, you had machine learning, which was primarily for predictions and decisions. And that was, you know, towards the ninety seven is when they started to really apply at scale computing power. Starting in two thousand and eleven, although up until two thousand seventeen, was the creation of the artificial neuron. Mimic to after human biology, creating an artificial network, neural network, a black box of statistical processing and predictions That artificial neural net enabled us to adapt to predictions.

And then most recently generative AI twenty twenty to twenty twenty three, and we're gonna see this in reverse, later in the presentation. Now taking a look at the various forms of AI. You've got anything from natural language processing like we discussed. To knowledge based systems, different types of, training AI, like supervised or unsupervised. The vast majority of what you're seeing right now in natural language processing is unsupervised.

There's a minimal amount of supervision that's enabling it to do what it's doing. So you will undoubtedly see various forms of AI beyond chat GPT continuing continually evolving. So you saw that, Lama, is now hosted with us. You have Orca that are new like baby models of chat GPT models that chat GPT created. So lots coming.

Now what can they do? You've seen some of what they can do already but they can, generate content, they can hyper personalized content. They can even blend disciplines. So I'm a physician or act as a physician, act as a physicist and describe my patent invention to a software engineer Right? Instantaneous summaries. Taking a look at extraordinarily complex cross discipline content and boiling it down to something an eight year old can understand. Code generation, I think publicly, we say forty to fifty percent acceleration with some of our partners.

We're seeing close eighty percent acceleration in code development, which means eighty percent of the code that was previously produced as of a couple years ago could not be written by AI. And then advanced search, search by image, search by word, search by subword. So a lot happening when it comes down to services as it is today. And remember, it's in its infancy. So OpenAI and Microsoft are not the same company.

Microsoft invests heavily in Open AI. Microsoft's mission is to, empower every organization and person on the planet to achieve more And Open AIs is to ensure the artificial general intelligence benefits humanity. Just raise of hands if you have heard of auto GPT? Okay. Couple. So auto GPT is when you can take one chatbot and turn it up against another chatbot and have them go at each other for, like, some, some of whatever purpose you get it.

Raise of hands you've ever heard of Chaos GPT. Okay. So you guys need to do some research after this. KAS GPT is a form of auto GPT that has one purpose to find every possible way to destroy humanity. And more are coming.

Right? So supercomputing at scale has its risks and also has its benefits. It's important that we as humans understand what they are. Now taking a look at the different models that we have currently available, and I am sure this will change practically from week to week. But you've got GPT three point five, and chat GPT, which are used for, lower complexity, reasoning, a smaller amount of input, smaller amount of output. It's highly economical.

It's it's immediately accessible on chat GPT at no cost. You've got GPT four, which is large tokens, high complexity. These are where you're gonna put all of your medical questions. You're you wanna process an IEP or a neuropsychy vowel. You might wanna use GPT four.

Alright. Codeex is for coding Dali is for image generation and Vali is for voice. In less than three seconds, your voice can be cloned. And we were actually playing with it internally trying to figure out if we could detect each other's voices whether they were AI or not. It is very, very difficult.

So taking a look at where this fits at least on the in the Microsoft world, on the application layer, you've got Microsoft three sixty five edge being windows all on top. You've got the power platform, which is our low code layer. So if you wanna do analytics and power BI, or create a script and power automate or a power virtual agent. And then one thing I I remind CIOs and CTOs all the time is Azure, AI cognitive services is way more than just open AI. And we're continually evolving it.

So the cognitive services are all the things that you see below. Applied AI services, a cognitive services and machine learning. And now with fabric, Microsoft fabric that was just released, You can do pretty incredible things like, relatively low compute data warehouses, instantly aggregating all of your data without having to build a whole data warehouse. Creating virtual data warehouses, anonymized data warehouses, stringing together various forms of AI and code in order to achieve an end result all in one ecosystem, multi cloud, multi AI. Right? So different worlds.

Now you've seen a little bit about this already because I've given you, a taste of it earlier. But for example, write a tagline for an ice cream shop. The prompt will give you. We serve up smiles with every scoop. Or give it at some table names, and you will get the SQL query.

I use this for Excel. I don't love writing all those complex sum if s statements. They they drive me mad a little bit. So I will use ChatGPT to write those statements for me among other things. And then, of course, a ball of fire with vibrant colors to show the speed of innovation at our media company, and then you get an image.

Right? Now, so the one of the most frequent questions I get when I'm visiting, organizations, but I would say, particularly in higher ed It's how do we code our value system into a chatbot? I don't want this thing running wild. We're a liberal college. We're a conservative college. We're a medical college. We're an engineering college.

We don't want this thing running wilds. How do we code our value system? So meet Dana. Dana is a conversational agent. What you see on the left is what we call meta context. And, Dana works for the marketing team.

Dana understands, the strategic goals of the company, unique products. Dana's responses should be informational and logical. Dana should only use these data set and should not use other datasets. And Dana should be safe and free from harm, non controversial. So then we write a tag.

We ask you to write a tag line for an ice cream up. That prompt goes against data, Dana, and the response is scoops of heaven in the heart of Phoenix. Now if you wanna further train Dana, you can ask it to generate a hundred, a thousand of those prompts, edit those prompts, and then send them back to Dana to Learn. What to do and what not to do. Now in one of these conversations, I mean, I was in a room of three hundred people and this is one of the questions they asked was scoops of heaven in the heart of Phoenix.

I don't wanna use the word heaven in my response. So two ways to do that. One is to not code it in the meta context. And you can see there's no actual code here. You can see how it's written.

These are these are actually how it looks in the Azure portal. Or you could train it to not use that language. Right? So you could say no religious references or training. But value systems are very important. Trust in governments.

So the last piece of the vegetable slide, I think one of the more important, elements of this relates to, privacy and security and what does it all mean? How does all work? So This will hopefully answer, a bunch of questions that, at least I get very routinely and if there is time, can tackle them after. But your data is your data. We are not training OpenAI's models, and we are not training Microsoft models on your data when it is hosted in Azure. If you're in a public, large language model, I would think twice before giving, any kind of, you know, private information like student name, student information, IEPs, neuropsychy vowels. And you wanna completely anonymize them before using a service like that unless you have something inside of a enterprise environment like this.

Now, these systems are protected by multiple walled gardens. Not just between customers, but even in its own environment. So you might have one environment that's a HIPAA environment, one that's a ferpa environment, one that's a NISTen sock environment, not only will they not see each other, the model does not have access to traverse those those data sets. Even the cache where you're operating with the model, it stays inside of your environment. It's protected by network control, protected by roles based access control, and encryption at data level, we couldn't access your data to help you even if we wanted to.

Right? So this leads to another conversation about responsible AI. It's really important to for your organizations to consider a responsible AI framework. Whether it's Microsoft or one of our, one of the other companies out there that's offering a framework like this, you should adopt one. They are typically, pretty straightforward. I'll I will give you, show you some of the items so you can inspect it in a moment.

Basically, it'll go anything from privacy and security to accountability, fairness, and beyond. Do not go big. Start small, even a one pager. This is what you should do. This is what you should not do.

Do not put PII or PHI inside of, inside of a public form of a large language model. Right? So stay simple in getting a simple policy together and then leverage one of the responsible AI frameworks that are out there. Now to inspect a little bit about what that means, you're gonna have everything from fit for purpose and human oversight to, a minimizing stereotyping and bias as well as things like accessibility, security, and privacy. So the the framework, the responsible AI framework will help your organization at least just get a scaffolding in place so that you can start the journey. Of what's okay and what's not okay with AI.

Having a responsible AI framework does not make you a responsible person. It will not make teachers and your school responsible people. This is we're going to learn this experientially. Right? I'm not here with Microsoft marketing slides. I am here with examples because I want you to see and learn and understand and you're gonna be doing the same thing in your programs.

Right? How are you commenting on student performance? When is it okay to use student performance? What's an acceptable review look like? What about developing curriculum or building rubric? What's okay? What's not okay? And when and where? So that's the vegetables. I now want to go into the AI in practice. And get a deeper sense of what's really going on here. So let's reverse out. I committed to you that would explain what a hallucination is.

So here we go. Starting at the bottom, I I reversed this into Maslov's pyramid structure. So that you can't forget it ever no matter how hard you try. So you're gonna have raw input at the bottom. This was like nineteen forties and earlier, the ability just to store data.

Right? Now you're talking about a nineteen fifties sixties which is effectively just pattern identification. So taking a look at what data modeling and processing is is Can I see the patterns in the data? Can I see the relationships in the data? Now let's apply that first artificial neuron where we started to mimic human biology to the study of patterns and relationships on data. And so this is your your current day wrapper, what it looks like. So now you have the ability to adapt beyond adaptation is your large language model, which is a higher order cognitive system that enables you to talk to your data. One way to look at chat GPT is an interface, not just an underlying technology.

Right? I I didn't think, you know, six months ago, if you had told me that you wouldn't be going to search engines very much if at all, I would have probably laughed at you. But I I aside from searching for the local pizza restaurant, I really have not done much searching. And I think the reason for that is because we are in a shift. We are shifting from the search for information to the search for answers. Right? Now taking a look at what this higher order cognitive system produces, it produces an initial output.

And that initial output is gonna be a combination of a fact and a hallucination. If you want to write poetry, build the artwork of a human heart, you're gonna veer towards hallucination. If you wanna build a self driving car, a medical instrument, or teach physics, you're gonna veer towards fact. It requires both. In order to create an inspired output, the ones that you see.

Great book to read by a controversial physicist. Wolfrom. He created a tool, Wolfrom alpha, So he has a it's it's online. You can order it as a paper book, but you can also just get it online in a blog format but it's more than a blog. I warn you.

It's called What's chat GPT. It's one of the best explanations I've seen. Where he talks about the nature of these tools and how they really are just giant prediction machines, predicting the very next word that's going to come out of the machine. He leaves only one line out of this pamphlet, which is that humans are also giant prediction machines. We're just far more efficient.

So, when I've been talking about on the ethics side, getting a lot of questions from parents, teachers, administrators, How do I think about this? This image was generated in Dali. And, I'm gonna give you, a way to visualize it. Imagine giving your most rooms that I go into. Seventy to eighty percent of people who have experimented with chat, Greeti. Some successfully, some unsuccessfully.

But the vast majority of people have had some sort of interaction with it, even higher numbers with kids. Right? Now imagine giving a kid a small wooden table and putting it next to them. And on that table, you put a nail, and you tell the child with the hammer sitting six inches from the nail, you tell the child push the nail into the table and push hard, but use your thumb. Whatever you do, don't use the hammer. There are organizations that say, sixty six to a hundred percent of jobs are gonna be impacted by generative AI.

Sixty six to a hundred percent. And it's our responsibility and education to bring up the next generation. I have four kids from three to seventeen. I worry about what their future is going to look like, I am here partially because I am worried about that. So when I when I think about this, and I and I see teachers gravitating towards using it for creating curriculum or creating rubric or, creating exercises, making their jobs easier.

Making their life easier. But then telling a child don't do they reminds me of the calculator moment. But let me be very clear. This is not the calculator moment. Unless you wanna consider it a calculator moment for every single subject on earth at the same time.

Right? We're gonna invest our money and our time and our energy, societally, and plagiarism detection, I think we may have missed the mark. The catch and kill is not going to work, in this generation. Right? So we have to think differently Now what am I seeing, in terms of trends? I am seeing a few big themes, you know, without going into any each bullet, the big themes here are hyper personalization and accelerated learning. That is a a a common common theme, a lot of common questions that we get around that, improved efficiencies. How do we make teachers' lives, more efficient? How do we make schools operate better? How do we get schools to spend more administrative time with the students to to leverage and reshift that time? And then lastly, preparing for the future, preparing for different jobs, preparing for job changes, with, you know, anywhere from three hundred thousand and up of even jobs in the tech sector that have recently been released.

I don't know that we're gonna get those back. So we have to really think about where we're going here and what we're empowering our our kids to do for the future. Ultimately, I personally believe that this is an equity and accessibility tool. I have two master's degrees and a doctorate. I can think of many, many, many, many times where I have either given up or wanted to give up.

In subjects. I simply I just wasn't designed for. My brain wasn't designed for. And if I had had a tool that would just help me explain it in a way that I understood and a language that I understood and a culture that I understood, things would be very different for me even. When, we look at some basic prompts just to give you an idea of what they look like, act as a tutor.

I need help understanding quadratic formulas. Now these are these can be even very dynamic. And so you can even say to the prompt engine you could put in there. Now ask me five questions in order to get a better sense of where I am. And then tune it over time and pause, wait for me.

And you will see chat GPT do that. Right? Or come up with ten ways to improve memory and recall for a physics exam or for teachers writing multiple choice questions or explaining lunar eclipse to an eight year old. Recently, I've been getting questions from schools. How do we take our top teachers to help elevate our our entry level teachers? How do we do that? Our top teacher is an English teacher. How do we bring up the physics department? This is the way.

We showed them live how to do cross disciplinary summaries and training and professional development. So just some quick trends, then we'll go into some quick use cases here, before wait, before I get pulled off the screen. You've got, some IT back end cases like HR support, twenty fourseven IT support, project management, interactive self-service chat bots, security. You know, some organizations are processing anywhere from hundreds of thousands to, trillions of security events every year. How do you protect against that? How do you process the signal from the noise? Or in k through twelve lesson planning or being able to provide services, administrative services scheduling, student progress summaries and, and, and more.

And then, higher education that you've got things like course design, event scheduling, academic advising. Even most recently, I've been hearing a trend of way finding inside of some of the larger universities that I have and probably applies to schools also. Maybe difficult to navigate. So how can I have a natural language navigation support? Right? So taking a look at one, example here. Now, this is for a k through this is an actual real life example.

K through twelve, we were looking at, various areas where we could assist, whether it was scheduling, source management, lesson planning, we took scheduling. This was an ad hoc session that I got dragged into after a conference. Someone's like, can we please just have forty five and I was like, sure. Let's do it. So they wanted to redesign the teaching and learning schedule to to creatively adjust, the school day.

They had a theory. The theory was that if they did this, they'd be able to retain more teachers. They'd be able to lessen stress. And make it more, exciting curriculum for students. So we took a look at what are the elements that we should give chat GPT that the things like just the school, the numbers of students, number of teachers, core content, number of minutes in some of these different areas.

I'm not sure if you can see, yeah, you can probably see from there some of the the inputs that we gave Chad GPT. And then boom, instantly started producing schedules. The chief academic officer, chief operating officer, about thirty of the administrative folks from this particular district were in the room. We then started to decompose that and said, you know what? We we were too narrow initially. Let's reverse out.

So how what are the critical elements? Let's start there. What are the critical elements of an elementary school? How can you maximize fluidity and flexibility of school, regardless of the inputs. And then we put the inputs and get the schedules. But now we tell it, act as a teacher and review the schedule. Act as an ACL coordinator.

Act as a special ed teacher, act as a transportation director, act as the IT person. And it kept refining and refining and refining the schedule. Now this schedule is going live this September for this particular district. This is some of the analysis that they were doing just to give you an idea of what they did after this. This was a forty five minute session.

We produced about fifty pages worth of an illnesses and materials. We produced, the letter to the board of ed explaining the schedule change, the letter to the parents, the letter to the teacher, the text message to the chief academic officer, letting her know that the schedule was ready for review and the key changes of the schedule right there. She stood up and let out some x with us at that moment. So I won't say who it was. But you can see how it was adjusting, core content.

We then I then went back and studied after that session. By the way, I also asked it to identify any area it couldn't fix. And then provide a risk management remediation program for the things that it couldn't fix. I then asked What is the one thing that you think that you that you really just can't even provide a a remediation plan for? It's that professional development. So getting teachers to professional development.

So I said, do you have any suggestions? So it suggested that I elongate the school day by thirty minutes on a Friday and provide pizza every Friday. This is not a joke. This is real. Every when I went back and did the analysis on the transcript and I looked at what ChachiPT had produced. Everything that you see here in black on these two slides were questions that chat GPT asked me naturally to get to the perfect schedule.

It asked me if I had a way to track the quality of changes that I was making in a student's life in the schedule, and then provided metrics for that. It asked me if there are concerns around burnout and fatigue for kids. And teachers or socialization issues, what's the stress level of students? Okay. So, one other example that I I this is more of a higher ed example, but it's gonna apply to you just the same. So I I have a another person, colleague who sent me a question that said, are you aware of this article? Article was titled, a little narcissism is a good thing in leadership roles.

And it goes through his twenty twenty three article and it was talking all about the scientific elements of what's okay and what's not okay. So, I said, you know what? This is a great question for Chad GPT. I went in and I asked it are you aware of the article? And it said, yes, it is aware of the article, which is, you know, it's it's a, you don't know anymore. You can you can give it a PDF with plugins. But, some things it just it just knows.

And, I said, okay, well, I wanna teach a class on it. Can you give me the title of the class? So it said unleash your inner leader, master the art of healthy narcissism. I'm just copying and pasting here, by the way, from chat GPT. Because I I couldn't do better myself. So then I said act as an executive coach and a leadership expert.

And if you had to teach healthy narcissism, taking into consideration mental health and healthy leadership principles, what competencies would you teach? And it extracted from a twenty twenty three scientific the competencies that were required to calibrate leaders so that they would not have unhealthy narcissism but had enough a narcissist system to lead. Give me one minute and then I'm going to you. So I then asked it to produce, the course title, course methods course outline all the associated powerpoints, it broke it up into six courses. It even produced a LinkedIn blog, posting for me. Right.

My total run time on chat GPT was eight minutes. So, just to wrap up and we'll get to anyone who wants to stay for questions. Three main things, that I recommend folks, do It's number one, focus on growing and learning. Improve your AI literacy skills. There's so many so many good trainings available for for many of the cloud providers.

Secondly, data, if I had to point to the one decision that over the last ten years that I've been CIO CTO various New York City agencies, Social Services, homeless services, and Department of Education. Focusing in on data, data quality, data centralization, and a data warehouse. Single best decision I've made. Right? And lastly, go small, provide a small use case and then scale from there, my three, areas of advice to leave you with. So that is it for today, but I'm happy to take any questions.

I think we we can both start there and then we'll go to you. You're next. Question is when, you were talking about the data, just now, the article from twenty twenty one. I was got inputted into, chatJPT or what are those inputted? Is I know it's So, good great question. I actually just straight ask chat, CPT.

Repeat it. Repeat the question. Yeah. So the the question is did how was it aware of the article? And by the way, I will not be upset if books need to leave. It's okay.

So it you there are plugins that enable you to actually provide the link of the research article or the link of a PDF that are directly inside of GPT four. But in this case, I just asked it directly and confirmed that it was aware of the article. Yeah. So I just wanna Yeah. So you still need to teach hallucinations.

So you still need to teach folks how to identify hallucinations and make sure it's not hallucinating. So, okay. Sure. So, we'll wrap up. Why don't you come over for additional questions? Thank you all. Yeah. Thanks for coming.