In theory, my students have been working on their start-up projects for months, but I’m under no illusions that any except a diligent few will get serious before the last weeks of term. As deadlines loom, I expect them to use any available timesaving technology to help develop their work, which probably means large language models.
So, I’ve been thinking about how to approach this upcoming assessment.
I’ve been showing students how to get the most out of artificial intelligence in my classes, and assume most will use it. I’ll also assume my students can do better than an AI. The assessment is an 11-week project to build a start-up. So, while the output is a simple pitch deck, the ideas are complex and require creativity and context-dependent research. Agent-based tools are out there, of course, but I remain confident that the humans can win this game.
Given these assumptions, my plan is simple. I’ll upload the assessment into ChatGPT (as it’s the tool that is most popular with my students), turn on all the deep-search settings and ask it to write the whole thing. I won’t curate or edit.
This should give me an output that is pure AI, not “AI as a coworker”. The robo-presentation will then become my baseline for marking. That is, I’ll use the level that the AI can produce out of the box as a fail or “basic”, according to the marking scheme.
I put the deep research tool to work and make a coffee; it takes 20 to 30 minutes to run.
- Spotlight guide: Bringing GenAI into the university classroom
- Should we kill the essay?
- How to enhance the accuracy of GenAI models
The result is alarming. Simply put, it is quite good. The system has identified a problem area and come up with a decent business idea. It has references and a professional script. I copy the slide plan into gamma.app and the script into HeyGen to create a presentation and avatars.
The resulting presentation is probably sufficient to beat a lot of students’ work. Undergraduate degree students are still learning, so they aren’t expected to get everything right. I realise that these new deep research tools seem to have nudged past the lower bound of undergraduate level during this term. I could not have planned for this.
This presents a challenge to how far in advance most assessment gets signed off – which adds another layer of consideration as educators navigate AI.
I try to nitpick holes in the work, almost offended that an AI tool has completed my clever assessment. But the bottom line is that I cannot go with my initial plan. If I baseline a fail to this output, some students will struggle to pass on their own merit.
I need another approach – and an ego check. A tool is available that can, for £20 per month, produce a passable attempt at my assessment. This doesn’t mean I’ve been outsmarted by a computer. It’s just an interesting development that I need to integrate into my teaching.
What exactly is my problem with this discovery? Cheating?
The AI policy for the course is that students can use AI to assist in their work, but it can’t create the work. That would be cheating.
OK, but could I detect it if they use AI? Yes, even without detectors, I find that AI-generated content has a certain plastic mouthfeel. Such scripts lack personality and offer few surprises. The issue is that I can’t prove when students have used AI. They may just be predictable writers. And given this assessment is group work, a certain generic tone is to be expected from merging six voices.
In an ideal world, I would have done this whole exercise at the start of term and shown the class the output. That way they could have all created a full AI project as a starting point, and I could have marked them on the value added, perhaps with interactive or process-based elements. Hindsight is 20/20.
So, what can I do now? I have ideas for next term, but nothing can shield me from the fact that on the current assessment, students could get ChatGPT to spit out something that would give a pass mark in about 30 minutes.
Perhaps I can apply the ideas from my 20/20 hindsight now? Armed with my new angle for thinking, I go back to the problem.
First, the assessment is a presentation, and so the questions at the end can be my interactive bit. If a student simply reads and memorises all the AI output, they will struggle if I ask questions outside the scope of the AI response. What part were they most surprised by? What interesting discovery did they not include in the presentation? What changes did they make to their original idea? Why?
These questions will be hard to answer if the student has skipped to the end of the process. The Q&A also fits nicely into the project-management and reflection elements of the mark scheme.
Second, I’ve seen the students every week as they have been developing their projects. I even have A3 papers covered in business model canvases and prototype sketches. If I use these materials as a factor when marking, that could be my evidence for process-based learning. Of course, an 11th-hour pivot could be a last-minute refinement or it could be a student merely going through the motions in class before using AI for the summative assessment. Questions, though, could interrogate why they pivoted.
Finally, the mark scheme includes credit for logging of AI use that could give a lot of scope for critically analysing their choice of tools. I think relatively few students will have applied deep research already. And I’ve made a big deal about being transparent about my own AI use during the course. This will hopefully tip the balance in my favour, with students staying transparent to get more technology marks. Honesty will get them more points.
So, I have incentives and a plan. I feel better about the whole situation until I remember boxer Mike Tyson’s observation that no plan survives a punch in the face. So, I guess I may well be writing another article after marking the work.
Chris Jones is an automation (AI) consultant/programmer and a visiting lecturer at Regent’s University London, Kingston University, MetFilm School and Ithaca College.
If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.
comment