I stopped trying to detect AI writing. Now I require an audit trail.
read a first year grad students lit review draft this week, and honestly, i wasnt mad, i was just worried.
at first glance it looked totally fine. smooth writing, citations in the right places, that polished academic tone we are all getting used to seeing. but a few pages in, it started to feel empty.
the topic was chemistry, so the details really mattered: mechanisms, reaction conditions, catalyst systems, control experiments, and which evidence supports which claim. the draft kept saying things like one paper showed improved performance or another gave mechanistic insight, but when i asked what the actual mechanism was, or which experiment backed up the claim, the student couldnt really explain it.
and that was the real problem.
not the fact that ai may have been used. i am not trying to ban ai in my lab; i actually think it can be useful for finding papers, organizing sources, comparing studies, pulling out experimental conditions, and spotting connections across a messy literature.
the problem is when the step between reading and writing disappears.
a literature review shouldnt just be a smooth paragraph generator. especially in chemistry, the work is not just find papers and summarize them. it is understanding why a result happened, whether the evidence actually supports the authors interpretation, and how one claim connects to another.
so i changed my labs ai policy.
i told the group that i am not interested in playing detective and trying to decide whether a paragraph was ai generated. that feels like a losing game. what i care about is whether the research process can be traced.
the new rule is pretty simple: if ai is used during the literature review process, the student needs to be able to show the path from search to source to claim.
zotero is still required for reference management. for data analysis, no black box ai stats tools. use r, python, stata, or whatever else u actually understand, but u need to know what the code is doing.
i dont mind ai tools for literature discovery. honestly, i want students exploring citation networks, following related papers, and seeing where different interpretations disagree. litmaps is useful for getting the big picture. i have also been testing sciclaw lately bc i like the idea of an ai assisted literature workflow that leaves a trail: what was searched, which papers came up, what was extracted, and where each claim came from.
but the specific tool matters less than the principle. if i ask where did this sentence come from, the answer cant be a blank stare.
for writing, i am also asking students to write the first draft themselves, not generate the whole thing with an llm. i am completely fine with the reverse workflow: write your own draft first, then use ai to critique it. something like, act as a skeptical chemistry professor and challenge the mechanistic claims in this review, is actually a great use of ai. that kind of prompt sharpens thinking instead of replacing it.
maybe this sounds strict, but i am becoming more convinced that research group ai policies should focus less on detection and more on traceability.
the issue isnt whether students use ai.
the issue is whether they can reconstruct how they got from a paper to a claim.
i am honestly happy to see my students using ai, but i also lowkey worry about them getting lazy in their learning and their thinking. i hope this new setup actually works. let me know if u guys have any other good ideas for this.