I haven't had the time to read it yet, tbh. Is it any good?
I like the simplicity of how Gemini pulls it off, and the result does look coherent at a glance.
Conjuration chat here:
I just realized rubric writing is basically solving a semantic Rubik’s Cube.
On my first real task, I found myself chasing errors in loops. Having already logged the recommended 1.5 hours.... aaaand apparently piling on pro bono time, with no end in sight. Questioning my choices. Doubting my coherence. Begrudging the automated system. Fuming at the whack-a-mole game I'd never signed up for.
Fix one issue, surface three more. I was treating the rubric like a linear checklist.
Then it clicked.
I had so far just “sorting” it. Like lining up one face of a cube while scrambling the rest. But I was supposed to weave it all together. Unscramble the pattern.
Once I started treating it as a constraint system instead of a sequence, ie a semantic rubrik's cube I was trying to solve, everything aligned at once.
AutoGrader flipped green across the board. To my own surprise, really. I literally stood there gawking in disbelief. There might have been of a vacant eyed,drool dripping situation going on briefly, before I erupted into cheers.
Super satisfying.
Can anyone relate? I think the take-away here is:
don’t miss the rubric forest for the criteria trees.
Writing rubrics isn't doing a chore, it's solving a semantic puzzle.
Data Annotation is kind of fun, actually.
I think I might. If given an option I'd much prefer to go project-based, and I believe that might improve my performance.
I'd be able to keep track of my effort/reward, there would be less administrative overhead, and I'd be fine with holding the revision onus (since my goal is to produce rubrics that don't require much revision at all).
It wold also make my time-management process more flexible. I personally really like to go on long walks while brainstorming, then everything flows much better when I sit down to write.
Wonder if anyone shares the sentiment?
PS:
Project-based fees incentivize you to get it right the first time to avoid unpaid revisions. That's better for writers, reviewers and Mercor.
And yes, scope creep might be an issue: but using initial AHT data to set ongoing project fees could deliver a best of both worlds dynamic.
