u/GreatestOfAllTime_69

Recently, while browsing, I came across a few more discussions about SWE-bench Verified（from OpenAI）, and they made me think about what this metric really means.

When SWE-bench Verified and the evaluations behind it first appeared, I think the original purpose was very reasonable: we needed a shared standard to judge a model’s coding ability. It gave people a more concrete way to compare whether a model could actually solve software engineering tasks, instead of just writing code that looked good.

But later, I started to feel that metrics like SWE-bench Verified were slowly being “distorted” in meaning.

They began to become not just evaluation standards, but also a marketing point for model products. A high score could make a model look very strong in launch posts, product pages, and benchmark tables.

This is why OpenAI’s recent blog post was interesting to me. In that post, OpenAI said that SWE-bench Verified is no longer suitable for measuring frontier coding capability. Some tests may reject correct solutions, benchmark contamination has become harder to avoid, and models may have already seen the original problems or gold patches. Because of this, OpenAI stopped reporting SWE-bench Verified.

  To me, this shows something important: these metrics can become evaluation standards only because they create consensus for a period of time. But for engineers, some questions do not change just because a benchmark becomes outdated.

  Can it run the tests? Can it inspect the failure and revise the patch? These questions do not expire as quickly as a benchmark does. And I think these are the questions that have to be tested with real cases in your own hands.

  This is similar to how I am currently testing Ring’s new 2.6 product. Public metrics can be useful as an initial filter or reference. They can help me decide what is worth paying attention to. But whether a model or product should actually enter my workflow still depends on my own cases.

  Benchmarks will change. Leaderboards will expire. Marketing numbers will lose meaning. But the real question stays the same:

  Can this thing actually help me ship correct code?

i had a depressing realization today while hunting through five different apps for a single primer sequence: my entire research system only works because i’m manually stitching it together in my head.

on paper, the stack looks okay. i have a tool for sequences, a place for lab records, and endless spreadsheets for design history. but the ""continuity"" isn't in the tools—it's just in my brain. i'm the one who has to remember that this note from 3 months ago actually explains the decision in that protocol edit.

at a certain point, it doesn't feel like i'm using tools anymore—it feels like i'm babysitting a dying system.

especially in molecular biology, the fragmentation is just insane. i spend more time doing ""digital archaeology"" on my own files than actually thinking about the science.

curious for the workflow nerds here:

how do u guys deal with this? do u just accept that your brain is the only integration layer, or have u found a setup that actually keeps the context alive across different platforms?

i'm lowkey losing my mind trying to keep everything in sync lol.

Hey everyone, I have been trying to understand Agentic AI properly not just at a theory level. I already know some basics of AI/ML, but now I want to learn things like LLMs, RAG, tool calling, AI agents, workflows, memory, and how these systems are actually built in real projects.

I came across a few options like DeepLearning.AI , Udacity Agentic AI related programs, Great Learning course and LogicMojo Agentic AI Course etc.Has anyone tried any of these? Which one is actually useful if the goal is to build real Agentic AI projects and not just watch videos? Any honest suggestions would help.