u/Ibz04

Day 3 of working on a computer-use agent

The idea is pretty simple. Instead of just generating text, I’m trying to get something that can actually use a device. It looks at what’s on the screen, decides what to do next, and takes actions step by step until it finishes a task

This clip is just a small result from today. Nothing crazy yet, but it’s starting to feel a bit more real

One thing that surprised me is most of the problems aren’t really about intelligence. The agent mostly fails when it misunderstands what’s on the screen. Even small changes in the interface can throw it off completely

I also noticed it works better when it keeps checking what just happened instead of assuming everything worked. Keeping the loop tight and simple seems to help more than trying to plan too far ahead

Right now it feels less like building something smart and more like trying to keep it grounded in what’s actually happening moment by moment

Still early, just sharing progress, see my progress live on GitHub: https://github.com/iBz-04/gloamy

A small experiment with a computer use agent called gloamy on gpt-4.1

The setup lets it actually interact with a device , sees the screen, decides what to do, taps or types, keeps going until the task is done. Simple cross-device task, nothing complex. The whole point was just to see if it could follow through consistently.

I’ve spent 2 years building and researching about computer use agents. Here’s a preview of Gloamy, my free & open source agent in this video it autonomously completes homework, saves files, takes notes, and uploads for delivery, all from a single prompt and without using significant tokens

It’s built on a lightweight runtime just below 10mb. My goal: match or beat closed tools like Cowork on computer use metrics.

GitHub: https://github.com/iBz-04/gloamy

website: https://gloamy.co

im wondering if computer use agents are applicable to any industries, I know Ghanaians wouldn’t use them for sure 😂😂

link: https://github.com/iBz-04/gloamy , I’ve been obsessed with computer-use agents for the past two years.

Not in a casual “this is interesting” way, but in the kind of way where an idea keeps following you around. You see a demo, you try things yourself, you hit walls, you rebuild, you question the whole approach, then somehow you still come back the next day because you know there’s something real there.

It’s an open source agent project I’ve been putting real thought and time into, and I’m finally at the point where I want to share it properly instead of just building in my own corner. I want to grow this into something much bigger, and I’d genuinely love to get eyes on it from people who actually care about this space.

What excites me most is not just “AI that does stuff,” but the bigger question of how we make agents feel actually useful, reliable, and grounded in the real world instead of just flashy. That’s the part I’ve been serious about for a long time.

I’m posting this here because I want real feedback. Not praise for the sake of it. I want thoughts, criticism, doubts, ideas, whatever you honestly think. If something feels off, say it. If something is promising, say that too. If you’ve been building in this space, I’d especially love to hear how you see it.

This project means a lot to me, and I’m hoping to take it much further from here.

Would love to hear what you think about gloamy.