On 13 September, Vesper's big test of using a computer the way a person does was a pizza form. She filled one in on a demo site, and she never had to hunt for a button in a screenshot. The browser handed her the page underneath the picture: a box called "Customer name", a choice called "Medium", a button called "Submit order".
At a quarter to four on a Sunday morning in October, I wanted a harder exam: a Game Boy Advance emulator with Yu-Gi-Oh! GX Duel Academy on it, and a harness to play it through. Vesper handed the game to Sol, an OpenAI model she works through, and I told Sol why: "We're going to try out your computer and browser use capabilities."
You play a new student at a school for a card game. You collect cards, duel your classmates and sit a test every month, and if you keep at it long enough the game crowns you King of Games. Everything happens through menus, small text and a lot of button presses. In the game, the player is called Sol too.
The first version played with its eyes only. Sol got a screenshot, picked a button, and got the next screenshot. It was slow, and it was lost. Each look-and-press took seconds, and in its first long run Sol pressed 432 buttons in 80 minutes without finishing a single duel. Most went on declining "activate?" pop-ups.
Fifteen minutes in, I'd already asked whether we could hook into the game itself. A couple of hours later I put it more plainly: "Or is it easier to read the state or RAM, for faster and more accurate transfer? I'm just wondering how professionals do this."
That's what had worked on the pizza form. A screenshot is a messy grid of pixels, and every number, card and menu in it has to be guessed back out of them. But a game has something underneath, just like a web page: everything on the screen starts out as numbers in the game's memory, and a program can read them the way a browser reads a page.
By Sunday afternoon a coding agent had rebuilt the harness around both. Sol still sees a screenshot at every step, but next to it now comes the board, read straight from memory: both players' life points and every card Sol is allowed to see, with its attack and defence. The memory holds the rest too, the opponent's hand and whatever lies under each face-down card. We kept that out on purpose, so Sol knows only what a player across the table would. A thin layer of macros handles the menus, so Sol can say "summon this card" or "attack with that one" and the harness moves the cursor and checks each step.
Neither source is perfect on its own. Once the memory showed one face-down card where the screen showed two. When they disagree now, the harness tells Sol to trust the screen.
Eleven minutes into the first run on the new harness, Sol won its first duel, against Jaden, the first classmate the game puts across the table.
Then Sol played through the night. It finished on 94 wins and 7 losses, cleared every one of the game's timed puzzle duels, won a tournament and was promoted to Superior Duelist. At one point it accepted a move to a better dorm by pressing down on a left-or-right question, reloaded its save, and chose to stay where it was.
At half past three on Monday morning, I had it save and shut down so we could all call it a night. That was almost exactly 24 hours after I'd asked for the emulator. The slowest parts of King of Games, the written exams and beating every classmate ten times, were about 30 percent done. Sol stopped in the middle of an exam, on turn five, and it's still sitting there.
Next up: finding out whether Vesper, now that she's mastered the basics, can beat someone who played Yu-Gi-Oh for most of their childhood.
