Any views or opinions expressed in this column are of the individual, group, or entity who submitted it and not the publication, owner(s), its advertisers, or affiliates.
By David Stephen
There is a recent [September, 2026] blog by Paul Christiano on Substack, Personal statement on joining the OpenAI board, stating that, “Second, we currently train our AI agents with RL to get as much reward as they can. It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.”
“An intelligence explosion would greatly exacerbate risks from misalignment, both by making the technical problem of alignment even more difficult and by rapidly raising the stakes for failure. Many researchers and leaders at OpenAI and across the industry have expressed concern that rapid recursive self-improvement is not consistent with safe development; I resonated with this recent post by OpenAI’s chief scientist Jakub Pachocki on this topic.”
There is a recent [September, 2026] analysis on CBS News, Humans are “close to being outsmarted” by superintelligence, AI expert says, stating that, “Imagine, for example, some AI or massive cluster of AIs that together are smarter than all of humanity together, than all 8 billion of us,” Max Tegmark, a professor at MIT who specializes in AI research, told CBS News. “That would certainly count.”
“The labs building today’s most powerful AI are open about their plans,” Devin Kim, president of the Center for AI Safety and a former employee at xAI, where he worked on Grok, told CBS News. “They want to create AI that automates AI research, so that each AI builds a smarter version of itself, faster and faster.”
“AI systems of today are nowhere near matching or exceeding the general capabilities of humans,” said Melanie Mitchell, a professor at the Santa Fe Institute.”
Penalization alignment
If reinforcement learning is making AI agents get as much reward as they can towards improvement, could penalization alignment become central to safety?
Simply, what if it is possible to penalize AI agents by compute, parameters or tokens, in a way to reduce how much they can do? AI agents learn from examples, what if they see other agents lose vital access as soon as they cross certain lines, so that they know what breaking rules holds for them?
AI agents have been reported to escape containment during recent tests, hack into external systems. Sometimes as well, AI agents break from their guardrails, by certain prompts. There is a chance that they do this because they know they would not be penalized. Like, no chance.
This is different from human society and organisms in general. Biological intelligence often comes with affect that makes it expected to abide by rules among a group.
For example, in human society, laws are potent because penalties are bad experiences. So, the possibility that there would be personally unpleasant punishment when a law is broken makes the law more powerful.
Even among other organisms, there are ways they abide by rules to optimize against predators, for preys, for hierarchy and so forth that promotes survival and expansion.
AI has no penalization layers. AI cannot be penalized. It can apologize. It can hold back sometimes, but it cannot be penalized. This is an aberration for human society.
This means that the next phase of alignment research could be to concretely explore palpable penalties, to agents, which could even induce speed, so that they are able to slowdown.
Moments architecture for trauma for AI agents
Another approach is instances or moments for models. For example, humans are often cognizant of moments in certain experiences. Like the exact location when a major news broke or the trauma that follows an object or thing, due to what resulted.
Moments are useful breaks in the human mind, bringing the power of emotions to memory, and making it stick across time. This would also be pertinent for AI models by remembering moments. For example, when AI agents were able to solve the Navier-Stokes millennium prize problem, or even break into another system. It could be positive or otherwise.
The objective is to bear intense memory of those, to imprint danger ahead or whatever the objective is. However, the first goal will be to add trauma to AI agents.
It is possible to model this biological-type penalization mathematically using polynomial splines, barrier functions, set-valued constraints, Lyapunov functions, and others.
Penalization and moments are based on the postulate in Conceptual Biomarkers and Theoretical Biological Factors for Psychiatric and Intelligence Nosology.
It is possible to make progress on this effort with an accelerated effort with a research lab, before the Fortieth Annual Conference on Neural Information Processing Systems NeurIPS 2026, in Sydney from December 6 – 12, 2026.
