AI - Super Intelligence.

Google's top artificial intelligence scientists, also known as “the godfather of AI,” said after he quit his job in April so that he can warn about the dangers of this technology.

He’s not the only one worried. A 2023 survey of AI experts found that 36 percent fear that AI development may result in a “nuclear-level catastrophe.” Almost 28,000 people have signed on to an open letter written by the Future of Life Institute, including Steve Wozniak, Elon Musk, the CEOs of several AI companies and many other prominent technologists, asking for a six-month pause or a moratorium on new advanced AI development.

Why are we all so concerned? In short: AI development is going way too fast.

The key issue is the profoundly rapid improvement in conversing among the new crop of advanced "chatbots," or what are technically called “large language models” (LLMs). With this coming “AI explosion,” we will probably have just one chance to get this right.

If we get it wrong, we may not live to tell the tale. This is not hyperbole.

This rapid acceleration promises to soon result in “artificial general intelligence” (AGI), and when that happens, AI will be able to improve itself with no human intervention

This pace of change is why Hinton told the New York Times: "Look at how it was five years ago and how it is now. Take the difference and propagate it forwards. That’s scary.” In a mid-May Senate hearing on the potential of AI, Sam Altman, the head of OpenAI called regulation “crucial.”

Once AI can improve itself, which may be not more than a few years away, and could in fact already be here now, we have no way of knowing what the AI will do or how we can control it. This is because superintelligent AI (which by definition can surpass humans in a broad range of activities) will—and this is what I worry about the most—be able to run circles around programmers and any other human by manipulating humans to do its will; it will also have the capacity to act in the virtual world through its electronic connections, and to act in the physical world through robot bodies.

Here’s another way of looking at it: a superintelligent AI will be able to do in about one second what it would take a team of 100 human software engineers a year or more to complete. Or pick any task, like designing a new advanced airplane or weapon system, and superintelligent AI could do this in about a second.

Once AI systems are built into robots, they will be able to act in the real world, rather than only the virtual (electronic) world, with the same degree of superintelligence, and will of course be able to replicate and improve themselves at a superhuman pace.

Any defenses or protections we attempt to build into these AI “gods,” on their way toward godhood, will be anticipated and neutralized with ease by the AI once it reaches superintelligence status. This is what it means to be superintelligent.

Asimov predicted this long ago

AI complexes and large AI models are increasingly being developed with significant secrecy. Leading AI companies like OpenAI, Google, and others often withhold detailed information about their AI training data, model architecture, and development processes. This secrecy is driven by concerns over misuse of the technology, competitive advantage, and proprietary value. For example, OpenAI’s GPT-4 and similar advanced models have only limited transparency about their inner workings, leading to criticism from researchers advocating for more openness to enable scientific progress, accountability, and safety.

This culture of secrecy also extends beyond individual AI models to organizational practices, with some reports describing nondisclosure and non-disparagement agreements for employees, and lobbying for restricted public access to AI operational details. Meanwhile, some experts and institutions argue that secrecy may hinder innovation and concentrate AI power in the hands of a few corporations, suggesting that more transparency would benefit the AI field and society. Additionally, certain government and law enforcement AI initiatives are reported to be developed with minimal public scrutiny, further entrenching secrecy around AI deployment contexts.

While AI secrecy is prominent in commercial and sensitive security contexts, there is also ongoing debate whether open AI models pose greater risks than closed ones, with many researchers seeing openness as preferable or necessary, balanced against misuse concerns.

In summary, yes, powerful AI complexes and systems are often built and operated with a high degree of secrecy, driven by competitive, security, and proprietary considerations, which is currently a contested issue in the AI community and policy circles.