Artificial intelligence on the verge: startups are looking for new ways to scale

The upcoming AI model from OpenAI, according to the sources of The Information, will show a less tangible performance increase compared to previous versions.

According to published data, the Orion model reached GPT-4 after only 20% of training. This suggests that GPT-5 will not be able to outperform GPT-4 as significantly as GPT-3 did when it transitioned to GPT-4.

“Orion does not demonstrate significant advantages in certain tasks. It copes well with language tasks, but it does not outperform previous models in programming,” the startup’s employees note.

Usually, the greatest improvements in neural networks are observed in the initial stages of training. After that, the progress gradually slows down. According to the sources of The Information, the remaining 80% of the training time is unlikely to bring a significant leap in performance.

AI has reached its limits

The recent OpenAI results, while not optimistic, highlight a more global problem: the lack of quality data for training.

In a study published in June, experts said that AI companies will exhaust all available textual materials between 2026 and 2032. This could be a turning point for traditional approaches to the development of artificial intelligence.

“Our results show that current trends in LLM development cannot be sustained only through data scaling,” the authors of the study note.

The paper emphasizes the need to look for alternative approaches to improve neural networks, including the creation of synthetic data or the use of classified information.

The InformationThe Information drew attention to the fact that the current strategy of teaching LLMs on open text data from websites, books, and other sources has reached a point of diminishing returns, because “developers have squeezed everything they can out of this format”.

There is a way out

OpenAI and other companies are significantly changing approaches to the creation of artificial intelligence.

“Amid the slowdown in GPT improvement, it seems that the industry is now focusing more on improving models after their initial training rather than scaling them during the learning process. This approach may lead to the emergence of new scaling rules,” reports The Information.

To achieve continuous development of its models, OpenAI divides the process into two main areas:

1. Series O – aimed at developing logical thinking abilities. These models require significantly more computing resources and are designed to perform complex tasks. The cost of their operation is six times higher than that of current models. However, their powerful analytical capabilities justify these costs for tasks that require in-depth analysis.
   
2. GPT Series – focused on solving a wide range of communication problems. These models use a large-scale knowledge base for effective interaction.

During the AMA session, Kevin Weil, Chief Product Officer at OpenAI, noted that in the future, it is planned to integrate these two approaches, which will combine their advantages.

The danger of synthetic data

Creating artificial data to solve the problem of its shortage can seriously affect the reliability of information. This problem was highlighted by researchers from several UK universities.

In their opinion, this approach can completely detach artificial intelligence from reality and lead to the so-called “model collapse”. The main problem is the use of unreliable data to create training sets for the next generations of AI.

To avoid this, OpenAI develops filtering mechanisms that help preserve the quality of information. These mechanisms integrate various verification methods to separate quality content from potentially problematic content.

Another effective solution is post-training optimization. Researchers are working on methods to improve AI performance after initial tuning, not just by expanding the dataset.

Earlier, the media reported that OpenAI plans to launch a new advanced model codenamed Orion by December. However, the company’s CEO Sam Altman later denied this information.

Approaches of other companies

Some scientists, researchers, and investors have shared with Reutersthe idea that the methods behind the new o1 AI model could “change the course of the artificial intelligence arms race”.

In September, OpenAI introduced o1, a large language model that was trained with reinforcement learning to perform complex logic tasks. The company claims that this neural network “knows how to think”, forming a detailed internal analysis process when solving problems.

Co-founder of AI startups Safe Superintelligence (SSI) and OpenAI Ilya Sutskever noted that the results of learning on large amounts of unlabeled data “have reached their peak”.

“The 2010s were the time of scaling, and now we are back in the era of miracles and discoveries. Everyone is looking for something new,” he said.

Sutskever refrained from disclosing details of his new company SSI, only hinting at an alternative approach to scaling up pre-learning.

Reuters sources noted that researchers at large AI labs have been experiencing delays and unsatisfactory results when trying to create a language model that outperforms OpenAI’s GPT-4, released almost two years ago.

They are experimenting with improving the performance of neural networks during the so-called “inference phase”. For example, instead of providing one answer, AI first generates several options and chooses the best one.

Recall that in October, informationthat OpenAI is working on the creation of its own AI chip appeared. We will tell more about this in the following articles, so don’t miss it.