In recent years, artificial intelligence (AI) has reshaped the global technology industry at an unprecedented pace. Generative AI, exemplified by large language models (LLMs), is moving from laboratories into industrial practice and increasingly permeating software development, scientific research, medical diagnosis, financial services, autonomous driving and smart manufacturing. From ChatGPT to multimodal models, from AI Agents to world models, today's AI is no longer confined to single algorithms or application scenarios but is gradually forming a complete technical ecosystem spanning models, compute, data and applications.
However, this transformation did not happen overnight. Today's AI technologies are built on decades of foundational research. Algorithms and techniques that now seem like infrastructure—such as ResNet, Adam, Transformer, BERT, U‑Net and Scikit‑learn—originated in milestone academic papers. They not only drove breakthroughs in algorithms but also changed the direction of subsequent research and industrial technology roadmaps.
This site published "Top 10 Highly Cited AI Papers in 2025: Milestones of Deep Learning and Generative Models" in June 2025, introducing ten highly cited AI papers at that time. After more than a year of development, we re‑examine these classic studies and, using the latest citation data available as of November 2026, update their measured influence.
It should be noted that there is no single global standard for counting paper citations. Different sources such as Google Scholar, Semantic Scholar and other academic databases differ significantly in how they merge paper versions, cover publications and identify citations. Therefore, this article does not focus on rigorously combining multiple databases into a single global ranking. Instead, it relies primarily on the latest data from Google Scholar to select ten representative highly cited papers and observe their academic influence and position in AI's technical development.
1. Citation changes for ten representative AI papers
The table below compares the citation counts recorded in the 2025 article with the latest Google Scholar statistics available now. Because the databases and coverage used in the two counts may not be identical, the numbers are intended to illustrate scale and trends and should not be interpreted as the exact additional citations within the same database over this period.
Several notable phenomena emerge from this data.
First, Transformer and Adam remain among the most influential foundational contributions in deep learning. They address key problems in sequence modeling and neural network optimization, respectively, and are still the technical basis of many modern AI systems.
Second, ResNet's influence remains prominent. Its residual connections have become an important design principle in deep neural network architecture, far beyond image recognition.
Third, classic vision models have not lost academic value amid the rise of generative AI. Research represented by AlexNet, VGG and U‑Net still forms a crucial part of the history of computer vision techniques.
Fourth, Long Short‑Term Memory (LSTM), which was omitted from last year's list, is included this time; it was one of the most popular deep learning frameworks before the Transformer era.
2. Ten representative milestone AI papers
1. Deep Residual Learning for Image Recognition (ResNet, 2015/2016)
Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
Paper link: arXiv:1512.03385
ResNet is one of the most important architectural innovations in modern deep learning. Proposed by Kaiming He and colleagues in 2015 and published in conference proceedings in 2016, it addressed the optimization degradation problem that occurs when neural networks grow deeper.
Traditional networks typically learn a direct mapping from input to output. ResNet introduced residual learning so that the network learns the difference between the input and the desired output. Its core expression is:
[H(x)=F(x)+x]
where x is the input and F(x) is the residual mapping the network needs to learn. By using identity skip connections, inputs can be propagated directly to subsequent layers, improving optimization for very deep networks.
The paper proposed architectures such as ResNet‑34, ResNet‑50, ResNet‑101 and ResNet‑152. ResNet‑152 achieved leading results on ImageNet at the time, spurring efforts to build even deeper networks.
Technical impact:
- Residual connections became a fundamental design in deep neural networks.
- They spurred subsequent architectures such as DenseNet and Inception‑ResNet.
- The residual idea has been widely applied in Transformers, vision models and multimodal networks.
ResNet's importance lies not only in improving image recognition accuracy but also in changing how researchers construct deep neural networks.
2. Attention Is All You Need (Transformer, 2017)
Authors: Ashish Vaswani et al.
Paper link: arXiv:1706.03762
If ResNet changed the design of deep vision networks, the Transformer fundamentally changed the trajectory of natural language processing.
In 2017, a team at Google Research published "Attention Is All You Need", proposing a sequence modeling architecture entirely based on attention, eliminating reliance on recurrent neural networks (RNNs) that process sequences step by step.
The core of the Transformer is the self‑attention mechanism. The model computes relationships between tokens in a sequence using Query, Key and Value vectors, allowing each token to dynamically aggregate information from other tokens according to context.
Its core computation is shown here:
This design brought several key advances:
- Parallel computation: Compared with traditional RNNs, Transformers can better leverage GPUs for parallel training.
- Long‑range dependency modeling: Self‑attention can directly model interactions between distant positions in a sequence.
- Architectural scalability: Models can be scaled by increasing layers, hidden dimensions and training data.
The original Transformer achieved BLEU scores of 28.4 and 41.8 on the WMT 2014 English‑German and English‑French tasks respectively, demonstrating the advantages of attention for sequence modeling.
Subsequently, BERT, GPT, T5 and many multimodal models adopted the Transformer or its variants. Today's large language model ecosystem is largely built on the architectural ideas introduced in this paper.
Technical impact: The Transformer is more than a new network architecture; it is the key starting point for modern foundation models. It enabled researchers to explore stronger model capabilities by scaling parameters, data and compute.
3. Adam: A Method for Stochastic Optimization (2014)
Authors: Diederik P. Kingma, Jimmy Ba
Paper link: arXiv:1412.6980
Adam (Adaptive Moment Estimation) is one of the most influential optimization algorithms in deep learning. It combines first‑moment estimation from Momentum with second‑moment estimation from RMSProp, providing adaptive learning rates for different model parameters.
Adam uses exponentially weighted averages to estimate the first and second moments of gradients and applies bias correction to improve estimates early in training.
Main advantages include:
- Automatically adjusting learning rates for different parameters.
- Good adaptation to sparse gradients and non‑stationary objectives.
- Stable and efficient optimization in many deep learning tasks.
Adam and its variants (AdamW, AMSGrad, RAdam, etc.) are widely used to train Transformers, BERT, GPT and various generative models.
It is worth noting that Adam is not superior to SGD in all tasks. For example, carefully tuned SGD can remain competitive in some computer vision tasks. Nonetheless, Adam's generality and ease of use have made it a key tool in modern deep learning workflows.
Technical impact: Adam lowered the engineering barrier for optimizing deep neural networks, allowing researchers to train complex models more efficiently. As models continue to scale, the stability and efficiency of optimization algorithms remain critical factors in training cost.
4. Random Forests (2001)
Author: Leo Breiman
Paper link: Machine Learning journal
Random Forests are one of the most representative ensemble learning algorithms in traditional machine learning.
In 2001, Leo Breiman systematically proposed random forests, which combine multiple decision trees to improve predictive performance and generalization.
Core mechanisms include:
- Bootstrap sampling: Drawing multiple subsets with replacement from the original training data.
- Random feature selection: Each tree considers only a random subset of features when splitting nodes.
- Ensemble decision: Classification by majority vote and regression by averaging predictions.
Random forests are robust, can handle nonlinear relationships and provide feature importance analysis, so they remain widely used in finance risk control, medical diagnosis, bioinformatics and industrial forecasting.
Compared with rapidly advancing deep learning, random forests do not require large GPU resources or massive datasets. For structured‑data tasks, they remain an important baseline.
Technical impact: This paper demonstrated that combining multiple simple models can yield more stable predictive performance than a single model. The ensemble learning paradigm continues to influence machine learning system design.
5. BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding (2018)
Authors: Jacob Devlin, Ming‑Wei Chang, Kenton Lee, Kristina Toutanova
Paper link: arXiv:1810.04805
BERT is an important milestone in the history of natural language processing.
In 2018, the Google AI team proposed BERT (Bidirectional Encoder Representations from Transformers), which learns contextualized text representations using a bidirectional Transformer encoder and changed the prior practice of training separate models for different NLP tasks.
BERT is pretrained on large amounts of unlabeled text and then fine‑tuned for specific tasks, using two main training objectives:
- Masked Language Modeling (MLM): Randomly mask some tokens in the input and let the model predict the masked content from context.
- Next Sentence Prediction (NSP): In the original BERT, determine whether two sentences are consecutive to help the model learn sentence relationships.
BERT achieved state‑of‑the‑art results on benchmarks such as GLUE, SQuAD and SWAG at the time.
Its main contribution was demonstrating that large‑scale pretraining can learn general language representations that transfer to various downstream tasks via fine‑tuning.
Technical impact:
- Spurred pretraining‑based language models to become the mainstream approach in NLP.
- Gave rise to RoBERTa, ALBERT, DistilBERT and others.
- Provided a technical foundation for search, text classification, question answering and semantic understanding systems.
It is important to distinguish that BERT is a bidirectional encoder‑based representation model, whereas the GPT series primarily uses autoregressive generation. Both build on the Transformer but differ in training objectives and application directions.
6. Very Deep Convolutional Networks for Large‑Scale Image Recognition (VGG, 2014)
Authors: Karen Simonyan, Andrew Zisserman
Paper link: arXiv:1409.1556
The VGG networks, proposed by the Visual Geometry Group at Oxford, systematically studied the effect of increased convolutional depth on image recognition performance.
Its most representative architectures are VGG‑16 and VGG‑19. VGG's core idea is to use multiple consecutive 3×3 convolutions to replace larger kernels, increasing depth to improve feature representation.
For example, stacking several 3×3 convolutional layers can achieve a larger receptive field while introducing more nonlinearities.
Key features of VGG include:
- Simple, highly regular network modules.
- Increasing depth by stacking small kernels.
- Ability to extract powerful visual features.
VGG performed strongly in the ImageNet 2014 competition. Although it has large parameter count and computational cost, its clear structure made it widely used in vision research and transfer learning.
Technical impact: VGG demonstrated the importance of network depth and architecture design for visual feature learning and provided an important reference for later convolutional network developments.
7. Long Short‑Term Memory (LSTM, 1997)
Authors: Sepp Hochreiter, Jürgen Schmidhuber
Paper link: DOI: 10.1162/neco.1997.9.8.1735
If traditional recurrent neural networks (RNNs) opened sequence modeling, LSTM solved the key problem of learning long‑term dependencies, laying the foundation for later NLP and sequence learning.
In 1997, Hochreiter and Schmidhuber published "Long Short‑Term Memory", proposing LSTM networks. Their core innovation introduced a cell state and gating mechanisms into recurrent networks, enabling selective retention, update and output of historical information and mitigating gradient vanishing during RNN training.
LSTM includes three gating structures:
- Forget gate: decides which historical information to keep and which to discard.
- Input gate: controls what information from the current input is written into the cell state.
- Output gate: determines which information from the cell state is used for the current output.
This design brought several breakthroughs:
- Long‑term dependency learning: improved RNNs' ability to capture long‑range relationships.
- Dynamic memory management: the model can decide which information to retain, update and output based on input.
- Wide applicability: suitable for NLP, speech recognition, machine translation and time series prediction.
Afterward, LSTM and its variant GRU were the mainstream for sequence modeling before the Transformer. Since the Transformer emerged in 2017, attention mechanisms have become central to large‑scale language models, but LSTM still plays a role in some real‑time speech processing, time series forecasting and resource‑constrained scenarios.
Technical impact: LSTM's key contribution was not only improving RNN training but also showing that neural networks can manage long‑term context through dedicated information control mechanisms. This idea inspired later research on organizing, filtering and using information within neural networks.
8. ImageNet Classification with Deep Convolutional Neural Networks (AlexNet, 2012)
Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
Paper link: ACM Digital Library
AlexNet is a pivotal turning point in the history of deep learning.
In 2012, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton used deep convolutional neural networks to achieve a breakthrough in the ImageNet large‑scale classification competition. The model had about 60 million parameters, with 5 convolutional and 3 fully connected layers, achieving roughly a 15.3% Top‑5 error rate that year—significantly better than traditional methods at the time.
AlexNet's main technical innovations included:
- ReLU activation: improving training efficiency for deep networks.
- GPU parallel training: using two GPUs to train large‑scale image models.
- Dropout: reducing overfitting in fully connected layers.
- Data augmentation: using image transformations to improve generalization.
Most importantly, AlexNet demonstrated that deep convolutional networks, supported by large datasets and GPU computing, could markedly outperform traditional image recognition methods.
This result directly promoted deep learning's adoption in computer vision and accelerated GPU use in AI research.
Technical impact: AlexNet marked the decisive breakthrough of deep learning in modern computer vision and served as a foundation for later networks like ResNet and VGG.
9. Scikit‑learn: Machine Learning in Python (2011)
Authors: F. Pedregosa et al.
Paper link: Journal of Machine Learning Research
Scikit‑learn is not a new algorithm but an influential Python machine learning library.
In 2011, Fabian Pedregosa and colleagues published a paper describing Scikit‑learn's design principles, algorithm implementations and software architecture.
Scikit‑learn provides a unified API covering classification, regression, clustering, dimensionality reduction, feature processing, model selection and evaluation.
One of its most important engineering designs is the standardized training and prediction interface:
fit(): train the model.predict(): generate predictions.transform(): perform feature transformations.
This standardized interface greatly reduced the complexity of machine learning application development, allowing researchers to switch algorithms, compose pipelines and run experiments more conveniently.
Technical impact: Scikit‑learn helped make Python a major language for machine learning and data science and established engineering practices familiar to many developers.
10. U‑Net: Convolutional Networks for Biomedical Image Segmentation (2015)
Authors: Olaf Ronneberger, Philipp Fischer, Thomas Brox
Paper link: arXiv:1505.04597
U‑Net was originally designed for biomedical image segmentation and has become one of the most influential architectures in image segmentation.
Its core design is a symmetric encoder–decoder structure:
- The encoder progressively reduces spatial resolution to extract higher‑level semantic features.
- The decoder gradually restores spatial resolution to produce fine‑grained pixel‑level predictions.
- Skip connections pass high‑resolution features from the encoder to the decoder, compensating for spatial information lost during downsampling.
U‑Net is especially suitable for medical image segmentation, where accurate identification of organs, tissues and lesions is required with limited training samples.
Since then, U‑Net and its variants have been widely applied in medical imaging, remote sensing, agriculture, industrial inspection and autonomous driving.
Technical impact: U‑Net established the classic encoder–decoder segmentation paradigm, and its skip‑connection design influenced many subsequent vision models.
11. Deep Learning (2015)
Authors: Yann LeCun, Yoshua Bengio, Geoffrey Hinton
Paper link: Nature
In 2015, Yann LeCun, Yoshua Bengio and Geoffrey Hinton published the review "Deep Learning" in Nature, systematically summarizing the basic principles, historical development and primary applications of deep learning.
The article reviewed progress in computer vision, speech recognition and natural language processing and analyzed how large‑scale data, GPU computing and optimization algorithms propelled deep learning's advance.
It emphasized the core idea of deep learning: learning hierarchical representations from raw input through multiple neural network layers to automatically extract features at different abstraction levels.
The paper summarized techniques such as CNNs, RNNs and deep generative models and explained why deep networks can achieve strong performance on complex tasks.
Technical impact:
- Systematically summarized the theoretical foundations and technical development of deep learning.
- Helped make deep learning the central direction of AI research.
- Provided important context for the later development of large‑scale neural networks, Transformers and foundation models.
Although it fell out of the top‑ten citation list, as a foundational review the paper's value is not only in summarizing existing results but also in helping academia and industry understand the common principles behind the deep learning revolution.
3. Three stages of AI development reflected by these ten papers
Reviewing these ten papers shows that modern AI did not advance from a single algorithm but from the mutual promotion of multiple foundational directions.
Stage one: Establishing core deep learning capabilities
AlexNet, VGG and ResNet represent the evolution of deep convolutional neural networks.
AlexNet demonstrated the power of combining GPUs, large datasets and deep networks; VGG explored the relationship between depth and feature representation; ResNet addressed critical optimization issues in very deep networks.
The core task of this stage was enabling networks to learn complex features automatically from massive data.
Stage two: Formation of general models and engineering systems
Adam, Random Forests and Scikit‑learn represent model optimization, ensemble learning and machine learning engineering, respectively.
These works together helped machine learning move from isolated algorithm experiments to reusable, scalable engineering systems.
Meanwhile, BERT and U‑Net further demonstrated the value of pretrained models and task‑specific architectures, enabling stronger performance in language understanding and fine‑grained vision tasks.
Stage three: The Transformer and the foundation model era
The 2017 proposal of the Transformer marks a key turning point in modern AI.
Compared with task‑specific models, the Transformer offers stronger parallelism and architectural scalability. By enlarging model parameters, training data and compute, researchers progressively built foundation models with general representations and generative abilities.
Since then, large language models, multimodal models and AI Agents have advanced rapidly, and AI has evolved from single‑task models to systems capable of addressing multiple tasks.
4. What do citation counts for classic papers mean?
Citation counts are an important indicator of academic influence, but they are not equivalent to technical value and should not be read as direct measures of real‑world adoption.
A high citation count may indicate a new foundational method, that a paper became a standard reference in a research area, or that it provided widely used tools and datasets.
For example, the Transformer's influence comes not only from citation numbers but from its deep impact on the entire foundation model roadmap; Scikit‑learn's value is more evident in the engineering ecosystem and developer practices it enabled.
Therefore, evaluating a paper's historical status requires at least three dimensions:
- Academic influence: the breadth and depth of subsequent research citing the paper.
- Technical influence: whether the proposed method changed later algorithm and architecture design.
- Industrial influence: whether the technology entered mainstream software, hardware and commercial products.
From this perspective, citation counts are a useful window into AI's evolution but not the only metric of technical contribution.
Conclusion: Where will the next AI breakthrough come from?
These ten papers span more than two decades and cover deep learning, optimization algorithms, computer vision, natural language processing and machine learning engineering. Together they make up the technical foundation of today's AI.
But AI has not stopped at the Transformer era.
Current research is moving toward more complex directions, including reasoning, reinforcement learning, multimodal fusion, world models, embodied intelligence, AI Agents and continual learning. Future breakthroughs may come not just from scaling parameter counts but from better data organization, more efficient training methods, more appropriate architectures and tighter integration between models and the physical world.
From AlexNet to ResNet, from BERT to Transformer and on to today's large language models, every major advance built on prior work.
What truly changes the direction of AI is often not how large a single model is, but whether researchers propose a method that the whole technical ecosystem can continue to adopt. That is why these classic papers remain worth revisiting years later.
▶️ Enjoyed this article? Creating great content takes effort!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares mean the world to us! 🙏
For more AI news, visit:
https://www.airosetta.com