AI Research Library
A structured route from mathematical foundations to current work on agents, alignment and production systems. Books, standards and reports are clearly distinguished from research papers.
Source quality. Entries marked “check before citing” still need bibliographic repair; search results are not treated as verified citations.
Foundations 1936–89Statistical learning 1990–2011Deep learning 2012–16Transformers to agents 2017–now
A Mathematical Theory of CommunicationShannonInformation theory: entropy, bits, channel capacity1948
Computing Machinery and IntelligenceTuringThe Turing Test: can machines think?1950
A Logical Calculus of Ideas Immanent in Nervous ActivityMcCulloch & PittsFirst mathematical model of a neuron1943
An Inductive Inference MachineSolomonoffFoundations of algorithmic probability1957
On Computable NumbersTuringTuring machine: theoretical basis of computation1936
Maximum Likelihood from Incomplete Data via the EM AlgorithmDempster, Laird, RubinEM algorithm for statistical estimation1977
Regression Shrinkage and Selection via the LassoTibshiraniL1 regularization for sparse models1996
Bayesian Reasoning and Machine LearningBarberComprehensive Bayesian ML framework2012
Elements of Information TheoryCover & ThomasKL divergence, mutual information1991
Monte Carlo Sampling Methods Using Markov ChainsHastingsMCMC methods foundational for probabilistic ML1970
Stochastic Relaxation, Gibbs Distributions and Bayesian Restoration of ImagesGeman & GemanGibbs sampling1984
An Introduction to the BootstrapEfron & TibshiraniBootstrap methods for statistical inference1993
The Kernel TrickAizerman, Braverman, RozonoerImplicit high-dimensional mappings1964
Singular Value Decomposition and Least Squares SolutionsGolub & ReinschSVD foundations used everywhere in ML1970
Random MatricesMehtaRandom matrix theory: used in deep learning theory1967
The Perceptron: A Probabilistic Model for Information Storage and OrganizationRosenblattFirst perceptron model1958
PerceptronsMinsky & PapertLimitations of single-layer perceptrons (XOR problem)1969
Nearest Neighbor Pattern ClassificationCover & Hartk-NN algorithm1967
A Theory of the LearnableValiantPAC learning theory1984
Induction of Decision TreesQuinlanID3 algorithm: decision trees1986
A Training Algorithm for Optimal Margin ClassifiersBoser, Guyon, VapnikSupport Vector Machines1992
Support-Vector NetworksCortes & VapnikSoft-margin SVMs1995
Statistical Learning TheoryVapnikVC dimension, structural risk minimization1998
Bagging PredictorsBreimanBootstrap aggregating for ensemble methods1996
Random ForestsBreimanRandom forest algorithm2001
A Decision-Theoretic Generalization of On-Line Learning (AdaBoost)Freund & SchapireAdaBoost: adaptive boosting1997
XGBoost: A Scalable Tree Boosting SystemChen & GuestrinGradient boosted trees at scale2016
Greedy Function Approximation: A Gradient Boosting MachineFriedmanGradient boosting framework2001
An Introduction to Variable and Feature SelectionGuyon & ElisseeffFeature selection methods2003
Latent Dirichlet AllocationBlei, Ng, JordanTopic modeling2003
A Tutorial on Principal Component AnalysisShlensPCA explained2014
Reducing the Dimensionality of Data with Neural NetworksHinton & SalakhutdinovAutoencoders for dimensionality reduction2006
K-Means++: The Advantages of Careful SeedingArthur & VassilvitskiiImproved k-means initialization2007
Gaussian Processes for Machine LearningRasmussen & WilliamsGaussian process foundations2006
No Free Lunch Theorems for OptimizationWolpert & MacreadyNo universal best algorithm1997
Learning Representations by Back-Propagating ErrorsRumelhart, Hinton, WilliamsBackpropagation algorithm1986
Multilayer Feedforward Networks are Universal ApproximatorsHornik, Stinchcombe, WhiteUniversal approximation theorem1989
Learning Internal Representations by Error PropagationRumelhart, Hinton, WilliamsDetailed backprop with hidden layers1985
Connectionist Learning ProceduresHintonSurvey of early neural network training1989
A Fast Learning Algorithm for Deep Belief NetsHinton, Osindero, TehDeep belief networks: reignited deep learning2006
Greedy Layer-Wise Training of Deep NetworksBengio et al.Layer-wise pretraining2007
Understanding the Difficulty of Training Deep Feedforward Neural NetworksGlorot & BengioXavier initialization2010
Rectified Linear Units Improve Restricted Boltzmann MachinesNair & HintonReLU activation function2010
Deep Sparse Rectifier Neural NetworksGlorot, Bordes, BengioSparsity benefits of ReLU2011
Maxout NetworksGoodfellow et al.Maxout activation2013
Improving Neural Networks by Preventing Co-Adaptation of Feature DetectorsHinton et al.Dropout: first paper2012
Dropout: A Simple Way to Prevent Neural Networks from OverfittingSrivastava et al.Dropout: comprehensive treatment2014
Batch Normalization: Accelerating Deep Network TrainingIoffe & SzegedyBatch normalization2015
Layer NormalizationBa, Kiros, HintonLayer normalization2016
Group NormalizationWu & HeGroup normalization2018
Deep LearningLeCun, Bengio, HintonLandmark review of deep learning in Nature2015
Representation Learning: A ReviewBengio, Courville, VincentSurvey on learning representations2013
On the Number of Linear Regions of Deep Neural NetworksMontufar et al.Expressivity of deep networks2014
Residual Learning (Identity Mappings in Deep Residual Networks)He et al.Pre-activation ResNets2016
The Lottery Ticket HypothesisFrankle & CarlinSparse subnetworks in dense networks2019
A Stochastic Approximation MethodRobbins & MonroStochastic gradient descent foundations1951
On the Importance of Initialization and Momentum in Deep LearningSutskever et al.Momentum in SGD2013
ADADELTA: An Adaptive Learning Rate MethodZeilerAdaptive learning rates2012
Adam: A Method for Stochastic OptimizationKingma & BaAdam optimizer2015
Decoupled Weight Decay Regularization (AdamW)Loshchilov & HutterAdamW: fixes weight decay in Adam2019
SGDR: Stochastic Gradient Descent with Warm RestartsLoshchilov & HutterCosine annealing learning rate schedule2017
Cyclical Learning Rates for Training Neural NetworksSmithCyclical learning rates2017
Super-ConvergenceSmith & TopinVery fast training with large learning rates2018
Large Batch Training of Convolutional Networks (LARS)You et al.Layer-wise adaptive rate scaling2017
Large Batch Optimization (LAMB)You et al.Layerwise Adaptive Moments for BERT training2020
Sharpness-Aware Minimization (SAM)Foret et al.Seeking flat minima for generalization2021
An Overview of Gradient Descent Optimization AlgorithmsRuderComprehensive optimization survey2016
Fixing Weight Decay Regularization in AdamLoshchilov & HutterWeight decay vs L2 in Adam2018
Mixed Precision TrainingMicikevicius et al.FP16 training for speed and memory2018
Automatic Mixed PrecisionNVIDIAPractical mixed precision2019
Gradient CheckpointingChen et al.Memory-efficient backprop2016
Data Parallelism (PyTorch Distributed)Li et al.Distributed data parallel training2020
ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsRajbhandari et al.ZeRO optimizer: memory efficient2020
DeepSpeedRasley et al.Deep learning optimization library2020
Megatron-LM: Training Multi-Billion Parameter Language ModelsShoeybi et al.Model parallelism for large LMs2020
Neocognitron: A Self-Organizing Neural Network ModelFukushimaPrecursor to CNNs1980
Gradient-Based Learning Applied to Document Recognition (LeNet)LeCun et al.LeNet: CNNs for digit recognition1998
ImageNet Classification with Deep CNNs (AlexNet)Krizhevsky, Sutskever, HintonAlexNet: deep learning revolution2012
Visualizing and Understanding Convolutional Networks (ZFNet)Zeiler & FergusCNN visualization & understanding2014
Very Deep Convolutional Networks (VGGNet)Simonyan & ZissermanVGG: depth matters2015
Going Deeper with Convolutions (GoogLeNet/Inception)Szegedy et al.Inception modules2015
Deep Residual Learning for Image Recognition (ResNet)He et al.Skip connections: 152 layers2016
Densely Connected Convolutional Networks (DenseNet)Huang et al.Dense connections between layers2017
Aggregated Residual Transformations (ResNeXt)Xie et al.Grouped convolutions at scale2017
Squeeze-and-Excitation Networks (SENet)Hu, Shen, SunChannel attention2018
EfficientNet: Rethinking Model ScalingTan & LeCompound scaling for CNNs2019
Network In NetworkLin, Chen, Yan1x1 convolutions2014
Spatial Transformer NetworksJaderberg et al.Learnable spatial transformations2015
Feature Pyramid Networks for Object DetectionLin et al.Multi-scale feature pyramids2017
Faster R-CNNRen et al.Region proposal networks2015
You Only Look Once (YOLO)Redmon et al.Real-time object detection2016
SSD: Single Shot MultiBox DetectorLiu et al.Single-shot detection2016
Focal Loss for Dense Object Detection (RetinaNet)Lin et al.Focal loss: class imbalance2017
Mask R-CNNHe et al.Instance segmentation2017
U-Net: CNNs for Biomedical Image SegmentationRonneberger et al.U-Net architecture2015
Fully Convolutional Networks for Semantic SegmentationLong, Shelhamer, DarrellFCN for segmentation2015
DeepLab: Semantic Image Segmentation with Deep CNNs and CRFsChen et al.Atrous convolution, CRFs2015
An Image is Worth 16x16 Words (ViT)Dosovitskiy et al.Vision Transformer2021
Swin TransformerLiu et al.Shifted window transformer for vision2021
ConvNeXt: A ConvNet for the 2020sLiu et al.Modernized CNN rivaling ViTs2022
Finding Structure in TimeElmanElman RNN: simple recurrent network1990
Long Short-Term Memory (LSTM)Hochreiter & SchmidhuberLSTM: solving vanishing gradients1997
Learning to Forget: Continual Prediction with LSTMGers, Schmidhuber, CumminsForget gate in LSTM2000
Learning Phrase Representations using RNN Encoder-Decoder (GRU)Cho et al.GRU: simplified gated RNN2014
Sequence to Sequence Learning with Neural NetworksSutskever, Vinyals, LeSeq2seq: encoder-decoder for translation2014
Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho, BengioAttention mechanism for NMT2015
Effective Approaches to Attention-based Neural Machine TranslationLuong, Pham, ManningGlobal vs local attention2015
Connectionist Temporal Classification (CTC)Graves et al.CTC loss for sequence labeling2006
Speech Recognition with Deep Recurrent Neural NetworksGraves, Mohamed, HintonDeep bidirectional LSTM for ASR2013
Pointer NetworksVinyals, Fortunato, JaitlyAttention as pointer to input2015
Generating Sequences with Recurrent Neural NetworksGravesHandwriting/text generation with RNNs2013
Bidirectional Recurrent Neural NetworksSchuster & PaliwalBidirectional RNNs1997
Show and Tell: A Neural Image Caption GeneratorVinyals et al.CNN-LSTM image captioning2015
Neural Turing MachinesGraves, Wayne, DanihelkaExternal memory with neural nets2014
Memory NetworksWeston, Chopra, BordesEnd-to-end memory networks2015
Temporal Convolutional Networks (TCN)Bai, Kolter, KoltunCausal convolutions vs RNNs2018
Quasi-Recurrent Neural NetworksBradbury et al.Parallelizable recurrence2017
Independently Recurrent Neural Network (IndRNN)Li et al.Longer memory, gradient stability2018
Relational Recurrent Neural NetworksSantoro et al.Relational reasoning in RNNs2018
WaveNet: A Generative Model for Raw Audiovan den Oord et al.Autoregressive audio generation2016
Auto-Encoding Variational Bayes (VAE)Kingma & WellingVariational Autoencoders2014
Generative Adversarial Networks (GAN)Goodfellow et al.GANs: adversarial training2014
Conditional Generative Adversarial NetsMirza & OsinderoClass-conditional GANs2014
Unsupervised Representation Learning with DCGANsRadford, Metz, ChintalaDeep convolutional GANs2016
Improved Techniques for Training GANsSalimans et al.GAN training stabilization2016
Wasserstein GANArjovsky, Chintala, BottouWGAN: Earth mover's distance2017
Progressive Growing of GANsKarras, Aila, Laine, LehtinenHigh-res image generation2018
A Style-Based Generator Architecture (StyleGAN)Karras, Laine, AilaStyleGAN2019
Analyzing and Improving StyleGAN (StyleGAN2)Karras et al.StyleGAN improvements2020
Image-to-Image Translation with Conditional GANs (Pix2Pix)Isola et al.Paired image translation2017
Unpaired Image-to-Image Translation (CycleGAN)Zhu et al.Unpaired domain transfer2017
Semantic Image Synthesis with Spatially-Adaptive Normalization (SPADE)Park et al.Semantic layout to image2019
Neural Discrete Representation Learning (VQ-VAE)van den Oord, Vinyals, KavukcuogluVector quantized VAE2017
Generating Diverse High-Fidelity Images with VQ-VAE-2Razavi, van den Oord, VinyalsHierarchical VQ-VAE2019
DALL·E: Zero-Shot Text-to-Image Generation (dVAE component)Ramesh et al.Text-to-image with discrete VAE2021
Glow: Generative Flow with Invertible 1x1 ConvolutionsKingma & DhariwalNormalizing flows2018
NICE: Non-linear Independent Components EstimationDinh, Krueger, BengioCoupling layers for flows2015
Density Estimation Using Real-NVPDinh, Sohl-Dickstein, BengioReal-valued Non-Volume Preserving flows2017
PixelCNNvan den Oord et al.Autoregressive image generation2016
Energy-Based Models for Atomic-Scale ModelingvariousEnergy-based generative modeling2020
Attention Is All You NeedVaswani et al.The Transformer architecture2017
Self-Attention with Relative Position RepresentationsShaw, Uszkoreit, VaswaniRelative positional encoding2018
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextDai et al.Segment-level recurrence, longer context2019
Generating Long Sequences with Sparse TransformersChild et al.Sparse attention patterns2019
Longformer: The Long-Document TransformerBeltagy, Peters, CohanLocal + global attention2020
Big Bird: Transformers for Longer SequencesZaheer et al.Random + window + global attention2020
Linformer: Self-Attention with Linear ComplexityWang et al.Low-rank attention approximation2020
Performers: Rethinking Attention (FAVOR+)Choromanski et al.Random features for linear attention2021
FlashAttention: Fast and Memory-Efficient Exact AttentionDao et al.IO-aware exact attention2022
FlashAttention-2: Faster Attention with Better ParallelismDaoImproved FlashAttention2023
Multi-Query AttentionShazeerShared KV heads for efficiency2019
GQA: Training Generalized Multi-Query Transformer ModelsAinslie et al.Grouped-query attention2023
RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE)Su et al.Rotary positional embeddings2021
ALiBi: Train Short, Test Long (Attention with Linear Biases)Press et al.Position via attention biases2022
Reformer: The Efficient TransformerKitaev, Kaiser, LevskayaLSH attention, reversible layers2020
Set TransformerLee et al.Transformers for set-structured data2019
Universal TransformersDehghani et al.Depth-wise recurrent Transformer2019
Scaling Transformers to 1M tokens (Ring Attention)Liu et al.Blockwise attention for long context2023
Extending Context Window of LLMs via Positional InterpolationChen et al.Context extension via interpolation2023
YaRN: Efficient Context Window ExtensionPeng et al.NTK-aware interpolation for RoPE2023
Distributed Representations of Words (Word2Vec)Mikolov et al.Word embeddings2013
Efficient Estimation of Word Representations in Vector SpaceMikolov et al.Skip-gram, CBOW2013
GloVe: Global Vectors for Word RepresentationPennington, Socher, ManningGlobal co-occurrence embeddings2014
Enriching Word Vectors with Subword Information (FastText)Bojanowski et al.Subword embeddings2017
Deep Contextualized Word Representations (ELMo)Peters et al.Contextualized embeddings via biLSTM2018
Semi-supervised Sequence LearningDai & LeLM pretraining for NLP2015
Universal Language Model Fine-tuning (ULMFiT)Howard & RuderTransfer learning for NLP2018
BERT: Pre-training of Deep Bidirectional TransformersDevlin et al.Masked LM pretraining2019
Improving Language Understanding by Generative Pre-Training (GPT-1)Radford et al.Generative pretraining2018
Language Models are Unsupervised Multitask Learners (GPT-2)Radford et al.Larger GPT, zero-shot abilities2019
RoBERTa: A Robustly Optimized BERT Pretraining ApproachLiu et al.Better BERT training recipe2019
ALBERT: A Lite BERTLan et al.Parameter-efficient BERT2020
DistilBERTSanh et al.Distilled BERT: smaller, faster2019
XLNet: Generalized Autoregressive PretrainingYang et al.Permutation language modeling2019
ELECTRA: Pre-training Text Encoders as DiscriminatorsClark et al.Replaced token detection2020
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionHe et al.Disentangled attention mechanism2021
Exploring the Limits of Transfer Learning with a Unified T5Raffel et al.Text-to-text framework2020
BART: Denoising Sequence-to-Sequence Pre-trainingLewis et al.Denoising autoencoder pretraining2020
mBERT / XLM: Cross-lingual Language Model PretrainingConneau & LampleMultilingual pretraining2019
XLM-RoBERTaConneau et al.Cross-lingual at scale2020
Sentence-BERT: Sentence Embeddings using Siamese BERTReimers & GurevychSentence embeddings2019
ERNIE: Enhanced Language RepresentatioN with Informative EntitiesZhang et al.Knowledge-enhanced pretraining2019
SpanBERT: Improving Pre-training by Representing and Predicting SpansJoshi et al.Span-level pretraining2020
Longformer for Long Document UnderstandingBeltagy et al.Efficient attention for long docs2020
Unified Pre-training for Program Understanding and Generation (UniXcoder)Guo et al.Pretraining for code2022
Language Models are Few-Shot Learners (GPT-3)Brown et al.175B params, in-context learning2020
GPT-4 Technical ReportOpenAIMultimodal LLM, SOTA benchmarks2023
PaLM: Scaling Language Modeling with PathwaysChowdhery et al.540B model, Pathways architecture2022
PaLM 2 Technical ReportGoogleImproved multilingual, reasoning2023
Gemini: A Family of Highly Capable Multimodal ModelsGoogle DeepMindNatively multimodal LLM2023
LLaMA: Open and Efficient Foundation Language ModelsTouvron et al.Open-weight efficient LLMs2023
Llama 2: Open Foundation and Fine-Tuned Chat ModelsTouvron et al.Open chat models with RLHF2023
Llama 3MetaLlama 3 family, 8B–405B2024
The Claude Model Card and EvaluationsAnthropicClaude model documentation2023
Mistral 7BJiang et al.Efficient 7B model, sliding window2023
Mixtral of ExpertsJiang et al.MoE with Mistral2024
Phi-1: Textbooks Are All You NeedGunasekar et al.Small model, high-quality data2023
Phi-2MicrosoftScaling small models with data quality2023
Phi-3 Technical ReportAbdin et al.Small powerful models2024
Chinchilla: Training Compute-Optimal Large Language ModelsHoffmann et al.Optimal data/parameter ratio2022
GopherRae et al.280B model, analysis of scaling2022
OPT: Open Pre-trained TransformerZhang et al.Open 175B model from Meta2022
BLOOM: A 176B-Parameter Open-Access Multilingual LMBigScienceOpen multilingual LLM2023
GLM-130B: An Open Bilingual Pre-Trained ModelZeng et al.Bilingual large model2023
Falcon LLMTIIHigh-quality open LLM2023
Yi: Open Foundation Models01.AIYi model family2024
Qwen Technical ReportAlibabaQwen model family2023
Qwen2 Technical ReportAlibabaImproved Qwen family2024
DeepSeek LLMDeepSeekEfficient large model training2024
DeepSeek-V2: A Strong, Economical, and Efficient MoE LLMDeepSeekMulti-head latent attention + MoE2024
DeepSeek-V3 Technical ReportDeepSeekMoE at frontier performance2024
Gemma: Open Models Based on GeminiGoogle DeepMindOpen Gemini-derived models2024
Command R+CohereRAG-optimized LLM2024
Jamba: A Hybrid Transformer-Mamba ModelAI21 LabsSSM + Transformer hybrid2024
OLMo: Accelerating the Science of Language ModelsGroeneveld et al.Fully open LLM pipeline2024
Training Language Models to Follow Instructions (InstructGPT)Ouyang et al.RLHF for instruction following2022
Learning to Summarize from Human FeedbackStiennon et al.RLHF applied to summarization2020
Constitutional AI: Harmlessness from AI Feedback (CAI)Bai et al.RLAIF: AI self-improvement2022
Direct Preference Optimization (DPO)Rafailov et al.RLHF without RL2023
RLHF: A SurveyCasper et al.Comprehensive RLHF survey2023
PPO: Proximal Policy Optimization AlgorithmsSchulman et al.PPO: backbone of RLHF2017
Self-Instruct: Aligning LMs with Self-Generated InstructionsWang et al.Synthetic instruction generation2023
LIMA: Less Is More for AlignmentZhou et al.1000 examples for alignment2023
Scaling Instruction-Finetuned Language Models (Flan-T5/PaLM)Chung et al.Instruction tuning at scale2022
FLAN: Finetuned Language NetWei et al.Task-diverse instruction tuning2022
The Flan CollectionLongpre et al.Design decisions for instruction tuning2023
Alpaca: A Strong, Replicable Instruction-Following ModelTaori et al.Fine-tuning with GPT-generated data2023
Vicuna: An Open-Source ChatbotChiang et al.ShareGPT-trained chat model2023
WizardLM: Empowering Large Language Models to Follow Complex InstructionsXu et al.Evol-Instruct complexity scaling2023
Orca: Progressive Learning from Complex Explanation TracesMukherjee et al.Learning from GPT-4 explanations2023
Zephyr: Direct Distillation of LM AlignmentTunstall et al.DPO for open models2023
UltraFeedbackCui et al.Large-scale preference dataset2023
Reinforcement Learning from AI Feedback (RLAIF)Lee et al.AI-generated preferences2023
KTO: Model Alignment as Prospect Theoretic OptimizationEthayarajh et al.Kahneman-Tversky optimization2024
ORPO: Monolithic Preference Optimization without Reference ModelHong et al.Odds ratio preference optimization2024
SimPO: Simple Preference Optimization with a Reference-Free RewardMeng et al.Simplified DPO variant2024
LoRA: Low-Rank Adaptation of Large Language ModelsHu et al.Parameter-efficient fine-tuning2022
QLoRA: Efficient Finetuning of Quantized LLMsDettmers et al.4-bit quantized LoRA2023
Prefix-TuningLi & LiangLearnable prefixes for generation2021
P-Tuning v2Liu et al.Deep prompt tuning2022
The Power of Scale for Parameter-Efficient Prompt TuningLester, Al-Rfou, ConstantSoft prompt tuning2021
DoRA: Weight-Decomposed Low-Rank AdaptationLiu et al.Improved LoRA via magnitude/direction2024
Adapters: Parameter-Efficient Transfer LearningHoulsby et al.Adapter modules2019
NEFTune: Noisy Embeddings Improve Instruction FinetuningJain et al.Noise in embeddings during finetuning2023
Self-Play Fine-Tuning (SPIN)Chen et al.Self-play for alignment2024
A Survey on In-Context LearningDong et al.Comprehensive ICL survey2023
What Makes In-Context Learning Work?Min et al.Analyzing ICL mechanisms2022
Rethinking the Role of DemonstrationsMin et al.Ground truth labels not always needed2022
Fantastically Ordered Prompts and Where to Find ThemLu et al.Example ordering matters2022
Calibrate Before Use: Improving Few-Shot Performance of LMsZhao et al.Bias calibration in prompting2021
Large Language Models are Zero-Shot ReasonersKojima et al."Let's think step by step"2022
Prompt Programming for Large Language Models: Beyond the Few-Shot ParadigmReynolds & McDonoughPrompt design strategies2021
Structured Prompting: Scaling In-Context Learning to 1000 ExamplesHao et al.Many-shot ICL2022
Many-Shot In-Context LearningAgarwal et al.Scaling ICL with long context2024
Pre-train, Prompt, and Predict: A Systematic SurveyLiu et al.Prompt-based learning survey2023
The Power of PromptingMicrosoftMedical domain prompting2023
Meta-Prompting: Enhancing LLMs with the Expert Interviewing TechniqueSuzgun & KalaiMeta-prompting framework2024
System 2 AttentionWeston & SukhbaatarRegenerating attention for reasoning2023
Anthropic's Prompt Engineering GuideAnthropicBest practices for Claude prompting2024
OpenAI's Prompt Engineering GuideOpenAIGPT prompting best practices2023
Chain-of-Thought Prompting Elicits Reasoning in LLMsWei et al.CoT prompting2022
Self-Consistency Improves Chain of Thought ReasoningWang et al.Sample-and-vote for CoT2023
Tree of Thoughts: Deliberate Problem Solving with LLMsYao et al.Tree-based reasoning2023
Graph of ThoughtsBesta et al.Graph-structured reasoning2023
Least-to-Most Prompting Enables Complex ReasoningZhou et al.Decompose then solve2023
Automatic Chain of Thought Prompting (Auto-CoT)Zhang et al.Automatic CoT generation2023
Complexity-Based Prompting for Multi-Step ReasoningFu et al.Complexity-guided CoT2023
Program of Thoughts Prompting (PoT)Chen et al.Code-aided reasoning2023
PAL: Program-Aided Language ModelsGao et al.Programs for reasoning2023
Faithful Chain-of-Thought ReasoningLyu et al.Faithful, verifiable CoT2023
Measuring Mathematical Problem Solving With the MATH DatasetHendrycks et al.Math reasoning benchmark2021
GSM8K: Training Verifiers to Solve Math Word ProblemsCobbe et al.Math word problem dataset2021
Let's Verify Step by StepLightman et al.Process reward models for math2023
STaR: Self-Taught ReasonerZelikman et al.Bootstrap reasoning ability2022
Quiet-STaR: LMs Can Teach Themselves to Think Before SpeakingZelikman et al.Implicit reasoning training2024
Orca 2: Teaching Small LMs How to ReasonMitra et al.Strategy selection for reasoning2023
Reflexion: Language Agents with Verbal ReinforcementShinn et al.Self-reflection for improvement2023
o1 / Large Reasoning ModelsOpenAITest-time compute for reasoning2024
DeepSeek-R1: Incentivizing Reasoning Capability in LLMsDeepSeekReasoning via RL2025
Scaling LLM Test-Time Compute OptimallySnell et al.Optimal test-time compute allocation2024
Think before you speak: Training LMs with Pause TokensGoyal et al.Pause tokens for thinking2024
Journey Learning: From Superficial to ReasoningWang et al.Staged reasoning curriculum2024
Mutual Reasoning Makes Smaller LLMs Stronger Problem-SolversQi et al.Cross-model reasoning2024
Beyond Chain-of-Thought: A Survey of Chain-of-X ParadigmsVariousSurvey of CoT variants2024
Rethinking LLM Reasoning: Are LLMs Truly Reasoning or Reciting?VariousAnalyzing reasoning vs memorization2024
Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al.Original RAG paper2020
REALM: Retrieval-Augmented Language Model Pre-TrainingGuu et al.Pretraining with retrieval2020
Dense Passage Retrieval (DPR)Karpukhin et al.Dense retrieval for QA2020
ColBERT: Efficient and Effective Passage SearchKhattab & ZahariaLate interaction retrieval2020
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late InteractionSanthanam et al.Improved ColBERT2022
RETRO: Improving Language Models by Retrieving from Trillions of TokensBorgeaud et al.Retrieval-enhanced LM2022
Atlas: Few-shot Learning with Retrieval Augmented LMsIzacard et al.RAG for few-shot2023
Self-RAG: Learning to Retrieve, Generate, and CritiqueAsai et al.Self-reflective RAG2023
Corrective RAG (CRAG)Yan et al.Self-correcting retrieval2024
Adaptive RAGJeong et al.Query-adaptive retrieval2024
RAPTOR: Recursive Abstractive Processing for Tree-Organized RetrievalSarthi et al.Hierarchical document chunking2024
HyDE: Precise Zero-Shot Dense Retrieval without Relevance LabelsGao et al.Hypothetical document embeddings2023
Lost in the Middle: How LMs Use Long ContextsLiu et al.Position bias in long contexts2023
Retrieval-Augmented Generation: A SurveyGao et al.Comprehensive RAG survey2024
From RAG to Agentic RAGVariousAgentic retrieval patterns2024
Graph RAG: Unlocking LLM Discovery on Narrative Private DataEdge et al.Graph-based RAG2024
Contextual RetrievalAnthropicContextual chunk headers2024
Embedding Models (E5, BGE, GTE)VariousModern embedding models2023-2024
MTEB: Massive Text Embedding BenchmarkMuennighoff et al.Embedding evaluation2023
Matryoshka Representation LearningKusupati et al.Multi-granularity embeddings2022
Toolformer: Language Models Can Teach Themselves to Use ToolsSchick et al.Self-supervised tool learning2023
WebGPT: Browser-Assisted Question-AnsweringNakano et al.LLM with web browsing2022
ART: Automatic Multi-step Reasoning and Tool-useParanjape et al.Automated tool selection2023
Gorilla: Large Language Model Connected with Massive APIsPatil et al.API-calling LLM2023
ToolLLM: Facilitating LLMs to Master 16000+ APIsQin et al.Large-scale API learning2024
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMsLi et al.Tool use benchmark2023
TaskMatrix.AI: Completing Tasks by Connecting Foundation ModelsLiang et al.Foundation model orchestration2023
HuggingGPT: Solving AI Tasks with ChatGPT and HuggingFaceShen et al.LLM as controller of AI models2023
Chameleon: Plug-and-Play Compositional ReasoningLu et al.Compositional tool use2023
NexusRaven: Function Calling LLMNexusflowSpecialized function-calling model2023
Anthropic's Tool Use DocumentationAnthropicTool use patterns for Claude2024
Model Context Protocol (MCP)AnthropicStandard protocol for tool integration2024
RestGPT: Connecting LLMs with Real-World RESTful APIsSong et al.REST API integration2023
MRKL SystemsKarpas et al.Modular Reasoning, Knowledge, Language2022
Faithful Reasoning Using Large Language ModelsCreswell & ShanahanTool-augmented faithful reasoning2022
A Survey on Large Language Model based Autonomous AgentsWang et al.Comprehensive agent survey2023
ReAct: Synergizing Reasoning and Acting in LMsYao et al.Reason-Act paradigm2023
Voyager: An Open-Ended Embodied Agent with LLMsWang et al.Lifelong learning agent in Minecraft2023
Generative Agents: Interactive Simulacra of Human BehaviorPark et al.Believable agent society2023
AutoGPT (technical overview)RichardsAutonomous GPT-4 agent2023
BabyAGINakajimaTask-driven autonomous agent2023
LangChain Agents (technical paper)ChaseAgent framework and chains2023
AgentBench: Evaluating LLMs as AgentsLiu et al.Agent evaluation benchmark2023
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous AgentsLiu et al.Agent orchestration benchmark2023
The Rise and Potential of LLM Based Agents: A SurveyXi et al.Agent capabilities survey2023
Agent Foundation ModelsVariousTraining agents end-to-end2024
OpenAgents: An Open Platform for Language AgentsXie et al.Open agent platform2023
AgentTuning: Enabling Generalized Agent Abilities For LLMsZeng et al.Fine-tuning for agent behavior2023
Cognitive Architectures for Language Agents (CoALA)Sumers et al.Cognitive architecture for agents2023
Language Agents: From Next-Token Prediction to Digital AutomationVariousVision for language agents2024
Agent-as-a-JudgeZhuge et al.Agents evaluating agents2024
AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM AgentsLiu et al.Lightweight agent library2024
Personal LLM Agents: Insights and Survey about the Capability, Efficiency and SecurityLi et al.Personal agent survey2024
Agent Smith: Scaling LLM AgentsVariousScaling agent systems2024
Internet of AgentsChen et al.Networked agent systems2024
Planning with Large Language Models for Code GenerationZhang et al.LLM-based planning2023
LLM+P: Empowering LLMs with Optimal Planning ProficiencyLiu et al.Classical planning with LLMs2023
Describe, Explain, Plan and Select (DEPS)Wang et al.Multi-step planning2023
Plan-and-Solve PromptingWang et al.Plan before solving2023
AdaPlanner: Adaptive Planning from FeedbackSun et al.Feedback-driven planning2023
LATS: Language Agent Tree SearchZhou et al.MCTS-style agent search2024
Inner Monologue: Embodied Reasoning through Planning with Language ModelsHuang et al.Inner monologue for robots2023
SayPlan: Grounding Large Language Models using 3D Scene GraphsRana et al.LLM planning with scene graphs2023
LLM Powered Autonomous Agents (Lilian Weng blog-paper)WengInfluential agent design overview2023
Reasoning with Language Model is Planning with World ModelHao et al.RAP: reasoning as planning2023
Everything of Thoughts: Defying the Law of Penrose TriangleDing et al.External solver + LLM2023
SWE-Agent: Agent-Computer Interfaces Enable Automated Software EngineeringYang et al.Agent-computer interface design2024
Trial and Error: Exploration-Based Trajectory OptimizationWang et al.Exploration-based agents2023
AgentCoder: Multi-Agent-based Code GenerationHuang et al.Agents for coding2024
Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMsYin et al.Modular open-source agents2024
CAMEL: Communicative Agents for "Mind" Exploration of LLM SocietyLi et al.Role-playing multi-agent2023
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent ConversationWu et al.Microsoft multi-agent framework2023
MetaGPT: Meta Programming for Multi-Agent Collaborative FrameworkHong et al.Software dev multi-agent2023
ChatDev: Communicative Agents for Software DevelopmentQian et al.Chat-driven software dev agents2023
AgentVerse: Facilitating Multi-Agent CollaborationChen et al.Multi-agent collaboration2023
Multi-Agent DebateDu et al.Debate improves factuality2023
Improving Factuality and Reasoning via Multi-Agent DebateLiang et al.Debate for accuracy2023
Dynamic LLM-Agent Network: Mixture of AgentsWang et al.Dynamic agent composition2024
Mixture of Agents (MoA)Together AILayered multi-agent inference2024
CrewAI (technical overview)MouraRole-based agent orchestration2024
Society of Mind (SoM) for LLMsZhuge et al.Modular cognitive agents2023
Corex: Pushing the Boundaries of Complex Reasoning through Multi-Agent CollaborationSun et al.Complex reasoning agents2024
MAD: Multi-Agent Debate with Large Language ModelsLiang et al.Structured multi-agent debate2024
AgentScope: A Flexible yet Robust Multi-Agent PlatformGao et al.Scalable agent platform2024
LLM-Blender: Ensembling LLMs with Pairwise RankingJiang et al.Ensemble multiple LLMs2023
Evaluating Large Language Models Trained on Code (Codex)Chen et al.Codex / GitHub Copilot2021
HumanEvalChen et al.Code generation benchmark2021
CodeGen: An Open Large Language Model for CodeNijkamp et al.Open code generation2023
StarCoder: May the Source Be with You!Li et al.Open source code model2023
StarCoder 2Lozhkov et al.Improved StarCoder2024
Code Llama: Open Foundation Models for CodeRozière et al.Meta's code LLM2023
DeepSeek-CoderGuo et al.Competitive code model2024
WizardCoder: Empowering Code LLMs with Evol-InstructLuo et al.Evolved instructions for code2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Jimenez et al.Real-world SWE benchmark2024
SWE-AgentYang et al.Agent for SWE-bench2024
Devin: AI Software EngineerCognitionAutonomous coding agent2024
OpenDevin / OpenHandsWang et al.Open-source Devin alternative2024
Aider: AI Pair Programming in TerminalGauthierTerminal-based AI coding2023
Claude Code (Anthropic)AnthropicCLI agentic coding tool2024
AlphaCode: Competition-Level Code GenerationLi et al.Competitive programming2022
AlphaCode 2Google DeepMindImproved competitive coding2023
CodeT: Code Generation with Generated TestsChen et al.Test-driven code generation2023
Self-Debugging: Teaching LLMs to Self-DebugChen et al.Self-debugging code2023
MBPP: Mostly Basic Python ProgrammingAustin et al.Python code benchmark2021
Repository-Level Code CompletionVariousRepo-level context for code2024
Reinforcement Learning: An IntroductionSutton & BartoRL textbook1998/2018
Q-LearningWatkins & DayanQ-learning algorithm1992
Policy Gradient Methods for RL with Function ApproximationSutton et al.Policy gradient theorem2000
Actor-Critic AlgorithmsKonda & TsitsiklisActor-critic framework2000
Simple Statistical Gradient-Following Algorithms (REINFORCE)WilliamsREINFORCE algorithm1992
Temporal-Difference LearningSuttonTD learning1988
Between MDPs and semi-MDPs: A Framework for Temporal Abstraction (Options)Sutton, Precup, SinghHierarchical RL: options1999
Multi-Agent Reinforcement Learning: A Selective OverviewZhang et al.MARL survey2021
Reward ShapingNg, Harada, RussellPotential-based reward shaping1999
Exploration and Exploitation in RLvariousExplore-exploit tradeoff2000s
Playing Atari with Deep Reinforcement Learning (DQN)Mnih et al.Deep Q-Network2013
Human-Level Control through Deep RL (DQN Nature)Mnih et al.DQN in Nature2015
Deep Reinforcement Learning with Double Q-Learningvan Hasselt et al.Double DQN2016
Prioritized Experience ReplaySchaul et al.Priority-based replay buffer2016
Dueling Network Architectures for Deep RLWang et al.Dueling DQN2016
Asynchronous Methods for Deep RL (A3C)Mnih et al.A3C: parallel RL2016
Continuous Control with Deep RL (DDPG)Lillicrap et al.Deep deterministic policy gradient2016
Trust Region Policy Optimization (TRPO)Schulman et al.TRPO2015
High-Dimensional Continuous Control Using Generalized Advantage Estimation (GAE)Schulman et al.GAE for variance reduction2016
Soft Actor-Critic (SAC)Haarnoja et al.Maximum entropy RL2018
Mastering the Game of Go with Deep Neural Networks (AlphaGo)Silver et al.AlphaGo2016
Mastering Go without Human Knowledge (AlphaGo Zero)Silver et al.Self-play from scratch2017
A General RL Algorithm that Masters Chess, Shogi, and Go (AlphaZero)Silver et al.General game-playing2018
MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned ModelSchrittwieser et al.Model-based + model-free2020
OpenAI FiveOpenAIDota 2 team-play2019
AlphaStar: Mastering StarCraft IIVinyals et al.StarCraft grand-master level2019
Curiosity-Driven Exploration (ICM)Pathak et al.Intrinsic motivation2017
World ModelsHa & SchmidhuberLearn world model, plan in latent space2018
Dream to Control (Dreamer)Hafner et al.Model-based latent imagination2020
DreamerV3: Mastering Diverse Domains through World ModelsHafner et al.General world model RL2023
Multimodal Learning with Deep Boltzmann MachinesSrivastava & SalakhutdinovEarly multimodal learning2012
Visual Question Answering (VQA)Antol et al.VQA task and dataset2015
ViLBERT: Pretraining Task-Agnostic Visiolinguistic RepresentationsLu et al.Two-stream vision-language2019
LXMERT: Learning Cross-Modality Encoder RepresentationsTan & BansalCross-modality pretraining2019
Oscar: Object-Semantics Aligned Pre-trainingLi et al.Object tags for VL alignment2020
UNITER: Universal Image-Text Representation LearningChen et al.Unified VL pretraining2020
VinVL: Revisiting Visual Representations in VL ModelsZhang et al.Better visual features for VL2021
Perceiver: General Perception with Iterative AttentionJaegle et al.Modality-agnostic architecture2021
Perceiver IOJaegle et al.Generalized multimodal IO2022
Data2Vec: A General Framework for Self-Supervised LearningBaevski et al.Unified self-supervised across modalities2022
ImageBind: One Embedding Space to Bind Them AllGirdhar et al.Six-modality binding2023
Meta-Transformer: A Unified Framework for Multimodal LearningZhang et al.12-modality transformer2023
Any-to-Any Generation via Composable Diffusion (CoDi)Tang et al.Cross-modal generation2023
NExT-GPT: Any-to-Any Multimodal LLMWu et al.End-to-end multimodal generation2023
4M: Massively Multimodal Masked ModelingBachmann et al.Multi-task multimodal pretraining2023
CLIP: Learning Transferable Visual Models from Natural Language SupervisionRadford et al.Contrastive language-image pretraining2021
ALIGN: Scaling Up Visual and Vision-Language Representation LearningJia et al.Noisy image-text pairs at scale2021
Florence: A New Foundation Model for Computer VisionYuan et al.Unified vision foundation2021
Flamingo: A Visual Language Model for Few-Shot LearningAlayrac et al.Few-shot VL model2022
BLIP: Bootstrapping Language-Image Pre-trainingLi et al.Bootstrapped VL pretraining2022
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and LLMsLi et al.Q-Former for vision-language2023
LLaVA: Visual Instruction TuningLiu et al.Visual instruction following2023
LLaVA-1.5Liu et al.Improved visual chat2023
InstructBLIPDai et al.Instruction-tuned BLIP-22023
MiniGPT-4: Enhancing Vision-Language UnderstandingZhu et al.GPT-4 vision abilities in small model2023
Qwen-VL: A Versatile VLMBai et al.Versatile VLM2023
CogVLM: Visual Expert for Pretrained Language ModelsWang et al.Deep visual-language fusion2023
InternVL: Scaling Up Vision Foundation ModelsChen et al.Scaling VL models2024
Cambrian-1: A Fully Open Vision-Centric Exploration of Multimodal LLMsTong et al.Open multimodal exploration2024
SigLIP: Sigmoid Loss for Language Image Pre-TrainingZhai et al.Simplified contrastive VL2023
SAM: Segment AnythingKirillov et al.Foundation model for segmentation2023
SAM 2: Segment Anything in Images and VideosRavi et al.Video segmentation foundation2024
Grounding DINO: Marrying DINO with Grounded Pre-TrainingLiu et al.Open-set object detection2023
OWL-ViT: Open-World Object Detection with Vision TransformersMinderer et al.Open-vocabulary detection2022
PaLI: A Jointly-Scaled Multilingual Language-Image ModelChen et al.Multilingual VL model2023
Deep Speech: Scaling Up End-to-End Speech RecognitionHannun et al.End-to-end ASR2014
Deep Speech 2Amodei et al.English + Mandarin ASR2016
Listen, Attend and Spell (LAS)Chan et al.Attention-based ASR2016
wav2vec: Unsupervised Pre-training for Speech RecognitionSchneider et al.Self-supervised speech2019
wav2vec 2.0: A Framework for Self-Supervised Learning of SpeechBaevski et al.Contrastive speech pretraining2020
HuBERT: Self-Supervised Speech Representation Learning by Masked PredictionHsu et al.Hidden-unit BERT for speech2021
Whisper: Robust Speech Recognition via Large-Scale Weak SupervisionRadford et al.Universal ASR model2023
Conformer: Convolution-augmented Transformer for SpeechGulati et al.Hybrid conv-transformer for speech2020
Tacotron 2: Natural TTS SynthesisShen et al.Neural text-to-speech2018
FastSpeech 2: Fast and High-Quality End-to-End TTSRen et al.Non-autoregressive TTS2021
VALL-E: Neural Codec Language Models are Zero-Shot TTSWang et al.Voice cloning with LLM2023
XTTS: Massively Multilingual TTSCasanova et al.Multilingual voice synthesis2024
AudioLM: A Language Modeling Approach to Audio GenerationBorsos et al.Audio as language modeling2023
MusicLM: Generating Music from TextAgostinelli et al.Text-to-music2023
Bark: Text-Prompted Generative Audio ModelSunoVersatile audio generation2023
AudioPaLM: A Large Language Model that Can Speak and ListenRubenstein et al.Multimodal speech LLM2023
SeamlessM4T: Massively Multilingual & Multimodal Machine TranslationMeta100+ language speech translation2023
Encodec: High Fidelity Neural Audio CompressionDéfossez et al.Neural audio codec2022
Voicebox: Text-Guided Multilingual Universal Speech GenerationLe et al.Universal speech generation2023
SoundStorm: Efficient Parallel Audio GenerationBorsos et al.Parallel audio decoding2023
Deep Unsupervised Learning using Nonequilibrium ThermodynamicsSohl-Dickstein et al.Diffusion process for generation2015
Denoising Diffusion Probabilistic Models (DDPM)Ho, Jain, AbbeelModern diffusion models2020
Score-Based Generative Modeling through SDESong et al.Score matching + SDEs2021
Denoising Diffusion Implicit Models (DDIM)Song, Meng, ErmonFaster diffusion sampling2021
Classifier-Free Diffusion GuidanceHo & SalimansCFG: standard guidance method2022
High-Resolution Image Synthesis with Latent Diffusion Models (LDM)Rombach et al.Latent space diffusion2022
Stable DiffusionStability AI / Rombach et al.Open image generation2022
DALL·E 2: Hierarchical Text-Conditional Image GenerationRamesh et al.CLIP-guided diffusion2022
Imagen: Photorealistic Text-to-Image Diffusion ModelsSaharia et al.T5-guided diffusion2022
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert DenoisersBalaji et al.Expert denoisers2023
SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisPodell et al.Stable Diffusion XL2023
DALL·E 3Betker et al.Improved text following2023
Midjourney (V5/V6 technical analysis)MidjourneyAesthetic image generation2023-2024
Consistency ModelsSong et al.One-step generation2023
Latent Consistency ModelsLuo et al.Fast latent diffusion2023
Rectified Flow (InstaFlow)Liu et al.Straight trajectory sampling2023
Stable Diffusion 3 (MM-DiT)Esser et al.Multimodal DiT architecture2024
FLUXBlack Forest LabsNext-gen flow-based generation2024
ControlNet: Adding Conditional Control to Text-to-Image DiffusionZhang & AgrawalaStructural control for diffusion2023
IP-Adapter: Text Compatible Image Prompt AdapterYe et al.Image prompt adapters2023
DreamBooth: Fine Tuning Text-to-Image Models for Subject-Driven GenerationRuiz et al.Subject customization2023
Textual Inversion: An Image is Worth One WordGal et al.Concept learning via tokens2023
InstructPix2Pix: Learning to Follow Image Editing InstructionsBrooks et al.Instruction-based image editing2023
Scalable Diffusion Models with Transformers (DiT)Peebles & XieTransformer-based diffusion2023
Flow Matching for Generative ModelingLipman et al.Optimal transport flows2023
Video Diffusion ModelsHo et al.Diffusion for video2022
Imagen Video: High Definition Video GenerationHo et al.Cascaded video diffusion2022
Make-A-Video: Text-to-Video Generation without Text-Video DataSinger et al.Text-to-video from image data2023
VideoPoet: A Large Language Model for Zero-Shot Video GenerationKondratyuk et al.LLM-based video generation2024
Sora: Video Generation Models as World SimulatorsOpenAIFrontier video generation2024
Stable Video DiffusionBlattmann et al.Open video generation2023
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion ModelsGuo et al.Motion modules for SD2023
Kling (technical analysis)KuaishouChinese video generation2024
Gen-2: Generating Novel Videos with Text, Images, or ClipsRunwayMultimodal video generation2023
VideoCrafter: A Toolkit for Text-to-Video GenerationChen et al.Open video toolkit2023
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-TrainingTong et al.Self-supervised video2022
TimeSformer: Is Space-Time Attention All You Need for Video Understanding?Bertasius et al.Space-time attention2021
ViViT: A Video Vision TransformerArnab et al.Video transformer2021
InternVideo: General Video Foundation ModelsWang et al.Video foundation model2022
Video-LLaVALin et al.Video understanding with VLMs2024
Semi-Supervised Classification with Graph Convolutional Networks (GCN)Kipf & WellingGraph convolutions2017
Attention-Based Graph Neural Network (GAT)Veličković et al.Graph attention2018
Inductive Representation Learning on Large Graphs (GraphSAGE)Hamilton, Ying, LeskovecScalable graph learning2017
How Powerful are Graph Neural Networks? (GIN)Xu et al.GNN expressivity analysis2019
Message Passing Neural Networks (MPNN)Gilmer et al.Unified message passing framework2017
Graph Isomorphism Network (GIN)Xu et al.WL-test equivalent GNN2019
Relational Graph Convolutional Networks (R-GCN)Schlichtkrull et al.GCNs for knowledge graphs2018
Graph Transformer NetworksYun et al.Transformers on graphs2019
DeepWalk: Online Learning of Social RepresentationsPerozzi, Al-Rfou, SkienaGraph embeddings via random walks2014
Node2Vec: Scalable Feature Learning for NetworksGrover & LeskovecBiased random walk embeddings2016
A Comprehensive Survey on Graph Neural NetworksWu et al.GNN survey2021
Equivariant Graph Neural NetworksSatorras et al.E(n)-equivariant GNNs2021
Graph Neural Networks: A Review of Methods and ApplicationsZhou et al.GNN applications review2020
Spectral Graph Theory and Deep LearningBruna et al.Spectral graph convolutions2014
Geometric Deep LearningBronstein et al.Unified geometric DL framework2021
Deep Compression: Pruning, Quantization, Huffman CodingHan et al.Three-stage compression2016
MobileNets: Efficient CNNs for Mobile Vision ApplicationsHoward et al.Depthwise separable convolutions2017
MobileNetV2: Inverted Residuals and Linear BottlenecksSandler et al.Inverted residuals2018
ShuffleNet: Extremely Efficient CNN for Mobile DevicesZhang et al.Channel shuffle2018
SqueezeNet: AlexNet-level Accuracy with 50x Fewer ParametersIandola et al.Compact architectures2016
LLM.int8(): 8-bit Matrix Multiplication for TransformersDettmers et al.8-bit quantization for LLMs2022
GPTQ: Accurate Post-Training Quantization for GPTFrantar et al.One-shot weight quantization2023
AWQ: Activation-aware Weight QuantizationLin et al.Activation-aware quantization2024
SqueezeLLM: Dense-and-Sparse QuantizationKim et al.Ultra-low bit quantization2024
The Era of 1-bit LLMs: Training-time 1.58-bit Models (BitNet)Ma et al.Ternary weight LLMs2024
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-ShotFrantar & AlistarhOne-shot pruning for LLMs2023
Wanda: A Simple and Effective Pruning ApproachSun et al.Pruning by weights and activations2024
Flash-Decoding for Long Context InferenceDao et al.Parallel decoding across KV2023
Speculative DecodingLeviathan, Kalman, MatiasDraft model for fast decoding2023
Medusa: Simple LLM Inference AccelerationCai et al.Multi-head speculative decoding2024
vLLM: Efficient Memory Management for LLM Serving (PagedAttention)Kwon et al.Paged KV cache management2023
TensorRT-LLMNVIDIAOptimized LLM inference2023
llama.cppGerganovCPU inference for LLMs2023
Ollama (technical overview)OllamaLocal LLM deployment2024
GGUF: GPT-Generated Unified FormatGerganovEfficient model format2023
Adaptive Mixtures of Local ExpertsJacobs et al.Original MoE concept1991
Outrageously Large Neural Networks: The Sparsely-Gated MoE LayerShazeer et al.Sparsely-gated MoE at scale2017
Switch Transformers: Scaling to Trillion Parameter ModelsFedus, Zoph, ShazeerSimplified MoE routing2022
GLaM: Efficient Scaling of Language Models with MoEDu et al.1.2T parameter MoE2022
ST-MoE: Designing Stable and Transferable Sparse Expert ModelsZoph et al.Stable MoE training2022
Mixtral of ExpertsJiang et al.Open MoE model2024
DeepSeekMoE: Towards Ultimate Expert SpecializationDai et al.Fine-grained expert design2024
Unified Scaling Laws for Routed Language ModelsClark et al.Scaling laws for MoE2022
MoE-MambaPioro et al.MoE with state-space models2024
Skywork-MoE: A Deep Dive into Training Techniques for MoE LLMsWei et al.MoE training analysis2024
Distilling the Knowledge in a Neural NetworkHinton, Vinyals, DeanKnowledge distillation2015
Born Again Neural NetworksFurlanello et al.Self-distillation2018
Be Your Own Teacher: Improve the Performance of CNNs via Self-DistillationZhang et al.Self-distillation in CNNs2019
TinyBERT: Distilling BERT for Natural Language UnderstandingJiao et al.Distilling BERT2020
MiniLM: Deep Self-Attention DistillationWang et al.Attention-based distillation2020
How Good Are You at Transferring? A Survey on Transfer LearningZhuang et al.Transfer learning survey2020
Domain Adaptation for Object RecognitionSaenko et al.Visual domain adaptation2010
DeCAF: A Deep Convolutional Activation FeatureDonahue et al.CNN feature transfer2014
How Transferable are Features in Deep Neural Networks?Yosinski et al.Layer transferability analysis2014
LLM2LLM: Boosting LLMs with Novel Iterative Data EnhancementLee et al.LLM-to-LLM distillation2024
Overcoming Catastrophic Forgetting in Neural Networks (EWC)Kirkpatrick et al.Elastic weight consolidation2017
Progressive Neural NetworksRusu et al.Growing network capacity2016
Continual Lifelong Learning with Neural Networks: A ReviewParisi et al.Continual learning survey2019
Learning without ForgettingLi & HoiemKnowledge distillation for continual learning2017
PackNet: Adding Multiple Tasks to a Single NetworkMallya & LazebnikIterative pruning for multi-task2018
Experience Replay for Continual LearningRolnick et al.Replay buffers2019
A Comprehensive Survey of Continual LearningDe Lange et al.Modern continual learning survey2022
Continual Pre-training of Language ModelsGururangan et al.Domain-adaptive pretraining2020
TRACE: A Comprehensive Benchmark for Continual Learning in LLMsWang et al.LLM continual learning benchmark2024
Online Continual Learning for LLMsVariousOnline adaptation of LLMs2024
Momentum Contrast for Unsupervised Visual Representation Learning (MoCo)He et al.MoCo: momentum contrast2020
MoCo v2Chen et al.Improved MoCo2020
A Simple Framework for Contrastive Learning (SimCLR)Chen et al.SimCLR: simple contrastive2020
Bootstrap Your Own Latent (BYOL)Grill et al.No negatives needed2020
Barlow Twins: Self-Supervised Learning via Redundancy ReductionZbontar et al.Redundancy reduction2021
VICReg: Variance-Invariance-Covariance RegularizationBardes, Ponce, LeCunVariance-invariance-covariance2022
DINO: Emerging Properties in Self-Supervised Vision TransformersCaron et al.Self-distillation for ViT2021
DINOv2: Learning Robust Visual FeaturesOquab et al.Improved self-supervised vision2023
Masked Autoencoders Are Scalable Vision Learners (MAE)He et al.Masked image modeling2022
SimMIM: A Simple Framework for Masked Image ModelingXie et al.Simplified masked image modeling2022
BEiT: BERT Pre-Training of Image TransformersBao, Dong, WeiBERT-style pretraining for vision2022
Supervised Contrastive LearningKhosla et al.SupCon loss2020
Understanding Contrastive Representation LearningArora et al.Theoretical analysis of contrastive learning2019
Self-Supervised Learning: Generative or Contrastive (Survey)Liu et al.SSL survey2021
I-JEPA: Image Joint-Embedding Predictive ArchitectureAssran et al.LeCun's JEPA for vision2023
Communication-Efficient Learning of Deep Networks from Decentralized Data (FedAvg)McMahan et al.Federated averaging2017
Federated Learning: Challenges, Methods, and Future DirectionsLi et al.FL survey2020
Advances and Open Problems in Federated LearningKairouz et al.Comprehensive FL challenges2021
Deep Learning with Differential PrivacyAbadi et al.DP-SGD2016
The Algorithmic Foundations of Differential PrivacyDwork & RothDP theory foundations2014
SecureML: A System for Scalable Privacy-Preserving MLMohassel & ZhangSecure computation for ML2017
Membership Inference Attacks Against ML ModelsShokri et al.Privacy attacks on models2017
Machine UnlearningBourtoule et al.Removing data from trained models2021
FedProx: Federated Optimization in Heterogeneous NetworksLi et al.Handling non-IID data in FL2020
Scaffold: Stochastic Controlled Averaging for FLKarimireddy et al.Variance reduction in FL2020
ImageNet Large Scale Visual Recognition ChallengeRussakovsky et al.ImageNet benchmark2015
SQuAD: 100,000+ Questions for Machine ComprehensionRajpurkar et al.Reading comprehension2016
GLUE: A Multi-Task Benchmark for NLUWang et al.NLU benchmark2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language UnderstandingWang et al.Harder NLU benchmark2019
MMLU: Measuring Massive Multitask Language UnderstandingHendrycks et al.57-subject knowledge test2021
MMLU-ProWang et al.Harder MMLU variant2024
BIG-Bench: Beyond the Imitation GameSrivastava et al.204 collaborative tasks2023
HellaSwag: Can a Machine Really Finish Your Sentence?Zellers et al.Common-sense completion2019
ARC: Think You Have Solved Question Answering?Clark et al.Science reasoning2018
WinoGrande: An Adversarial Winograd Schema ChallengeSakaguchi et al.Common-sense reasoning2020
TruthfulQA: Measuring How Models Mimic Human FalsehoodsLin, Hilton, EvansTruthfulness benchmark2022
Chatbot Arena / LMSYS LeaderboardZheng et al.Human preference rankings2023
MT-BenchZheng et al.Multi-turn conversation benchmark2023
AlpacaEvalLi et al.Automatic LLM evaluation2023
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkRein et al.Expert-level questions2023
LiveBench: A Challenging, Contamination-Free LLM BenchmarkWhite et al.Contamination-free eval2024
MATH and GSM8K (mentioned earlier)Hendrycks; CobbeMath evaluation2021
HumanEval and MBPP (mentioned earlier)Chen; AustinCode evaluation2021
MGSM: Multilingual Grade School MathShi et al.Multilingual math2023
SimpleQAOpenAIFactuality benchmark2024
IFEval: Instruction-Following EvaluationZhou et al.Instruction following2023
MuSR: Multi-Step Soft ReasoningSprague et al.Multi-step reasoning2024
Agentic Benchmarks: WebArena, OSWorld, etc.Zhou; Xie et al.Agent environment benchmarks2024
COCO: Common Objects in ContextLin et al.Object detection/captioning2014
Holistic Evaluation of Language Models (HELM)Liang et al.Comprehensive LM evaluation2023
Concrete Problems in AI SafetyAmodei et al.Practical safety problems2016
AI Alignment: A Comprehensive SurveyJi et al.Alignment survey2024
The Alignment Problem (overview)ChristianAI alignment book2020
Reward Hacking in Reinforcement LearningSkalse et al.Reward misspecification2022
Scalable Agent Alignment via Reward Modeling (Anthropic)Leike et al.Recursive reward modeling2018
Language Models Don't Always Say What They Think: Unfaithful ExplanationsTurpin et al.Sycophancy and unfaithfulness2024
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingHubinger et al.Backdoor persistence2024
Risks from Learned Optimization in Advanced ML Systems (Mesa-Optimizers)Hubinger et al.Inner alignment2019
Goal Misgeneralization: Why Correct Specifications Aren't EnoughShah et al.Goal misgeneralization2022
Red Teaming Language Models to Reduce HarmsGanguli et al.Red teaming methodology2022
Red Teaming Language Models with Language ModelsPerez et al.Automated red teaming2022
Universal and Transferable Adversarial Attacks on Aligned LMsZou et al.GCG attack on aligned LLMs2023
Jailbroken: How Does LLM Safety Training Fail?Wei et al.Jailbreak analysis2024
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical BehaviorPan et al.Safety-performance tradeoffs2023
The Model Spec (OpenAI)OpenAILLM behavior specification2024
Claude's Character (Anthropic)AnthropicValues-based AI behavior2024
Responsible Scaling Policies (RSPs)AnthropicScaling safety commitments2023
Managing AI Risks in an Era of Rapid ProgressBengio et al.Open letter on AI risks2023
Pause Giant AI Experiments: An Open LetterFuture of LifeAI moratorium call2023
On the Dangers of Stochastic ParrotsBender et al.LLM risks and harms2021
Fairness and Machine LearningBarocas, Hardt, NarayananML fairness textbook2019
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender ClassificationBuolamwini & GebruBias in facial recognition2018
Datasheets for DatasetsGebru et al.Dataset documentation2021
Model Cards for Model ReportingMitchell et al.Model documentation2019
Blueprint for an AI Bill of RightsWhite HouseAI rights framework2022
"Why Should I Trust You?" Explaining the Predictions of Any Classifier (LIME)Ribeiro et al.Local interpretable explanations2016
A Unified Approach to Interpreting Model Predictions (SHAP)Lundberg & LeeShapley values for ML2017
Grad-CAM: Visual Explanations from Deep NetworksSelvaraju et al.Gradient-based attention maps2017
Attention is not ExplanationJain & WallaceQuestioning attention as explanation2019
Attention is not not ExplanationWiegreffe & PinterDefense of attention explanations2019
Network Dissection: Quantifying Interpretability of Deep Visual RepresentationsBau et al.Unit-level interpretability2017
Zoom In: An Introduction to CircuitsOlah et al.Mechanistic interpretability2020
A Mathematical Framework for Transformer CircuitsElhage et al.Transformer mechanistic analysis2021
Toy Models of SuperpositionElhage et al.Superposition in neural networks2022
Towards Monosemanticity: Decomposing Language Models with Dictionary LearningBricken et al.Sparse autoencoders for interpretability2023
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 SonnetTempleton et al.SAE at scale on Claude2024
Representation Engineering: A Top-Down Approach to AI TransparencyZou et al.Representation-level interventions2023
Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelLi et al.ITI for truthfulness2024
Sparse Probing for LLM RepresentationsGurnee et al.Probing LLM internals2023
Circuit Discovery in LLMsConmy et al.Automated circuit finding2023
Interpretability in the WildBills et al.GPT-4 explaining neurons2023
The Geometry of Truth: Emergent Linear Structure in LLM RepresentationsMarks & TegmarkTruth geometry in LLMs2024
Polysemanticity and Capacity in Neural NetworksAnthropicUnderstanding polysemantic neurons2023
Probing Classifiers: Promises, Shortcomings, and AdvancesBelinkovProbing methodology2022
Causal Abstraction for Faithful Model InterpretationGeiger et al.Causal analysis of models2023
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan)Ahn et al.LLMs grounded in robot abilities2022
RT-1: Robotics Transformer for Real-World ControlBrohan et al.Transformer for robot control2023
RT-2: Vision-Language-Action ModelsBrohan et al.VLM as robot policy2023
Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsOpen X-EmbodimentCross-embodiment dataset2023
PaLM-E: An Embodied Multimodal Language ModelDriess et al.Embodied multimodal LM2023
Code as Policies: Language Model Programs for Embodied ControlLiang et al.Code for robot control2023
Language Models as Zero-Shot PlannersHuang et al.LLM planning for robots2022
TidyBot: Personalized Robot Assistance with LLMsWu et al.Personal robot assistant2023
RoboCat: A Self-Improving Generalist Agent for Robotic ManipulationBousmalis et al.Self-improving robot2023
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ALOHA)Zhao et al.Low-cost bimanual manipulation2023
Octo: An Open-Source Generalist Robot PolicyTeam et al.Open robot foundation model2024
π0: A Vision-Language-Action Flow Model for General Robot ControlPhysical IntelligenceGeneral-purpose robot control2024
NVIDIA Isaac and Omniverse for RoboticsNVIDIASimulation for robotics2024
Dexterous Manipulation with Reinforcement LearningOpenAI (Rubik's cube)Dexterous robot hand2019
Mobile ALOHA: Learning Bimanual Mobile ManipulationFu et al.Mobile manipulation2024
World Models (Ha & Schmidhuber)Ha & SchmidhuberLearning world models2018
A Path Towards Autonomous Machine IntelligenceLeCunJEPA and world model vision2022
Genie: Generative Interactive EnvironmentsBruce et al.Playable world generation2024
DIAMOND: Diffusion for World ModelingAlonso et al.Diffusion world models2024
UniSim: Learning Interactive Real-World SimulatorsYang et al.Universal simulator2024
Genesis: Generative and Universal Physics EngineVariousPhysics simulation platform2024
Learning General World Models in a Handful of Reward-Free DeploymentsSeo et al.Reward-free world models2023
GAIA-1: A Generative World Model for Autonomous DrivingHu et al.Driving world model2023
Language Models Meet World ModelsHao et al.LLM-world model integration2024
Video as World ModelVariousVideo generation as simulation2024
AlphaFold: Highly Accurate Protein Structure PredictionJumper et al.Protein folding solved2021
AlphaFold 2 (full paper)Jumper et al.Detailed AlphaFold architecture2021
AlphaFold 3Abramson et al.All-atom structure prediction2024
RoseTTAFoldBaek et al.Three-track protein prediction2021
ESM-2: Language Models of Protein Sequences at the Scale of EvolutionLin et al.Protein language model2023
ProteinMPNN: Robust Protein Sequence DesignDauparas et al.Inverse protein folding2022
GNoME: Scaling Deep Learning for Materials DiscoveryMerchant et al.2.2M new materials2023
MatterGen: A Generative Model for Inorganic Materials DesignZeni et al.Materials generation2024
FunSearch: Mathematical Discoveries from Program SearchRomera-Paredes et al.LLM for mathematical discovery2024
AlphaGeometry: Solving Olympiad Geometry without Human DemonstrationsTrinh et al.AI for geometry proofs2024
AlphaProof and AlphaGeometry 2Google DeepMindIMO-level mathematics2024
Med-PaLM: Large Language Models Encode Clinical KnowledgeSinghal et al.Medical AI2023
Med-PaLM 2Singhal et al.Expert-level medical QA2023
PMC-LLaMA: Towards Building Open-source Language Models for MedicineWu et al.Open medical LLM2024
BioGPT: Generative Pre-trained Transformer for Biomedical TextLuo et al.Biomedical LM2022
Galactica: A Large Language Model for ScienceTaylor et al.Science-focused LLM2022
ScienceQA: Science Question AnsweringLu et al.Science reasoning2022
ChemCrow: Augmenting LLMs with Chemistry ToolsBran et al.Chemistry agent2023
ClimateBERTWebersinke et al.Climate NLP2022
GraphCast: Learning Skillful Medium-Range Global Weather ForecastingLam et al.ML weather forecasting2023
Scaling Laws for Neural Language ModelsKaplan et al.Power-law scaling relationships2020
Training Compute-Optimal LLMs (Chinchilla)Hoffmann et al.Data-compute tradeoff2022
Emergent Abilities of Large Language ModelsWei et al.Emergence at scale2022
Are Emergent Abilities of LLMs a Mirage?Schaeffer et al.Questioning emergent abilities2023
Scaling Data-Constrained Language ModelsMuennighoff et al.Data-limited scaling2023
Scaling Laws for Autoregressive Generative ModelingHenighan et al.Cross-modality scaling2020
Beyond Neural Scaling LawsCaballero et al.Broken neural scaling laws2022
Scaling Laws for Reward Model OveroptimizationGao et al.Reward model scaling2023
Scaling Vision TransformersZhai et al.ViT scaling laws2022
Observational Scaling LawsRuan et al.Predicting LLM capabilities2024
Textbooks Are All You Need (Phi-1 scaling)Gunasekar et al.Data quality over quantity2023
The Pile: An 800GB Dataset for Diverse Text ModelingGao et al.Large-scale training data2020
RedPajama: An Open Dataset for Training LLMsTogether AIOpen pretraining data2023
FineWeb: Decanting the Web for the Finest Text DataPenedo et al.High-quality web data2024
DataComp: In Search of the Next Generation of Multimodal DatasetsGadre et al.Multimodal data curation2023
Mamba: Linear-Time Sequence Modeling with Selective State SpacesGu & DaoSelective state space model2023
Mamba-2: Structured State Space DualityDao & GuImproved Mamba2024
Efficiently Modeling Long Sequences with Structured State Spaces (S4)Gu et al.S4: structured SSMs2022
Hyena Hierarchy: Towards Larger Convolutional Language ModelsPoli et al.Long convolution alternative2023
RWKV: Reinventing RNNs for the Transformer EraPeng et al.Linear RNN at transformer quality2023
Griffin: Mixing Gated Linear Recurrences with Local AttentionDe et al.Hybrid recurrence + attention2024
RetNet: Retentive Network: A Successor to TransformerSun et al.Retention mechanism2023
xLSTM: Extended Long Short-Term MemoryBeck et al.Modernized LSTM2024
Jamba: Hybrid Transformer-MambaAI21 LabsSSM-Transformer hybrid2024
Mixture of Depths: Dynamically Allocating Compute in Transformer-Based ModelsRaposo et al.Dynamic compute allocation2024
Neural Architecture Search (NAS)Zoph & LeAutomated architecture design2017
EfficientNet (NAS-derived)Tan & LeNAS for efficient models2019
Differential TransformerYe et al.Differential attention mechanism2024
Kolmogorov-Arnold Networks (KAN)Liu et al.Alternative to MLPs2024
TTT: Learning to (Learn at Test Time)Sun et al.Test-time training layers2024
Sparks of AGI: Early Experiments with GPT-4Bubeck et al.AGI capabilities analysis2023
Levels of AGIMorris et al.Taxonomy of AGI levels2023
The Bitter LessonSuttonCompute over engineering2019
Reward is EnoughSilver et al.Reward maximization for intelligence2021
Artificial General Intelligence: Concept, State of the Art, and Future ProspectsGoertzelAGI survey2014
On the Measure of IntelligenceCholletARC benchmark, intelligence definition2019
ARC Prize and ARC-AGIChollet et al.Abstraction reasoning challenge2024
Language Agent Tree Search Unifies Reasoning, Acting, and PlanningZhou et al.Unified agent framework2024
Superintelligence: Paths, Dangers, Strategies (summary)BostromSuperintelligence risks2014
The Case for AI SafetyVariousModern safety arguments2024
Translating Embeddings for Modeling Multi-relational Data (TransE)Bordes et al.Knowledge graph embeddings2013
Knowledge Graph Embedding by Translating on Hyperplanes (TransH)Wang et al.Improved KG embeddings2014
RotatE: Knowledge Graph Embedding by Relational RotationSun et al.Rotation-based KG embedding2019
KGQA: Knowledge Graph Question AnsweringVariousQA over knowledge graphs2021
Unifying LLMs and Knowledge Graphs: A RoadmapPan et al.LLM + KG integration survey2024
Think-on-Graph: Deep and Responsible Reasoning of LLMs on KGsSun et al.LLM reasoning with KGs2024
KnowPrompt: Knowledge-aware Prompt-tuningChen et al.Knowledge-enhanced prompting2022
Wikidata and Knowledge Graphs at ScaleVrandečić & KrötzschWikidata for AI2014
QA-GNN: Reasoning with Language Models and KGsYasunaga et al.GNN + LLM for QA2021
Graph-based Deep Learning for NLPVariousGraph methods in NLP survey2023
Named Entity Recognition with Bidirectional LSTM-CNNsChiu & NicholsNER with deep learning2016
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFMa & HovyLSTM-CRF for NER2016
Neural Machine Translation of Rare Words with Subword Units (BPE)Sennrich et al.Byte-pair encoding2016
SentencePieceKudo & RichardsonUnigram/BPE tokenizer2018
A Call for Clarity in Reporting BLEU ScoresPostMT evaluation standards2018
BERTScore: Evaluating Text Generation with BERTZhang et al.Semantic evaluation metric2020
ROUGE: A Package for Automatic Evaluation of SummariesLinSummarization evaluation2004
Get To The Point: Summarization with Pointer-Generator NetworksSee et al.Copy mechanism for summarization2017
Abstractive Text Summarization using Seq-to-Seq RNNsNallapati et al.Neural abstractive summarization2016
A Structured Self-Attentive Sentence EmbeddingLin et al.Self-attention for sentence embedding2017
Self-InstructWang et al.LLM-generated instructions2023
Textbooks Are All You NeedGunasekar et al.Synthetic textbook data2023
Cosmopedia: Creating Large-Scale Synthetic DataHuggingFaceLarge synthetic dataset2024
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language ModelingMaini et al.Web data rephrasing2024
Magpie: Alignment Data Synthesis from ScratchXu et al.Synthetic alignment data2024
WizardLM: Empowering LLMs to Follow Complex InstructionsXu et al.Evol-Instruct2023
AgentInstruct: Toward Generative Teaching with Agentic FlowsMitra et al.Agent-generated training data2024
SPIN: Self-Play Fine-Tuning Converts Weak LMs to StrongChen et al.Self-play data generation2024
DataDreamer: A Tool for Synthetically Generating, Transforming, and Analyzing NLP DataPatel et al.Synthetic data toolkit2024
Scaling Synthetic Data Creation with 1B PersonasChan et al.Persona-driven synthetic data2024
Memorizing TransformersWu et al.kNN-augmented attention2022
∞-former: Infinite Memory TransformerMartins et al.Unbounded memory2022
LongNet: Scaling Transformers to 1B TokensDing et al.Dilated attention2023
Leave No Context Behind: Efficient Infinite-Context Transformers with Infini-attentionMunkhdalai et al.Compressive memory2024
Needle in a Haystack EvaluationKamradtLong context evaluation2024
Ruler: What's the Real Context Size of Your LLM?Hsieh et al.Long context benchmark2024
MemGPT: Towards LLMs as Operating SystemsPacker et al.Virtual context management2023
LongRoPE: Extending LLM Context Window Beyond 2M TokensDing et al.Extreme context extension2024
Claude's 200K Context (Anthropic)AnthropicLong context in production2024
Gemini 1.5 Pro: Long Context (1M tokens)Google DeepMindMillion-token context2024
Outlines: Structured Text GenerationWillard & LoufGuaranteed JSON/schema output2023
Guidance: Constrained GenerationMicrosoftConstrained LLM output2023
Grammar of Thought: LLM Structured ReasoningVariousGrammar-constrained reasoning2024
LMQL: Language Model Query LanguageBeurer-Kellner et al.Query language for LLMs2023
DSPy: Compiling Declarative Language Model Calls into PipelinesKhattab et al.Programming framework for LMs2024
SGLang: Efficient Execution of Structured LM ProgramsZheng et al.Structured generation runtime2024
Instructor: Structured Outputs from LLMsLiuPydantic-based extraction2024
TypeChatMicrosoftType-safe LLM outputs2023
JSON Mode / Structured Outputs (OpenAI, Anthropic)VariousNative structured output2024
Semantic KernelMicrosoftAI orchestration SDK2023
BloombergGPT: A Large Language Model for FinanceWu et al.Finance-specific LLM2023
FinGPT: Open-Source Financial Large Language ModelsYang et al.Open finance LLM2023
FinBERT: Financial Sentiment AnalysisAraciBERT for finance2019
Deep Learning for Stock Market PredictionVariousDL in trading2020
LLMs for Financial NLP: Opportunities and ChallengesVariousFinance NLP survey2024
Fraud Detection using Machine LearningVariousML for fraud2020
Credit Scoring with Machine LearningLessmann et al.ML credit scoring2015
KYC/AML with NLPVariousNLP for compliance2023
Explainable AI for BankingVariousXAI in BFSI2024
Document AI for Financial ServicesVariousDocument processing in finance2024
LayoutLM: Pre-training of Text and Layout for Document AIXu et al.Layout-aware document model2020
LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingHuang et al.Improved document model2022
Donut: Document Understanding TransformerKim et al.OCR-free document understanding2022
PaddleOCRPaddlePaddleOpen-source OCR toolkit2020
TrOCR: Transformer-based Optical Character RecognitionLi et al.Transformer for OCR2023
Table Transformer (TATR)Smock et al.Table detection and structure2022
DocPrompting: Generating Code by Retrieving the DocsZhou et al.Document-augmented code gen2023
Nougat: Neural Optical Understanding for Academic DocumentsBlecher et al.Academic PDF to markdown2023
ColPali: Efficient Document Retrieval with Vision Language ModelsFaysse et al.Visual document retrieval2024
Marker: PDF to Markdown ConversionDatalabPDF extraction2024
End to End Learning for Self-Driving CarsBojarski et al.End-to-end driving2016
PointNet: Deep Learning on Point SetsQi et al.3D point cloud processing2017
PointNet++: Deep Hierarchical Feature Learning on Point SetsQi et al.Hierarchical point processing2017
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object DetectionZhou & TuzelVoxel-based 3D detection2018
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera ImagesLi et al.Camera-based BEV2022
UniAD: Planning-Oriented Autonomous DrivingHu et al.Unified autonomous driving2023
LLM-based Driving AgentsVariousLLMs for driving decisions2024
DriveLM: Driving with Graph Visual Question AnsweringSima et al.VLM for driving2024
DriveGPT4: Interpretable End-to-end Autonomous DrivingXu et al.GPT-4 for driving2024
Waymo Open DatasetSun et al.Large-scale driving dataset2020
TensorFlow: Large-Scale Machine Learning on Heterogeneous SystemsAbadi et al.TensorFlow framework2016
PyTorch: An Imperative Style, High-Performance Deep Learning LibraryPaszke et al.PyTorch framework2019
JAX: Composable Transformations of Python+NumPyBradbury et al.JAX framework2018
MLflow: A Platform for the ML LifecycleZaharia et al.ML experiment tracking2018
Hidden Technical Debt in ML SystemsSculley et al.ML technical debt2015
Ray: A Distributed Framework for Emerging AI ApplicationsMoritz et al.Distributed computing for AI2018
Kubernetes for ML Workloads (Kubeflow)VariousML on Kubernetes2019
Feature Stores for MLBaylor et al.Feature management2017
Continuous Delivery for ML (CD4ML)Sato et al.MLOps practices2019
NVIDIA GPU Architecture: From Pascal to HopperNVIDIAGPU evolution for AI2016-2024
WebArena: A Realistic Web Environment for Building Autonomous AgentsZhou et al.Web browsing benchmark2024
Mind2Web: Towards a Generalist Agent for the WebDeng et al.Web navigation agent2023
OS-World: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsXie et al.OS-level agent benchmark2024
Computer Use (Anthropic Claude)AnthropicGUI interaction capability2024
CogAgent: A Visual Language Model for GUI AgentsHong et al.Visual GUI agent2024
AppAgent: Multimodal Agents as Smartphone UsersZhang et al.Mobile app agents2024
ScreenAgent: A Vision Language Model-Driven Computer Control AgentNiu et al.Screen interaction agent2024
UFO: A UI-Focused Agent for Windows OS InteractionZhang et al.Windows UI agent2024
WebVoyager: Building an End-to-End Web AgentHe et al.Web browsing agent2024
Anthropic MCP for AgentsAnthropicAgent-tool protocol2024
NeRF: Representing Scenes as Neural Radiance FieldsMildenhall et al.Neural radiance fields2020
Instant NGPMüller et al.Fast neural graphics primitives2022
3D Gaussian Splatting for Real-Time Radiance Field RenderingKerbl et al.Gaussian splatting2023
DreamFusion: Text-to-3D using 2D DiffusionPoole et al.Text-to-3D generation2023
Magic3D: High-Resolution Text-to-3D Content CreationLin et al.High-res 3D generation2023
Zero-1-to-3: Zero-shot One Image to 3D ObjectLiu et al.Single image to 3D2023
Point-E: A System for Generating 3D Point Clouds from Complex PromptsNichol et al.Text-to-3D point cloud2022
Shap-E: Generating Conditional 3D Implicit FunctionsJun & Nichol3D implicit generation2023
Meshy / TripoSR / LRMVariousFast 3D reconstruction2024
4D Generation: Recent Progress and OpportunitiesVariousDynamic 3D generation2024
Intriguing Properties of Neural NetworksSzegedy et al.Adversarial examples discovered2014
Explaining and Harnessing Adversarial Examples (FGSM)Goodfellow, Shlens, SzegedyFast gradient sign method2015
Towards Deep Learning Models Resistant to Adversarial Attacks (PGD)Madry et al.PGD adversarial training2018
Certified Adversarial Robustness via Randomized SmoothingCohen, Rosenfeld, KolterCertified defenses2019
DeepFool: A Simple and Accurate Method to Fool Deep Neural NetworksMoosavi-Dezfooli et al.Minimal adversarial perturbation2016
Adversarial Examples in the Physical WorldKurakin et al.Physical adversarial examples2017
Robust and Accurate Object Detection via Adversarial LearningChen et al.Adversarial training for detection2021
Adversarial Attacks on LLMsVariousLLM robustness survey2024
Prompt Injection AttacksVariousLLM prompt injection2023-2024
Backdoor Attacks on LLMsVariousBackdoor vulnerabilities2024
Model-Agnostic Meta-Learning (MAML)Finn, Abbeel, LevineMAML2017
Matching Networks for One Shot LearningVinyals et al.Metric-based few-shot2016
Prototypical Networks for Few-Shot LearningSnell, Swersky, ZemelPrototype-based classification2017
Learning to Learn with Compound HD ModelsSantoro et al.Meta-learning with memory2016
Meta-Learning: A SurveyHospedales et al.Meta-learning survey2022
Reptile: A Scalable Metalearning AlgorithmNichol, Achiam, SchulmanSimple meta-learning2018
LEO: Latent Embedding OptimizationRusu et al.Latent space meta-learning2019
How to Train Your MAMLAntoniou et al.MAML training improvements2019
Few-Shot Learning via Saliency-Guided HallucinationZhang et al.Data augmentation for few-shot2019
P>M>F: Pre-training, Meta-training, Fine-tuningHu et al.Combining paradigms2022
Neural-Symbolic Computing: An Effective Methodology for Principled IntegrationGarcez et al.Neuro-symbolic survey2019
Neural Module NetworksAndreas et al.Compositional neural reasoning2016
End-to-End Differentiable Proving (NTP)Rocktäschel & RiedelDifferentiable reasoning2017
DeepProbLog: Neural Probabilistic Logic ProgrammingManhaeve et al.Neural + probabilistic logic2018
Scallop: A Language for Neurosymbolic ProgrammingLi et al.Neurosymbolic programming2023
LLMs and Symbolic ReasoningVariousLLM + symbolic integration2024
Program Synthesis with LLMsVariousLLMs for program synthesis2024
Logic-LM: Empowering LLMs with Symbolic SolversPan et al.LLM + logic solvers2023
Code-as-ReasoningVariousCode generation as reasoning2024
Binding Language Models in Symbolic LanguagesCheng et al.Binder framework2023
EU AI Act (technical analysis)EUAI regulation framework2024
Executive Order on AI Safety (US)White HouseUS AI safety policy2023
Governing AI: A Blueprint for the FutureVariousAI governance proposals2024
International AI Safety ReportAI Safety SummitGlobal AI safety assessment2024
Foundation Model Transparency IndexBommasani et al.Model transparency scoring2023
Mixture of Agents (MoA)Wang et al.LLM-based agent ensembles2024
Inference Scaling LawsSardana & FrankleTest-time compute scaling2024
LLM Operating SystemsPacker et al.; VariousLLMs as OS kernels2024
SWE-bench VerifiedOpenAIVerified agent coding2024
Anthropic's Responsible Scaling Policy (RSP) UpdateAnthropicUpdated safety policy2024
Q* / Process Reward ModelsVarious (speculated/inferred)Search + reward for reasoning2024
OpenAI o1 and o1-proOpenAIReasoning models2024
Claude 3.5 Sonnet and Computer UseAnthropicFrontier + computer use2024
Gemini 2.0 FlashGoogle DeepMindAgentic multimodal model2024
DeepSeek-R1DeepSeekOpen reasoning model2025
Grok-2xAIReal-time information model2024
Nemotron-4NVIDIASynthetic data generation2024
Apple Intelligence (technical overview)AppleOn-device LLM2024
Llama 3.1 405BMetaLargest open-weight model2024
Scaling Test-Time Compute (Survey)VariousComprehensive test-time survey2025
Byte Pair Encoding (BPE) for NMTSennrich et al.Subword tokenization2016
SentencePiece: Unsupervised Text TokenizerKudo & RichardsonLanguage-agnostic tokenizer2018
Tokenizer Choice MattersVariousImpact of tokenization on LLMs2024
MegaByte: Predicting Million-Byte SequencesYu et al.Byte-level modeling2023
BLT: Byte Latent TransformerMetaDynamic tokenization-free model2024
The Tokenizer Landscape for LLMsVariousTokenizer comparison2024
Tiktoken (OpenAI)OpenAIFast BPE tokenizer2023
Data Deduplication for LLM TrainingLee et al.Near-dedup for training data2022
Quality Filtering for LLM Training DataVariousData quality methods2024
Dolma: An Open Corpus of Trillion TokensSoldaini et al.Open training corpus2024
A Survey of Large Language ModelsZhao et al.Comprehensive LLM survey2023
Harnessing the Power of LLMs in PracticeYang et al.Practical LLM guide2023
A Survey on Hallucination in LLMsHuang et al.Hallucination survey2023
Siren's Song in the AI Ocean: A Survey on HallucinationZhang et al.Hallucination taxonomy2023
A Survey on Multimodal Large Language ModelsYin et al.MLLM survey2024
A Survey on Evaluation of LLMsChang et al.LLM evaluation survey2024
A Comprehensive Survey on Vector DatabaseHan et al.Vector DB survey2024
Retrieval-Augmented Generation for AI-Generated Content: A SurveyZhao et al.RAG survey2024
Tool Learning with Foundation ModelsQin et al.Tool use survey2024
LLM Agents: A Survey of ApplicationsVariousAgent applications survey2024
The Landscape of Emerging AI Agent ArchitecturesMasterman et al.Agent architecture survey2024
A Survey on Self-Evolution of LLMsTao et al.Self-improvement survey2024
From RAG to Rich: Retrieval-Augmented Generation SurveyVariousAdvanced RAG survey2024
AI Agents That MatterKapoor et al.Agent evaluation framework2024
Position: What Can LLMs Do for ME?VariousPractical applications2024
The State of AI Report 2024Benaich & HogarthAnnual AI overview2024
Stanford AI Index Report 2024StanfordAI progress metrics2024
Attention Mechanisms in Computer Vision: A SurveyGuo et al.Attention in vision survey2022
Vision Transformers: A SurveyKhan et al.ViT survey2022
Efficient Transformers: A SurveyTay et al.Efficient attention survey2022
The Unreasonable Effectiveness of DataHalevy, Norvig, PereiraData over algorithms2009
Batch Normalization (revisited for theory)Santurkar et al.Why BN works2018
Mixup: Beyond Empirical Risk MinimizationZhang et al.Data augmentation via interpolation2018
CutMix: Regularization Strategy to Train Strong ClassifiersYun et al.Cut-and-paste augmentation2019
RandAugment: Practical Automated Data AugmentationCubuk et al.Simplified augmentation2020
AutoAugment: Learning Augmentation Strategies from DataCubuk et al.Learned augmentation2019
Label Smoothing RevisitedMüller et al.Label smoothing analysis2019
When Does Label Smoothing Help?Müller et al.Label smoothing theory2019
Swish: A Self-Gated Activation FunctionRamachandran et al.Swish/SiLU activation2017
GELU: Gaussian Error Linear UnitsHendrycks & GimpelGELU activation2016
Squeeze-and-Excitation NetworksHu et al.Channel attention mechanism2018
CBAM: Convolutional Block Attention ModuleWoo et al.Spatial + channel attention2018
Non-local Neural NetworksWang et al.Self-attention for vision2018
Deformable Convolutional NetworksDai et al.Deformable convolutions2017
Dynamic Routing Between CapsulesSabour, Frosst, HintonCapsule networks2017
An Empirical Evaluation of Generic Convolutional and Recurrent Networks (TCN paper)Bai et al.TCN vs RNN comparison2018
Weight StandardizationQiao et al.Micro-batch normalization2019
Fixup Initialization: Residual Learning Without NormalizationZhang et al.Training without BN2019
Deep SetsZaheer et al.Permutation-invariant architectures2017
Attention Augmented Convolutional NetworksBello et al.Self-attention in CNNs2019
Do Vision Transformers See Like Convolutional Neural Networks?Raghu et al.ViT vs CNN representation analysis2021
What Do Vision Transformers Learn?Park & KimViT feature analysis2022
Scaling Vision with Sparse MoERiquelme et al.MoE for vision2021
Token Merging: Your ViT but FasterBolya et al.Token efficiency in ViTs2023
FNet: Mixing Tokens with Fourier TransformsLee-Thorp et al.FFT replaces attention2022
OpenAI o3 (analysis)OpenAIAdvanced reasoning model2024
Claude 4 / Opus 4 (model card)AnthropicFrontier model2025
Gemini 2.0 ProGoogle DeepMindAdvanced multimodal2025
Llama 4MetaNext-gen open model2025
Multi-Token PredictionGloeckle et al.Predicting multiple tokens2024
Native Multi-Modality vs Late FusionVariousArchitecture comparison2024
RLVR: Reinforcement Learning with Verifiable RewardsVariousVerifiable reward RL2025
Constitutional AI 2.0 ConceptsAnthropicAdvanced alignment research2024
Post-Training ScalingVariousScaling after pretraining2025
Weak-to-Strong GeneralizationBurns et al.Superalignment research2023
The Platonic Representation HypothesisHuh et al.Convergent representations2024
Representation EngineeringZou et al.Control via representations2023
LLMs as Optimizers (OPRO)Yang et al.Prompt optimization with LLMs2024
Can LLMs Generate Novel Research Ideas?Si et al.AI for scientific ideation2024
The AI Scientist: Towards Fully Automated Open-Ended Scientific DiscoveryLu et al.Autonomous research agent2024
Absolute Zero: Reinforced Self-Play Reasoning with Zero DataZhao et al.Self-play from nothing2025
Agentic Workflows and Compound AI SystemsZaharia et al.Compound AI systems vision2024
From Models to Compound AI SystemsMatei ZahariaSystems > models2024
LLM-based Multi-Agent Reinforcement Learning: Current and Future DirectionsChuanneng Sun, Songjun Huang, Dario PompiliMARL with LLMs2024
Self-Improving AI SystemsVariousRecursive self-improvement2024
Frontier AI Safety Commitments (Seoul Declaration)Various govtsInternational safety commitments2024
Situational AwarenessAschenbrennerAGI timeline analysis2024
What We Learned from a Year of Building with LLMsYan et al.Practical LLM lessons2024
Building Effective AgentsAnthropicAgent design patterns2024
The Next Wave: AI Systems that Reason, Act, and LearnVariousFuture of agentic AI2025