Representation 3 - Prompting (9/26/2023)
Prompting + Sequence-to-sequence Pre-training
- LLMs
- Prompting Methods
- Instruction Tuning
- Recommended Reading: Pre-Train, Prompt, and Predict. A Systematic Survey of Prompting Methods in NLP (Liu et al. 2021)
- Reference: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al. 2019)
- Reference: GPT-2: Language Models are Unsupervised Multitask Learners (Radford et al. 2019)
- Recommended Reading: GPT-3: Language Models are Few-Shot Learners (Brown et al. 2020)
- Reference: Demystifying Prompts in Language Models via Perplexity Estimation (Gonen et al. 2022)
- Reference: AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (Shin et al. 2020)
- Reference: Prefix-Tuning: Optimizing Continuous Prompts for Generation (Li and Liang 2021)
- Reference: The Power of Scale for Parameter-Efficient Prompt Tuning (Lester et al. 2021)
- Reference: Calibrate Before Use: Improving Few-Shot Performance of Language Models (Zhao et al. 2021)
- Reference: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Min et al. 2022)
- Reference: HELM: Holistic Evaluation of Language Models (Liang et al. 2022)
- Reference: Chain-of-Thought Prompting (Wei et al. 2022)
- Reference: Large Language Models are Zero-Shot Reasoners (Kojima et al. 2022)
- Reference: Self-Consistency Improves Chain-of-Thought Reasoning in Language Models (Wang et al. 2023)
- Reference: Adapting Language Models for Zero-shot Learning by Meta-Tuning on Dataset and Prompt Collections (Sanh et al. 2021)
- Recommended Reading: T0: Multitask Prompted Training Enables Zero-Shot Task Generalization (Sanh et al. 2021)
- Reference: FLAN: Finetuned Language Models are Zero-Shot Learners (Wei et al. 2021)
- Reference: Scaling Instruction-Finetuned Language Models (Chung et al. 2022)
- Reference: Which Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (Wang et al. 2022)
Slides: Prompting Slides