This course is part of the Master in High-performance Computing held by SISSA & ICTP (Trieste).
Main Info:
šĀ Course topics: Introduction to Large Language Model
š„Ā Audience: Master and PhDs students of MHPC Master
ā°Ā Duration: 20 hours
Course Abstract
The course will begin with a review of the basics of Deep Learning in PyTorch, followed by an advanced algorithm such as the Variational Auto Encoder (VAEs). Then the Transformer architecture and its Self-attention Mechanism will be introduced and coded. A simple, small but complete autoregressive generative language model such as GPT-2 will be built. This will allow us to understand several relevant aspects of more sophisticated pre-trained LLMs, such as GPT4, Mistral or Llama.
Lecture Material
- Slides: Link here (.pdf version here)
- Lecture 1: The Transformer Architecture
- Lecture 2: Coding GPT-2 step by step
- Lecture 3: Continued Pre-training & Fine-tune Llama on Unsloth
- Lecture 4: RAG and AI Agents with Langchain
- Lecture 5: LLMs Evaluation
- Lecture 2*: (optional): Pre-training GPT-2 on Unlabeled Data (link from here)