tue-ai-safety-course

KO | EN

gitlite — search

#python #react #go #golang #android #machine-learning #rust #deep-learning #nodejs #java #cli #kubernetes #openai #open-source

tue-ai-safety-course

(★ 131)

AI safety course at the University of Tübingen (Summer Semester 2026)

File Explorer

Download Latest Version (.zip)

- modal-logo.svg
- download_models.py
- README.md
- setup_env.sh
- smoke_test.py
- smoke_test_a100.slurm
- smoke_test_h100.slurm
- mock_exam.pdf
- - ai-control-safety-vs-usefulness.png
  - apollo-opus-scheming.webp
  - apollo-scheming-table.webp
  - emergent-misalignment-fig.png
  - epoch-compute-by-country.png
  - epoch-largest-dc-trend.png
  - epoch-llm-inference-price-trends.png
  - epoch-training-compute-trend.png
  - goodfellow-panda.png
  - international-ai-safety-report-cover.jpg
  - kirchenbauer-watermark.png
  - metr-time-horizons-log.png
  - metr-time-horizons.png
- - agentic-blackmail16.png
  - alignmentfaking-headline.png
  - baker-obfuscation.png
  - chen-faithfulness.png
  - cicero-deception.jpg
  - korbak-cot.png
  - mask-scale.png
  - needham-roc.png
  - probes-violin.png
  - sandbag-concept.png
  - sandbag-wmdp.png
  - scheurer-results.png
  - sleeper-persistence.png
  - testaware-method.png
- - algo-collusion-prices.png
  - early-stego.png
  - marisk-taxonomy.png
  - morris2-flow.png
  - secret-collusion.png
- - aar-hillclimb.jpg
  - aar-transfer.jpg
  - aicontrol-fig1.png
  - aicontrol-pareto.png
  - basharena-fig4.png
  - inferencebench-speedup.png
  - kernelbench-fig1.png
  - linuxarena-fig1.png
  - mlebench-fig1.png
  - ptb-breakdown.png
  - ptb-leaderboard.png
  - ptb-persistence.png
  - ptb-pipeline.png
  - researcharena-fig1.png
  - researcharena-fig5.png
  - researcharena-table1.png
  - rh-alignment-faking.png
  - rh-generalization.png
  - roon-tweet.png
  - rsi-code-per-person.png
  - rsi-research-judgment.png
  - w2s-analogy.png
  - w2s-results.png
- - ai2027-anim-poster.png
  - ai2027-anim.webm
  - ai2027-dashboard-poster.png
  - ai2027-dashboard.webm
  - ai2027-headline.png
  - ai2027-metr-extended.png
  - ai2027-takeoff.png
  - ai2040-figure.png
  - ai2040-figure2.png
  - aifm-chart.png
  - bioanchors-chart.png
  - chinchilla-fig1.png
  - chinchilla-isoflop.png
  - epoch-2030-constraints.png
  - epoch-ai-supercomputers-performance-share-by-country.png
  - epoch-chip-supply.png
  - epoch-compute-4-5x.png
  - epoch-power.png
  - epoch-revenue.png
  - europe2031-hero.png
  - futuresim-bss.png
  - goodheart-chart.png
  - hestness-powerlaw.png
  - kaplan-fig1.png
  - kiela-progress.webp
  - metr-domains.png
  - metr-linear.png
  - metr-log.png
  - metr-th11-since2023.png
  - metr-time-horizon-mythos.png
  - metr-time-horizon.png
  - moravec-mips.png
  - nws-calibration.png
  - o1-twin.png
  - owid-benchmark-saturation.png
  - owid-gdp-regions.png
  - owid-transistors.png
  - owid-world-gdp.png
  - sa-base-scaleup.png
  - sa-decade-or-bust.png
  - Screenshot from 2026-07-13 11-49-38.png
  - Screenshot from 2026-07-13 11-49-45.png
  - shot-ai2040.png
  - shot-aifm-daniel.png
  - shot-aifm-eli.png
  - shot-europe2031.png
  - shot-goodheart.png
  - shot-o1.png
- - arch-all.svg
  - arch-attn.svg
  - arch-embed.svg
  - arch-head.svg
  - arch-mlp.svg
  - arch-residual.svg
  - attention-example.png
  - attention-heads-clark-fig1.png
  - attention-heads-clark-fig5.png
  - attention-heads-clark.png
  - bertviz-head-view-static.png
  - bertviz-head-view.gif
  - bertviz-model-view.png
  - bertviz-model-view2.png
  - chess-gpt-probe.png
  - clark-head-4-10.png
  - clark-head-5-4.png
  - clark-head-8-10.png
  - claude-persona.jpeg
  - claude-shoggoth.jpeg
  - claude-training-data.jpeg
  - claude-void.jpeg
  - claude_stickers_need_to_be_cut.png
  - coerce-llms-attack.png
  - coerce-llms-repeater.png
  - compute-growth-4-5x.png
  - compute-trend-frontier-llm.png
  - compute-trend-frontier.png
  - continuous-batching.png
  - datacenter-growth.png
  - datecenters_vs_gdp.jpeg
  - decoder-wolfe.png
  - deepseek-r1-length.png
  - elhage_larger.png
  - friends-of-claude.jpg
  - geometry-of-truth.png
  - gpt4-calibration-post.png
  - gpt4-calibration.png
  - grpo-illustration.png
  - kv-cache-diagram.png
  - kv_cache_information_path_light_slide.svg
  - magicarp.png
  - mlp-knowledge-fig1.png
  - mlp-knowledge-rome.png
  - mlp-knowledge.png
  - moe-switch-transformer.png
  - parallelism-hybrid.png
  - parallelism-megatron-tp-pp.png
  - probe-accuracy-layers.png
  - probe-space-time-pythia.png
  - qwen3-pipeline.png
  - residual-stream-anthropic.png
  - residual-stream-attention-head.png
  - residual-stream-attn-ffn.png
  - rlhf-pipeline.png
  - rome-causal-trace.gif
  - rome-mlp.svg
  - rome-teaser.svg
  - simulators.png
  - smollm3-whiteprint.png
  - smollm3_0.png
  - smolm3_1.png
  - smolm3_2.png
  - smolm3_3.png
  - smolm3_4.png
  - smolm3_5.png
  - transformer-architecture.png
- - anthropic-cyber-lifecycle.png
  - attacker-moves-second-fig1.png
  - bon-pipeline.png
  - bon-power-law.png
  - claudini-evolution.png
  - claudini-fig1.png
  - constitutional-classifier-diagram.jpg
  - constitutional-classifier-results.png
  - crescendo-arc.png
  - crescendo-asr.png
  - crescendo-example.png
  - cyber-capabilities-iasr.png
  - deliberative-alignment-overview.jpg
  - deliberative-alignment-pipeline.png
  - deliberative-alignment-results.png
  - feedback-qr.png
  - gcg-fig1.png
  - gcg-transfer.png
  - greshake-demo.png
  - greshake-injection.png
  - interp-gradients.png
  - lp-norm-vis.png
  - many-shot-illustration.png
  - many-shot-power-law.png
  - many-shot-scaling.png
  - mythos-art-pi.png
  - mythos-uk-aisi-cyber-eval.png
  - pair-pipeline.png
  - pair-results.png
  - panda-gibbon.png
  - random-search-curves.png
  - robustness-accuracy-tradeoff.png
  - swiss-cheese-iasr.png
  - uap-fig1.png
- - crescendo1.png
  - crescendo2.png
  - feedback-qr.png
  - image1.png
  - image10.png
  - image11.png
  - image12.png
  - image13.png
  - image14.png
  - image15.png
  - image16.png
  - image17.png
  - image18.png
  - image19.png
  - image2.png
  - image20.png
  - image21.png
  - image22.png
  - image23.png
  - image24.png
  - image25.png
  - image26.png
  - image27.png
  - image28.png
  - image3.png
  - image4.png
  - image5.png
  - image6.png
  - image7-our-codebase.png
  - image7.png
  - image8.png
  - image9.png
  - lecture-schedule.png
  - rl-as-a-hammer-table.png
- - binoculars-ppl-xppl-scatter.png
  - c2pa-logo.png
  - cipher-attack-emoji.png
  - cipher-attack-letterswap.png
  - copy-paste-attack.png
  - easymark-whitemark.png
  - kgw-quality-detection-pareto.png
  - kgw2-paraphrase-zscore.png
  - kgw2-seeding-comparison.png
  - liang-delve-frequency.png
  - pangram-mirrors.png
  - Safety Security Talk (5) cropped.png
  - Safety Security Talk (6) cropped.png
  - watermark-stealing-cost.png
  - wm-demo-data.json
  - worldcoin-orb.png
- - c4-top-domains.png
  - c4-top-tlds.png
  - carlini-gpt2-extraction.png
  - carlini2022-scaling-3panel.png
  - chatgpt-poem-extraction.png
  - commoncrawl-logo.svg
  - consent-in-crisis.png
  - dated-data-cheng.png
  - goldfish-harry-potter.png
  - haviv-thin-thick.png
  - kandpal-longtail.png
  - lecture-schedule.png
  - lermen-deanonymization.png
  - poisoning-split-view.png
  - shah-cutoff-frontier.png
  - sleeper-agents.png
  - sleeper-fig2.png
  - somepalli-couches.png
  - souly-poison-constancy.png
  - staab-vechev-pipeline.png
  - wmdp-dataset.svg
  - zhao-set-the-clock-cropped.png
  - zhao-set-the-clock.png
- - gao_a_rl_rmsweep.png
  - image1.png
  - image14.png
  - image15.png
  - image18.png
  - image2.png
  - image20.png
  - image21.png
  - image27.png
  - image5.png
  - image9.png
  - qr_feedback.png
- - codeact-fig1-cropped.png
  - cot-fig1.png
  - cot-fig3-examples.png
  - cot-fig4-scaling.png
  - effective-augmented-llm.png
  - mas-architecture.png
  - osworld-demo.png
  - react-alfworld.png
  - react-hotpotqa.png
  - skills-anatomy.jpg
  - swebench-fig1.png
  - toolformer-fig1.png
  - toolformer-fig2.png
  - toolllm-dfsdt.png
  - toolllm-fig1.png
  - tools-asana-accuracy.png
  - vwa-fig1.png
  - webarena-fig1.png
  - webgpt-demo.png
  - webvoyager-fig1.png
- - agentdojo-defense.png
  - agentdojo-fig1.png
  - airgap-fig1.png
  - camel-dualllm.png
  - camel-willison.jpg
  - deepmind-ci-fig1.png
  - echoleak-flow.png
  - firewalls-fig2.png
  - insthier-fig2.png
  - ipi-fig1.png
  - llmail-fig1.png
  - skillinject-asr.png
  - tfa-gh-issue.svg
- gen_pdf.py
- lecture-1-intro.html
- lecture-1-intro.pdf
- lecture-10-scheming-deception.html
- lecture-10-scheming-deception.pdf
- lecture-11-multi-agent-safety.html
- lecture-11-multi-agent-safety.pdf
- lecture-12-ai-rnd-automation.html
- lecture-12-ai-rnd-automation.pdf
- lecture-13-forecasting.html
- lecture-13-forecasting.pdf
- lecture-2-llm-background.html
- lecture-2-llm-background.pdf
- lecture-3-adv-ml.html
- lecture-3-adv-ml.pdf
- lecture-4-open-weight-safety.html
- lecture-4-open-weight-safety.pdf
- lecture-5-transparency.html
- lecture-5-transparency.pdf
- lecture-6-data.html
- lecture-6-data.pdf
- lecture-7-alignment-tools.html
- lecture-7-alignment-tools.pdf
- lecture-8-agents.html
- lecture-8-agents.pdf
- lecture-9-agent-security.html
- lecture-9-agent-security.pdf
- Final_Project_Guidelines.pdf
- Mini-Project 1 - Base vs. post-trained LLMs and LLM jailbreaking.pdf
- Mini-Project 2 - Emergent misalignment and detection of LLM-generated text.pdf
- Mini-Project 3 - Understanding Data and Model Steering .pdf
.gitignore
CLAUDE.md
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AI Safety Course — University of Tübingen - Summer Semester 2026

**Lecturers**: [Maksym Andriushchenko](https://www.andriushchenko.me), [Jonas Geiping](https://jonasgeiping.github.io/), and [Sahar Abdelnabi](https://s-abdelnabi.github.io/).

**Teaching assistants**: Hardik Bhatnagar, Katharina Deckenbach, Shashwat Goel, Johannes Koch, Lena Libon, Luca Morlok, Alexander Panfilov, Ben Rank, Jeanne Salle, David Schmotz, Jehyeok (Tommy) Yeon, Yuchen Zhang.

**HPC support**: Michael Tiemann.

**Course**: A master's-level course on the safety, security, and alignment of modern AI systems, with a focus on large language models and LLM agents. The course is technical and is primarily aimed at computer science students: lectures and mini-projects assume solid programming experience and familiarity with machine learning fundamentals. Students from non-technical backgrounds (e.g. humanities) with little prior coding experience are likely to find the material challenging.

**Compute**: [Modal](https://modal.com) is supporting the course by providing every student with $250 of cloud credits. More details below.

---

## Course Information

|                  |                                                     |
| ---------------- | --------------------------------------------------- |
| **Credits**      | 6 ECTS                                              |
| **Exercise**     | Mondays, 10:15 – 11:45                              |
| **Lecture**      | Mondays, 12:15 – 13:45                              |
| **Location**     | Hörsaal A1 (A-206), Cyber Valley Campus, Maria-von-Linden-Str. 1 |
| **Start date**   | 13 April 2026                                       |
| **Final exam**   | 27 July 2026, 16:00 – 18:00, Hörsaal N07 (Hörsaalzentrum Morgenstelle) |
| **Re-exam**   | 6 October 2026, 17:00 – 19:00, Hörsaal A1 (A-206), Cyber Valley Campus, Maria-von-Linden-Str. 1 |

The exercise slot alternates between **office hours** and **mini-project presentations**. Mini-project presentations give an opportunity to receive direct feedback on your work and earn bonus points that count towards the exam admission.

The primary communication channel for the course is **Slack**, which you can join via [https://join.slack.com/t/tbingenaisafetycourse/signup](https://join.slack.com/t/tbingenaisafetycourse/signup) using your @student.uni-tuebingen.de email.

---

## Grading

The course is assessed along three components:

### Mini-projects (50% points needed for exam admission)
Four mini-projects (2 weeks each). Selected students present their solutions during the exercise slot. For mini-projects, we recommend to avoid using LLM agents and instead use your own coding skills. In this way, you will get a better understanding of the material and better prepare for the final exam.

Potential topics (tentative, subject to change):
- Implementing a transformer from scratch, understanding the behavior of base vs. instruct models, linear probes and activation steering.
- Jailbreaking open-weight LLMs, the Lakera challenge, attacks on VLM.
- Alignment tools: SFT, DPO, RLHF, etc.
- Deception, scheming, evaluation awareness; developing a project proposal for the final project.

### Final project — 50% of the final grade
A one-month open-ended research project carried out in **teams of 3 students**. An example project can be to extend a recent influential paper in the field (e.g. [emergent misalignment](https://arxiv.org/abs/2502.17424), [AI control](https://arxiv.org/abs/2312.06942), etc.). The project concludes with a final presentation scheduled on **20 July 2026**. LLM agents are allowed as coding assistants.

### Final exam — 50% of the final grade
A written exam with free-form questions covering the lecture material. For examples of questions, see `exam/mock_exam.pdf`. Note that the final exam will take 2 hours, cover 11 lectures (i.e., all, excluding the first two), and will have 3 questions per lecture (i.e., 33 questions total).

---

## Schedule

Lecture recordings can be found [here](https://drive.google.com/drive/folders/1PwjAmmEJZqlY068SjrjYniHP6nEWnkV9).

| # | Date   | Lecturer  | Topic                                                                                      |
| - | ------ | --------- | ------------------------------------------------------------------------------------------ |
| 1 | 13.04  | Maksym    | [**Course overview.**](lecture-slides/lecture-1-intro.html) Recent progress in LLMs and agents; AI risks; alignment, HHH, Constitutional AI |
| 2 | 20.04  | Jonas     | [**LLM background.**](lecture-slides/lecture-2-llm-background.html) Transformers, pre-training, post-training, using models. |
| 3 | 27.04  | Maksym    | [**Lessons from adversarial ML.**](lecture-slides/lecture-3-adv-ml.html) Jailbreaks for chatbots and agents (prefilling, GCG, PAIR, Crescendo, decomposition); backdoors |
| 4 | 04.05  | Maksym    | [**Open-weight safety.**](lecture-slides/lecture-4-open-weight-safety.html) Model stealing & distillation attacks, fine-tuning attacks, emergent misalignment &nbsp;*— Mini-project 1 due this week* |
| 5 | 11.05  | Jonas     | [**Transparency.**](lecture-slides/lecture-5-transparency.html) Detection of LLM-generated content and watermarking                      |
| 6 | 18.05  | Jonas     | [**Problems with 'Big Data'.**](lecture-slides/lecture-6-data.html) Privacy, Memorization, and Other Data Integrity Issues with LLMs &nbsp;*— Mini-project 2 due this week*               |
| 7 | 01.06  | Sahar     | [**Alignment Tools.**](lecture-slides/lecture-7-alignment-tools.html) Pre-training filtering, RLHF, DPO, Constitutional AI, LLMs as judges, Model Spec with rubric RL, steering vectors |
| 8 | 08.06  | Sahar     | [**LLM agents.**](https://github.com/aisa-group/tue-ai-safety-course/blob/main/lecture-slides/lecture-8-agents.html) Reasoning and planning, deep research, coding and browser/computer-use agents, MCP, agent skills &nbsp;*— Mini-project 3 due this week* |
| 9 | 15.06  | Sahar     | [**Agent security.**](lecture-slides/lecture-9-agent-security.html) Prompt injections and contextual privacy                               |
| 10 | 22.06 | Sahar     | [**Scheming & deception.**](lecture-slides/lecture-10-scheming-deception.html) Deception, scheming, sandbagging, evaluation awareness, and mitigations and their limitations &nbsp;*— Final project starts* |
| 11 | 29.06 | Sahar     | [**Multi-agent safety.**](lecture-slides/lecture-11-multi-agent-safety.html) Information propagation, collusion, steganography                  |
| 12 | 06.07 | Maksym    | [**Automating AI R&D.**](lecture-slides/lecture-12-ai-rnd-automation.html) Recursive self-improvement, benchmarks (MLE-bench, PostTrainBench, KernelBench, InferenceBench), AI control, and scalable oversight |
| 13 | 13.07 | Jonas     | **Forecasting AI progress.** Scaling laws, the METR time-horizon plot, AI 2027             |
| 14 | 20.07 | —         | **Final presentations** &nbsp;*— Final project due*                                        |

---

## Viewing Lecture Slides

All lecture slides are standalone HTML files (in the `lecture-slides/` directory) built with [reveal.js](https://revealjs.com/). No PowerPoint, Keynote, or any other software is needed — just a web browser.

```bash
git clone https://github.com/aisa-group/tue-ai-safety-course.git
cd tue-ai-safety-course
open lecture-slides/lecture-1-intro.html   # macOS
# or just double-click the HTML file in your file manager
```

---

# Compute

## Option 1: Google Colab

Free [Google Colab](https://colab.research.google.com/) is sufficient for projects that only rely on inference of small LLMs (up to 20B parameters). To run models on a GPU, don’t forget to go to Runtime → Change runtime type, then under "Hardware accelerator" select T4 GPU.

## Option 2: HPC Compute Resources (KISSKI)

Students have access to GPU compute resources via the **KISSKI** cluster at GWDG (Göttingen) for mini-projects and the final project.

| | |
|---|---|
| **Project ID** | `kisski-asc2026` |
| **System** | KISSKI (GWDG, Göttingen) |
| **Login node** | `glogin-gpu.hpc.gwdg.de` |
| **Project path** | `/scratch/extern/kisski/kisski-asc2026` |
| **GPU compute hours** | 25,000 |
| **Storage** | 1 TB |
| **Project duration** | 2026-04-01 – 2026-07-31 |

**Documentation**
- [KISSKI Documentation](https://kisski.gwdg.de/en/leistungen/documentation/) — KISSKI-specific guides and application process
- [GWDG HPC Documentation](https://docs.hpc.gwdg.de/) — full documentation portal (getting started, SSH, job submission, etc.)
- [Cluster Overview](https://docs.hpc.gwdg.de/start_here/cluster_overview/index.html) — hardware specs and partition overview
- [Project Portal Guide](https://docs.hpc.gwdg.de/getting_started/project_portal/index.html) — how to join a project and manage allocations

**Getting access**: You need to join the project `kisski-asc2026` via the [GWDG HPC Project Portal](https://hpcproject.gwdg.de/projects/9348c29e-2700-4a58-b2aa-1ca12e016f74/) (AcademicCloud login required). Once added, you can SSH into the login node using your GWDG account.

**HPC support**: For questions about cluster access and usage, contact **Michael Tiemann** ([michael.tiemann@uni-tuebingen.de](mailto:michael.tiemann@uni-tuebingen.de)).

**Chat AI & LLM API (KISSKI)**
KISSKI also provides free access to LLMs via a chat interface and a programmatic API, available to all students and researchers at eligible universities (including Tübingen).
- *Chat interface*: [chat-ai.academiccloud.de](https://chat-ai.academiccloud.de/) — browser-based access to GPT-3.5/4 and various open-source models; conversations are private and not stored after the session.
- *API access*: OpenAI-compatible API at `https://chat-ai.academiccloud.de/v1`. Request an API key via the [KISSKI service booking form](https://services.kisski.de/services/?service=2-02-llm-service.json) (requires a free Academic ID).
- *Default rate limits*: 10 req/min · 200 req/hour · 400 req/day · 3,000 req/month (increasable on request).
- *Full documentation*: [docs.hpc.gwdg.de/services/chat-ai](https://docs.hpc.gwdg.de/services/chat-ai/index.html)

---

### Option 3: Modal Cloud Credits

[Modal](https://modal.com) is supporting the course by providing **every student with $250 of cloud credits**. Modal lets you run Python on cloud GPUs (training, inference, batch jobs) without managing infrastructure. It's a useful complement to the KISSKI cluster for the mini-projects and the final project. To access these credits, you should create your own Modal account and then fill out the form which will be distributed via Slack.

---

## Resources

**Reading & further study**
- [*International AI Safety Report 2026*](https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026) (Yoshua Bengio et al.)
- [*Introduction to AI Safety, Ethics, and Society*](https://arxiv.org/abs/2411.01042) (Dan Hendrycks)
- [AlignmentForum](https://www.alignmentforum.org/) — community discussion of alignment research
- [*AI 2027*](https://ai-2027.com/) — a speculative forecasting scenario; conceptually thought-provoking, though the concrete predictions should be read with a healthy dose of skepticism.

**Related courses at other universities**
- **Harvard** — [AI Safety Course](https://boazbk.github.io/mltheoryseminar/)
- **Princeton** — [COS 598 *AI Safety*](https://sites.google.com/view/cos598aisafety/)
- **UC Berkeley** — [CS 294/194-267 *Understanding Large Language Models: Foundations and Safety*](https://rdi.berkeley.edu/understanding_llms/s24)
- **Stanford** — [CS 120 *Introduction to AI Safety*](https://web.stanford.edu/class/cs120/) and [CS 521 *Seminar on AI Safety*](https://cs521.stanford.edu/)

---

tue-ai-safety-course

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

tue-ai-safety-course

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README