CC6205

KO | EN

gitlite — search

#javascript #python #hacktoberfest #typescript #android #deep-learning #nodejs #macos #windows #kubernetes #vue #security #pytorch #openai

(★ 269)

Natural Language Processing

File Explorer

Download Latest Version (.zip)

- - 2rep.png
  - acceptor.png
  - activations.png
  - anger.pdf
  - ant.pdf
  - atten_formula.png
  - attention-alignment.png
  - attention_concept.png
  - attentionformula.png
  - backwardPass.png
  - bagOfwordsClassification.pdf
  - bert.png
  - BERTeffect.png
  - BERTFineTuning.png
  - BERTGLUE.png
  - biRNN.png
  - biRNNstar.png
  - bow.png
  - BSPE.png
  - CBOW.png
  - chainoftought.png
  - chatgpt.png
  - chomsky.png
  - cluster.png
  - cnn-twitter.png
  - CNN.png
  - comgraphexample.png
  - compGraph.png
  - confmatrix.png
  - confmatrixmulti.png
  - cos.png
  - CRF1.pdf
  - CRF2.pdf
  - CRF3.png
  - cv.png
  - deepEmoji1.png
  - deepEmoji2.png
  - deepEmoji3.png
  - denseonehot.png
  - devsets.png
  - disg.pdf
  - distributionalSocher.png
  - doc2vec.png
  - documents.jpg
  - drop-out-in-neural-networks.jpg
  - elman.pdf
  - elmanRNN.png
  - elmo.png
  - elmo2.png
  - elmo_results.png
  - emb_matrix.pdf
  - emb_matrix2.png
  - embeddings_enc.png
  - encdecattention.png
  - encoder_with_tensors.png
  - encoder_with_tensors_2.png
  - escudocolor.png
  - fear.pdf
  - federalist.png
  - fewshotresults.png
  - forwardPass.png
  - gatedEx.png
  - generator.png
  - gensim.png
  - goldbergNLP.jpg
  - gpt3.png
  - gpt4.png
  - gradientdescent.png
  - greedysearch.png
  - GRU.png
  - hamilton.png
  - hardtanh.png
  - instructionfinetuning.png
  - invFile.png
  - joy.pdf
  - kmeans.png
  - lambda.png
  - lambdaresults.png
  - langbigrams.png
  - langbigramstest.png
  - linear-relationships.png
  - llmscale.png
  - llmscale_trans.png
  - llmtimeline.png
  - lmdiagram.png
  - LMneuronal.png
  - lmnotuserassistant.png
  - lmsampling.png
  - logo-fcfm.jpg
  - logo.pdf
  - lossfunctions.png
  - lstm_nlm.png
  - LSTMchain.png
  - LSTMform.png
  - madison.png
  - medarticle.png
  - minibatch-SGD.png
  - MLM.png
  - MLvsDL.png
  - morphology.png
  - mt.png
  - multi_head_attention.png
  - multiple_paths_chain_rule.png
  - multiple_paths_chain_rule_general.png
  - naive_example.png
  - negWordsTranfer.pdf
  - NER.png
  - neural_net.pdf
  - neural_net_mat_mul.pdf
  - nmt_progress.png
  - NN-example.png
  - NSP.png
  - onehot.png
  - Online-SGD.png
  - parseTree.png
  - parseTree1.png
  - porter.png
  - POStags.png
  - posWordsTranfer.pdf
  - prompting.png
  - recTensor1.png
  - recTensor2.png
  - relu.png
  - RLHF.png
  - RNN-recursion.png
  - RNN-unrolled.png
  - RNN.png
  - rnnLM.png
  - sadness.pdf
  - searchengine.png
  - self-attention-matrix-calculation-2.png
  - self-attention-matrix-calculation.png
  - self-attention-output.png
  - self-attention_softmax.png
  - selfattention.png
  - sent.png
  - seq2se2train.png
  - seqseq.png
  - sgd.png
  - shannon.png
  - sigmoid.png
  - sigmoid2.png
  - simple_chain_rule.png
  - skip-gram.png
  - skip_gram_net_arch.png
  - spam.png
  - SSPE.png
  - SSPERes.png
  - stackedRNN.png
  - surprise.pdf
  - SVM.png
  - tanh.png
  - the_transformer_3.png
  - The_transformer_encoder_decoder_stack.png
  - The_transformer_encoders_decoders.png
  - transducer.png
  - transformer.png
  - transformer_attention_heads_qkv.png
  - transformer_attention_heads_weight_matrix_o.png
  - transformer_attention_heads_z.png
  - transformer_decoder.png
  - transformer_decoder_output_softmax.png
  - Transformer_encoder.png
  - transformer_multi-headed_self-attention-recap.png
  - transformer_positional_encoding_example.png
  - transformer_positional_encoding_vectors.png
  - transformer_resideual_layer_norm.png
  - transformer_resideual_layer_norm_2.png
  - transformer_self-attention_visualization.png
  - transformer_self_attention_score.png
  - transformer_self_attention_vectors.png
  - transformerdecoder.png
  - transformerencoder.png
  - transformerresults.png
  - trust.pdf
  - twitter.png
  - types_of_attention.png
  - u.png
  - ulmfit1.png
  - ulmfit2.png
  - ulmfit3.png
  - ulmfit4.png
  - validation.png
  - VDCNN.png
  - viterbi1.pdf
  - viterbi2.pdf
  - wordnet.png
  - xor.png
  - xor2.png
  - zeroonefew.png
  - zipf1.png
- - 1_1.png
  - 1_2.png
  - 1_3.png
  - band1.png
  - boat.png
  - combo.jpg
  - conv.jpg
  - data.jpg
  - esahubble.png
  - fig_PCA.png
  - ha-gray-conv-crp.jpg
  - head1.png
  - head2.png
  - head3.png
  - Histogram_train2.png
  - lineas.png
  - matri6-01.png
  - matrix2-01.png
  - nine.jpg
  - nphoton.jpg
  - portada.png
  - portada2.png
  - portada3.png
  - portada4.png
  - portada5.png
  - Screenshot.png
  - Small-train2.png
  - Small-train3-matrix.png
  - Small-train3.png
  - Small-train6.png
  - small_ex.png
  - uno.jpg
- apunte.pdf
- apunte.ptc
- apunte.tex
- bio.bib
- cap_cnn.tex
- cap_embeddings.tex
- cap_etisec.tex
- cap_intro.tex
- cap_ir.tex
- cap_lineales.tex
- cap_llm.tex
- cap_nb.tex
- cap_plm.tex
- cap_redes.tex
- cap_rnn.tex
- cap_sec.tex
- cap_trans.tex
- copy.pdf
- copy.tex
- structure.tex
- StyleInd.ist
- star_wars_chatbot.json
- .$llms.bkp
- .$llms2.drawio.bkp
- .$llms3.drawio.bkp
- .$llms4.drawio.bkp
- .~lock.confusion.ods#
- confusion.ods
- CRF.xopp
- CRF.xopp~
- CRF1.xopp
- CRF1.xopp~
- CRF2.xopp
- demostracionPerplexity.jpg
- dynamic_programing.py
- elman.xopp
- elmanRNN.pdf
- emb_matrix.xopp
- llms
- llms.pdf
- llms2.drawio
- llms2.pdf
- llms3.drawio
- llms3.pdf
- llms4.drawio
- llms4.pdf
- neural_net.xopp
- neural_net.xopp~
- neural_net_mat_mul.xopp
- notesHMM.pdf
- notesHMM.xopp
- stemming.py
- - 2rep.png
  - acceptor.png
  - activations.png
  - agentexample.png
  - anger.pdf
  - ant.pdf
  - assistantpattern.pdf
  - atten_formula.png
  - attention-alignment.png
  - attention_concept.png
  - attentionformula.png
  - autonomousagent.png
  - backwardPass.png
  - bagOfwordsClassification.pdf
  - bert.png
  - BERTeffect.png
  - BERTFineTuning.png
  - BERTGLUE.png
  - biRNN.png
  - biRNNstar.png
  - bow.png
  - BSPE.png
  - CBOW.png
  - chainoftought.png
  - chatgpt.png
  - chomsky.png
  - cluster.png
  - cnn-twitter.png
  - CNN.png
  - comgraphexample.png
  - compGraph.png
  - confmatrix.png
  - confmatrixmulti.png
  - cos.png
  - CRF1.pdf
  - CRF2.pdf
  - CRF3.png
  - cv.png
  - deepEmoji1.png
  - deepEmoji2.png
  - deepEmoji3.png
  - denseonehot.png
  - devsets.png
  - disg.pdf
  - distributionalSocher.png
  - doc2vec.png
  - documents.jpg
  - drop-out-in-neural-networks.jpg
  - elman.pdf
  - elmanRNN.png
  - elmo.png
  - elmo2.png
  - elmo_results.png
  - emb_matrix.pdf
  - embeddings_enc.png
  - encdecattention.png
  - encoder_with_tensors.png
  - encoder_with_tensors_2.png
  - escudocolor.png
  - fear.pdf
  - federalist.png
  - fewshotresults.png
  - forwardPass.png
  - gatedEx.png
  - generator.png
  - gensim.png
  - goldbergNLP.jpg
  - gpt3.png
  - gpt4.png
  - gptllama.png
  - gradientdescent.png
  - greedysearch.png
  - GRU.png
  - hamilton.png
  - hardtanh.png
  - instructionfinetuning.png
  - invFile.png
  - joy.pdf
  - kmeans.png
  - lambda.png
  - lambdaresults.png
  - langbigrams.png
  - langbigramstest.png
  - langchain3.png
  - Large-Language-Models.jpg
  - linear-relationships.png
  - LLMbenchmarks.png
  - llmfinetuning.pdf
  - llmsapp.pdf
  - llmscale.png
  - llmscale_trans.png
  - llmtimeline.png
  - lmdiagram.png
  - lmnotuserassistant.png
  - lmsampling.png
  - logo-fcfm.jpg
  - logo.pdf
  - lora.png
  - lossfunctions.png
  - lstm_nlm.png
  - LSTMchain.png
  - LSTMform.png
  - madison.png
  - medarticle.png
  - minibatch-SGD.png
  - MLM.png
  - MLvsDL.png
  - MMLU.png
  - morphology.png
  - mt.png
  - mtbench.png
  - mtbenchex1.png
  - mtbenchex2.png
  - multi_head_attention.png
  - multiple_paths_chain_rule.png
  - multiple_paths_chain_rule_general.png
  - naive_example.png
  - negWordsTranfer.pdf
  - NER.png
  - neural_net.pdf
  - neural_net_mat_mul.pdf
  - nmt_progress.png
  - NN-example.png
  - NSP.png
  - onehot.png
  - Online-SGD.png
  - openLLMbenchmark.png
  - parseTree.png
  - parseTree1.png
  - porter.png
  - POStags.png
  - posWordsTranfer.pdf
  - prompting.png
  - qlora.png
  - quant-freeze.png
  - react.png
  - recTensor1.png
  - recTensor2.png
  - relu.png
  - retrievalaugmented.pdf
  - RLHF.png
  - RNN-recursion.png
  - RNN-unrolled.png
  - RNN.png
  - sadness.pdf
  - searchengine.png
  - self-attention-matrix-calculation-2.png
  - self-attention-matrix-calculation.png
  - self-attention-output.png
  - self-attention_softmax.png
  - sent.png
  - seq2se2train.png
  - seqseq.png
  - sgd.png
  - shannon.png
  - sigmoid.png
  - sigmoid2.png
  - simple_chain_rule.png
  - skip-gram.png
  - skip_gram_net_arch.png
  - spam.png
  - SSPE.png
  - SSPERes.png
  - stackedRNN.png
  - superglue.png
  - surprise.pdf
  - SVM.png
  - tanh.png
  - the_transformer_3.png
  - The_transformer_encoder_decoder_stack.png
  - The_transformer_encoders_decoders.png
  - transducer.png
  - transformer.png
  - transformer_attention_heads_qkv.png
  - transformer_attention_heads_weight_matrix_o.png
  - transformer_attention_heads_z.png
  - transformer_decoder.png
  - transformer_decoder_output_softmax.png
  - Transformer_encoder.png
  - transformer_multi-headed_self-attention-recap.png
  - transformer_positional_encoding_example.png
  - transformer_positional_encoding_vectors.png
  - transformer_resideual_layer_norm.png
  - transformer_resideual_layer_norm_2.png
  - transformer_self-attention_visualization.png
  - transformer_self_attention_score.png
  - transformer_self_attention_vectors.png
  - transformerdecoder.png
  - transformerencoder.png
  - transformerresults.png
  - trust.pdf
  - twitter.png
  - types_of_attention.png
  - u.png
  - ulmfit1.png
  - ulmfit2.png
  - ulmfit3.png
  - ulmfit4.png
  - validation.png
  - VDCNN.png
  - vectordatabase.png
  - viterbi1.pdf
  - viterbi2.pdf
  - wordnet.png
  - xor.png
  - xor2.png
  - zeroonefew.png
  - zipf1.png
- bio.bib
- lmslides.pdf
- NLP-CNN.pdf
- NLP-CNN.tex
- NLP-CRF.pdf
- NLP-CRF.tex
- NLP-HMM.pdf
- NLP-HMM.tex
- NLP-introduction.pdf
- NLP-introduction.tex
- NLP-IR.pdf
- NLP-IR.tex
- NLP-linear.pdf
- NLP-linear.tex
- NLP-LLM.pdf
- NLP-LLM.tex
- NLP-LLMpatterns.pdf
- NLP-LLMpatterns.tex
- NLP-NB.pdf
- NLP-NB.tex
- NLP-neural.pdf
- NLP-neural.tex
- NLP-PLM.pdf
- NLP-PLM.tex
- NLP-recursive.pdf
- NLP-recursive.tex
- NLP-RNN.pdf
- NLP-RNN.tex
- NLP-seq2seq.pdf
- NLP-seq2seq.tex
- NLP-transformer.pdf
- NLP-transformer.tex
- NLP-wordvectors.pdf
- NLP-wordvectors.tex
- tagging.pdf
- - 2rep.png
  - acceptor.png
  - activations.png
  - agentexample.png
  - anger.pdf
  - ant.pdf
  - assistantpattern.pdf
  - atten_formula.png
  - attention-alignment.png
  - attention_concept.png
  - attentionformula.png
  - autonomousagent.png
  - backwardPass.png
  - bagOfwordsClassification.pdf
  - bert.png
  - BERTeffect.png
  - BERTFineTuning.png
  - BERTGLUE.png
  - bias_variance.png
  - biRNN.png
  - biRNNstar.png
  - bow.png
  - BSPE.png
  - CBOW.png
  - chainoftought.png
  - chatgpt.png
  - chomsky.png
  - cluster.png
  - cnn-twitter.png
  - CNN.png
  - comgraphexample.png
  - compGraph.png
  - confmatrix.png
  - confmatrixmulti.png
  - cos.png
  - CRF1.pdf
  - CRF2.pdf
  - CRF3.png
  - cv.png
  - deepEmoji1.png
  - deepEmoji2.png
  - deepEmoji3.png
  - denseonehot.png
  - devsets.png
  - disg.pdf
  - distributionalSocher.png
  - doc2vec.png
  - documents.jpg
  - dogs_vs_cats1.png
  - dogs_vs_cats2.png
  - dogs_vs_cats3.png
  - dogs_vs_cats4.png
  - drop-out-in-neural-networks.jpg
  - elman.pdf
  - elmanRNN.png
  - elmo.png
  - elmo2.png
  - elmo_results.png
  - emb_matrix.pdf
  - embeddings_enc.png
  - encdecattention.png
  - encoder_with_tensors.png
  - encoder_with_tensors_2.png
  - escudocolor.png
  - fear.pdf
  - federalist.png
  - fewshotresults.png
  - forwardPass.png
  - gatedEx.png
  - generator.png
  - gensim.png
  - goldbergNLP.jpg
  - gpt3.png
  - gpt4.png
  - gptllama.png
  - gradientdescent.png
  - greedysearch.png
  - GRU.png
  - hamilton.png
  - hardtanh.png
  - instructionfinetuning.png
  - invFile.png
  - joy.pdf
  - kmeans.png
  - label_classification.png
  - label_pos_ner.png
  - label_translation.png
  - lambda.png
  - lambdaresults.png
  - langbigrams.png
  - langbigramstest.png
  - langchain3.png
  - Large-Language-Models.jpg
  - linear-relationships.png
  - linear_models.ods
  - LLMbenchmarks.png
  - llmfinetuning.pdf
  - llmsapp.pdf
  - llmscale.png
  - llmscale_trans.png
  - llmtimeline.png
  - lmdiagram.png
  - lmnotuserassistant.png
  - lmsampling.png
  - logo-fcfm.jpg
  - logo.pdf
  - lora.png
  - lossfunctions.png
  - lstm_nlm.png
  - LSTMchain.png
  - LSTMform.png
  - madison.png
  - medarticle.png
  - minibatch-SGD.png
  - MLM.png
  - MLvsDL.png
  - MMLU.png
  - morphology.png
  - mt.png
  - mtbench.png
  - mtbenchex1.png
  - mtbenchex2.png
  - multi_head_attention.png
  - multiple_paths_chain_rule.png
  - multiple_paths_chain_rule_general.png
  - naive_example.png
  - negWordsTranfer.pdf
  - NER.png
  - neural_net.pdf
  - neural_net_mat_mul.pdf
  - neural_network.png
  - nlp_intersection.png
  - nmt_progress.png
  - NN-example.png
  - NSP.png
  - onehot.png
  - Online-SGD.png
  - openLLMbenchmark.png
  - parseTree.png
  - parseTree1.png
  - porter.png
  - POStags.png
  - posWordsTranfer.pdf
  - prompting.png
  - qlora.png
  - quant-freeze.png
  - random_forest.png
  - react.png
  - recTensor1.png
  - recTensor2.png
  - relu.png
  - retrievalaugmented.pdf
  - RLHF.png
  - RNN-recursion.png
  - RNN-unrolled.png
  - RNN.png
  - sadness.pdf
  - searchengine.png
  - self-attention-matrix-calculation-2.png
  - self-attention-matrix-calculation.png
  - self-attention-output.png
  - self-attention_softmax.png
  - sent.png
  - seq2se2train.png
  - seqseq.png
  - sgd.png
  - shannon.png
  - sigmoid.png
  - sigmoid2.png
  - simple_chain_rule.png
  - skip-gram.png
  - skip_gram_net_arch.png
  - smashing1.png
  - smashing2.png
  - spam.png
  - SSPE.png
  - SSPERes.png
  - stackedRNN.png
  - superglue.png
  - supervised.png
  - surprise.pdf
  - SVM.png
  - tanh.png
  - the_transformer_3.png
  - The_transformer_encoder_decoder_stack.png
  - The_transformer_encoders_decoders.png
  - transducer.png
  - transformer.png
  - transformer_attention_heads_qkv.png
  - transformer_attention_heads_weight_matrix_o.png
  - transformer_attention_heads_z.png
  - transformer_decoder.png
  - transformer_decoder_output_softmax.png
  - Transformer_encoder.png
  - transformer_multi-headed_self-attention-recap.png
  - transformer_positional_encoding_example.png
  - transformer_positional_encoding_vectors.png
  - transformer_resideual_layer_norm.png
  - transformer_resideual_layer_norm_2.png
  - transformer_self-attention_visualization.png
  - transformer_self_attention_score.png
  - transformer_self_attention_vectors.png
  - transformerdecoder.png
  - transformerencoder.png
  - transformerresults.png
  - trust.pdf
  - twitter.png
  - types_of_attention.png
  - u.png
  - ulmfit1.png
  - ulmfit2.png
  - ulmfit3.png
  - ulmfit4.png
  - validation.png
  - VDCNN.png
  - vectordatabase.png
  - viterbi1.pdf
  - viterbi2.pdf
  - wordnet.png
  - wrong_caption.png
  - xor.png
  - xor2.png
  - zeroonefew.png
  - zipf1.png
- agents.pdf
- agents.tex
- bio.bib
- language_questions.pdf
- language_questions.tex
- language_questions_2026.pdf
- NLP-RNN.pdf
- NLP-RNN.tex
- NLP_Bert-GPT.pdf
- NLP_Interpretability.pdf
- NLP_Intro.pdf
- NLP_Intro.tex
- NLP_Linear.pdf
- NLP_Linear.tex
- NLP_Neural.pdf
- NLP_PLM.pdf
- NLP_PLM.tex
- NLP_Seq2Seq-Attention.pdf
- NLP_Transformers.pdf
- rag.pdf
- rag.tex
- vector_semantics.pdf
- vector_semantics.tex
- word_vectors.pdf
- word_vectors.tex
- - aux0.ipynb
  - example.html
  - pandoc.css
  - README.md
- - Auxiliar_1.ipynb
- - Auxiliar_2.ipynb
- - Auxiliar_3.ipynb
- - Auxiliar4_parte1.ipynb
  - Auxiliar4_parte2.ipynb
  - Auxiliar4_parte4.ipynb
- - Auxiliar_5.ipynb
  - README.md
- - material_de_apoyo_competencia1.ipynb
- - BoW-Problem.png
  - emails_clases.PNG
  - limpiapiscinas.PNG
  - overview-skipgram.png
  - Skip-gram.png
  - spam.PNG
  - spam_2.PNG
.gitignore
calendar.md
old_calendar.md
README.md
update.sh

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# CC6205 - Natural Language Processing

This is a course on natural language processing.

- Lecturers: [Juan José Alegría](https://www.linkedin.com/in/juanjo-alegr%C3%ADa/) and [Jorge Ortiz Fuentes](https://ortizfuentes.com)
- Based on the work of [Felipe Bravo-Marquez](https://felipebravom.com/)
- TAs: [Felipe Urrutia](http://www.dim.uchile.cl/~furrutia/) and Juan Pablo Herrera
- [Course Notes (in Spanish)](https://raw.githubusercontent.com/dccuchile/CC6205/master/apunte/apunte.pdf)
- Lectures: Tuesday 14:30 - 16:00, Thursday 14:30 - 16:00
- [Course Program](https://docs.google.com/document/d/1DNja7nf0b26aRWF_gMNJf9L6SLtvtyFpucDhfcgG4d0/edit?usp=sharing) (in Spanish)
- [Course Calendar](calendar.md)
- [2024 Course Calendar (just as a reference)](old_calendar.md)
- [Youtube Playlist with lectures](https://www.youtube.com/playlist?list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi)

### Info

This course aims to provide a comprehensive introduction to Natural Language Processing (NLP) by covering essential concepts. We strive to strike a balance between traditional techniques, such as N-gram language models, Naive Bayes, and Hidden Markov Models (HMMs), and modern deep neural networks, including word embeddings, recurrent neural networks (RNNs), and transformers.

The course material draws from various sources. In many instances, sentences from these sources are directly incorporated into the slides. The neural network topics primarily rely on the book [Neural Network Methods for Natural Language Processing](https://link.springer.com/book/10.1007/978-3-031-02165-7) by Goldberg. Non-neural network topics, such as Probabilistic Language Models, Naive Bayes, and HMMs, are sourced from [Michael Collins' course](http://www.cs.columbia.edu/~mcollins/) and [Dan Jurafsky's book](https://web.stanford.edu/~jurafsky/slp3/). Additionally, some slides are adapted from online tutorials and other courses, such as [Manning's Stanford course](http://web.stanford.edu/class/cs224n/).

**2025 Update**: We are currently updating the course program to incorporate new capabilities of Large Language Models and explore interesting applications. As a result, we will omit some topics covered in previous iterations, such as Naive Bayes, Hidden Markov Models, and Convolutional Neural Networks. Additionally, we will assume a basic understanding of machine learning concepts, including train/test/validation splits, cross-validation, and fundamental models (SVM, Naive Bayes, and possibly Random Forest). However, the material from previous years remains available.

## Slides

As we implement this new version of the course, we will upload all the new slides.

### Unit 1: NLP fundamentals

1. [Introduction to Natural Language Processing](slides_v2/NLP_Intro.pdf)
2. [Vector semantics (in Spanish)](slides_v2/vector_semantics.pdf)
3. [Fundamental questions about language](slides_v2/language_questions_2026.pdf)
4. [Probabilistic Language Models](slides_v2/NLP_PLM.pdf)
5. [Linear Models](slides_v2/NLP_Linear.pdf) | [Spreadsheet with toy example](https://docs.google.com/spreadsheets/d/1CC5V52oxvWCPk17-5J8GDAQz2zjLkNd_VY6fMW5btUg/edit?usp=drive_link)

### Unit 2: Neural networks and NLP

6. [Neural Networks](slides_v2/NLP_Neural.pdf)
7. [Word Vectors](slides_v2/word_vectors.pdf)
8. [Recurrent Neural Networks](slides_v2/NLP-RNN.pdf)
9. [Sequence-to-sequence + Attention](slides_v2/NLP_Seq2Seq-Attention.pdf)
10. [Transformers](slides_v2/NLP_Transformers.pdf)

### Unit 3: Large Language Models: new paradigm and open questions

11. [BERT & GPT](slides_v2/NLP_Bert-GPT.pdf)
12. [Retrieval Augmented Generation](slides_v2/rag.pdf)
13. [Interpretability](slides_v2/NLP_Interpretability.pdf)
14. [Agents](slides_v2/agents.pdf)

## Evaluation

- **NC**: 2 tests (after Unit 1 and Unit 2)
- **NT**: 3 group homework assignments
- **NP**: Group presentation of a paper of your choice (we will upload the list of papers soon)

- **NF (final grade)**: (NC + NT + NP) / 3
  - **Important**: NC, NT and NP must be >= 4.0

## Slides from previous years

1. [Introduction to Natural Language Processing](slides/NLP-introduction.pdf) | ([tex source file](slides/NLP-introduction.tex)), [video 1](https://youtu.be/HEKTNOttGvU), [video 2](https://youtu.be/P8cwnI-f-Kg)
2. [Vector Space Model and Information Retrieval](slides/NLP-IR.pdf) | ([tex source file](slides/NLP-IR.tex)), [video 1](https://www.youtube.com/watch?v=FXIVClF370w&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=2&t=0s), [video 2](https://www.youtube.com/watch?v=f8nG1EMmPZk&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=2)
3. [Probabilistic Language Models](slides/NLP-PLM.pdf) | ([tex source file](slides/NLP-PLM.tex)), [notes](http://www.cs.columbia.edu/~mcollins/lm-spring2013.pdf), [video 1](https://www.youtube.com/watch?v=9E2jJ6kcb4Y&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=3), [video 2](https://www.youtube.com/watch?v=ZWqbEQXLra0&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=4), [video 3](https://www.youtube.com/watch?v=tsumFqwFlaA&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=5), [video 4](https://www.youtube.com/watch?v=s3TWdv4sqkg&list=PLppKo85eGXiXIh54H_qz48yHPHeNVJqBi&index=6)
4. [Text Classification and Naive Bayes](slides/NLP-NB.pdf) | ([tex source file](slides/NLP-NB.tex)) , [notes](https://web.stanford.edu/~jurafsky/slp3/4.pdf), [video 1](https://youtu.be/kG9BK9Oy1hU), [video 2](https://youtu.be/Iqte5kKHvzE), [video 3](https://youtu.be/TSJg0_X3Abk)
5. [Linear Models](slides/NLP-linear.pdf) | ([tex source file](slides/NLP-linear.tex)), [video 1](https://youtu.be/zhBxDsNLZEA), [video 2](https://youtu.be/Fooua_uaWSE), [video 3](https://youtu.be/DqbzhdQa1eQ), [video 4](https://youtu.be/1nfWWXqfAzA)
6. [Neural Networks](slides/NLP-neural.pdf) | ([tex source file](slides/NLP-neural.tex)), [video 1](https://youtu.be/oHZHA8h2xN0), [video 2](https://youtu.be/2lXank0W6G4), [video 3](https://youtu.be/BUDIi9qItzY), [video 4](https://youtu.be/KKN2Ipy-vGk)
7. [Word Vectors](slides/NLP-wordvectors.pdf) | ([tex source file](slides/NLP-wordvectors.tex)) [video 1](https://youtu.be/wtwUsJMC9CA), [video 2](https://youtu.be/XDxzQ7JU95U), [video 3](https://youtu.be/Ikyc3DRVodk)
8. [Sequence Labeling and Hidden Markov Models ](slides/NLP-HMM.pdf) | ([tex source file](slides/NLP-HMM.tex)), [notes](http://www.cs.columbia.edu/~mcollins/hmms-spring2013.pdf), [video 1](https://youtu.be/-ngfOZz8yK0), [video 2](https://youtu.be/Tjgb-yQOg54), [video 3](https://youtu.be/aaa5Qoi8Vco), [video 4](https://youtu.be/4pKWIDkF_6Y)
9. [MEMMs and CRFs](slides/NLP-CRF.pdf) | ([tex source file](slides/NLP-CRF.tex)), [notes 1](http://www.cs.columbia.edu/~mcollins/crf.pdf), [notes 2](http://www.cs.columbia.edu/~mcollins/fb.pdf), [video 1](https://youtu.be/qlI-4lSUDkg), [video 2](https://youtu.be/PLoLKQwkONw), [video 3](https://youtu.be/ZpUwDy6o28Y) (optional)
10. [Convolutional Neural Networks](slides/NLP-CNN.pdf) | ([tex source file](slides/NLP-CNN.tex)), [video](https://youtu.be/lLZW5Fn40r8)
11. [Recurrent Neural Networks](slides/NLP-RNN.pdf) | ([tex source file](slides/NLP-RNN.tex)), [video 1](https://youtu.be/BmhjUkzz3nk), [video 2](https://youtu.be/z43YFR1iIvk),
12. [Sequence to Sequence Models and Attention](slides/NLP-seq2seq.pdf) | ([tex source file](slides/NLP-seq2seq.tex)), [video 1](https://youtu.be/OpKxRjISqmM), [video 2](https://youtu.be/WQ7ihm5voB0)
13. [Transformer Architecture](slides/NLP-transformer.pdf) | ([tex source file](slides/NLP-seq2seq.tex)), [video 1](https://youtu.be/8RE23Uq8rU0)
14. [Contextualized Embeddings and Large Language Models](slides/NLP-LLM.pdf), [video 1](https://youtu.be/sSGbgZpHymI), [video 2](https://youtu.be/C-QfzWU6eUE), [video 3](https://youtu.be/5j4Mgl3GuVY)
15. [Large Language Models Usage and Evaluation Patterns](slides/NLP-LLMpatterns.pdf), [video](https://youtu.be/l2WpZC4dBAs?feature=shared)

## NLP Libraries and Tools

1. [NLTK: Natural Language Toolkit](https://www.nltk.org/)
2. [Gensim](https://radimrehurek.com/gensim/)
3. [spaCy: Industrial-strength NLP](https://spacy.io/)
4. [Torchtext](https://torchtext.readthedocs.io/en/latest/)
5. [AllenNLP: Open source project for designing deep leaning-based NLP models](https://allennlp.org/)
6. [HuggingFace Transformers](https://huggingface.co/docs/transformers/index)
7. [ChatGPT](https://chat.openai.com/)
8. [Google Bard](https://bard.google.com)
9. [Stanza - A Python NLP Library for Many Human Languages](https://stanfordnlp.github.io/stanza/)
10. [FlairNLP: A very simple framework for state-of-the-art Natural Language Processing (NLP)](https://github.com/flairNLP/flair)
11. [WEFE: The Word Embeddings Fairness Evaluation Framework](https://wefe.readthedocs.io/en/latest/)
12. [WhatLies: A library that tries help you to understand. "What lies in word embeddings?"](https://rasahq.github.io/whatlies/)
13. [LASER:a library to calculate and use multilingual sentence embeddings](https://github.com/facebookresearch/LASER)
14. [Sentence Transformers: Multilingual Sentence Embeddings using BERT / RoBERTa / XLM-RoBERTa & Co. with PyTorch](https://github.com/UKPLab/sentence-transformers)
15. [Datasets: a lightweight library with one-line dataloaders for many public datasets in NLP](https://github.com/huggingface/datasets)
16. [RiverText: A Python Library for Training and Evaluating Incremental Word Embeddings from Text Data Streams](https://dccuchile.github.io/rivertext/)

## Notes and Books

1. [Speech and Language Processing (3rd ed. draft) by Dan Jurafsky and James H. Martin](https://web.stanford.edu/~jurafsky/slp3/).
2. [Michael Collins' NLP notes](http://www.cs.columbia.edu/~mcollins/).
3. [A Primer on Neural Network Models for Natural Language Processing by Joav Goldberg](https://u.cs.biu.ac.il/~yogo/nnlp.pdf).
4. [Natural Language Understanding with Distributed Representation by Kyunghyun Cho](https://arxiv.org/abs/1511.07916)
5. [A Survey of Large Language Models](https://arxiv.org/abs/2303.18223)
6. [Natural Language Processing Book by Jacob Eisenstein](https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf)
7. [NLTK book](http://www.nltk.org/book/)
8. [Embeddings in Natural Language Processing by Mohammad Taher Pilehvar and Jose Camacho-Collados](http://josecamachocollados.com/book_embNLP_draft.pdf)
9. [Dive into Deep Learning Book](https://d2l.ai/)
10. [Contextual Word Representations: A Contextual Introduction by Noah A. Smith](https://arxiv.org/pdf/1902.06006.pdf)
11. [Neural Networks and Deep Learning by Michael Nielsen](http://neuralnetworksanddeeplearning.com/)

## Other NLP Courses

1. [CS224n: Natural Language Processing with Deep Learning, Stanford course](http://web.stanford.edu/class/cs224n/)
2. [Deep Learning in NLP: slides by Horacio Rodríguez](https://www.cs.upc.edu/~horacio/ahlt/DeepLearning02.pdf)
3. [David Bamman NLP Slides @Berkley](http://people.ischool.berkeley.edu/~dbamman/nlp18.html)
4. [CS 521: Statistical Natural Language Processing by Natalie Parde, University of Illinois](http://www.natalieparde.com/teaching/cs521_spring2020.html)
5. [10 Free Top Notch Natural Language Processing Courses](https://www.kdnuggets.com/2019/10/10-free-top-notch-courses-natural-language-processing.html)

### Videos

1. [Natural Language Processing MOOC videos by Dan Jurafsky and Chris Manning, 2012](https://www.youtube.com/playlist?list=PLoROMvodv4rOFZnDyrlW3-nI7tMLtmiJZ&disable_polymer=true)
2. [Natural Language Processing MOOC videos by Michael Collins, 2013](https://www.youtube.com/channel/UCB_JX4jH3QQmp69rmkWpl1A/playlists?shelf_id=3&view=50&sort=dd)
3. [Natural Language Processing with Deep Learning by Chris Manning and Richard Socher, 2017](https://www.youtube.com/playlist?list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6)
4. [CS224N: Natural Language Processing with Deep Learning | Winter 2019](https://www.youtube.com/playlist?list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z)
5. [Computational Linguistics I by Jordan Boyd-Graber University of Maryland](https://www.youtube.com/playlist?list=PLegWUnz91WfuPebLI97-WueAP90JO-15i)
6. [Visualizing and Understanding Recurrent Networks](https://skillsmatter.com/skillscasts/6611-visualizing-and-understanding-recurrent-networks)
7. [BERT Research Series by Chris McCormick](https://www.youtube.com/playlist?list=PLam9sigHPGwOBuH4_4fr-XvDbe5uneaf6)
8. [Successes and Challenges in Neural Models for Speech and Language - Michael Collins](https://www.youtube.com/watch?v=jfwqRMdTmLo)
9. [More on Transforemers: BERT and Friends by Jorge Pérez](https://tv.vera.com.uy/video/55388)

## Other Resources

1. [ACL Portal](https://www.aclweb.org/portal/)
2. [Awesome-nlp: A curated list of resources dedicated to Natural Language Processing](https://github.com/keon/awesome-nlp)
3. [NLP-progress: Repository to track the progress in Natural Language Processing (NLP)](http://nlpprogress.com/)
4. [Corpora Mailing List](https://mailman.uib.no/listinfo/corpora)
5. [🤗 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
6. [Real World NLP Book: AllenNLP tutorials](http://www.realworldnlpbook.com/)
7. [The Illustrated Transformer: a very illustrative blog post about the Transformer](http://jalammar.github.io/illustrated-transformer/)
8. [Better Language Models
   and Their Implications OpenAI Blog](https://openai.com/blog/better-language-models/)
9. [Understanding LoRA and QLoRA — The Powerhouses of Efficient Finetuning in Large Language Models](https://medium.com/@gitlostmurali/understanding-lora-and-qlora-the-powerhouses-of-efficient-finetuning-in-large-language-models-7ac1adf6c0cf)
10. [RNN effectiveness](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
11. [SuperGLUE: an benchmark of Natural Language Understanding Tasks](https://super.gluebenchmark.com/)
12. [decaNLP The Natural Language Decathlon: a benchmark for studying general NLP models that can perform a variety of complex, natural language tasks](http://decanlp.com/).
13. [Chatbot and Related Research Paper Notes with Images](https://github.com/ricsinaruto/Seq2seqChatbots/wiki/Chatbot-and-Related-Research-Paper-Notes-with-Images)
14. [Ben Trevett's torchtext tutorials](https://github.com/bentrevett/)
15. [PLMpapers: a collection of papers about Pre-Trained Language Models](https://github.com/thunlp/PLMpapers)
16. [The Illustrated GPT-2 (Visualizing Transformer Language Models)](https://jalammar.github.io/illustrated-gpt2/)
17. [Linguistics, NLP, and Interdisciplinarity Or: Look at Your Data, by Emily M. Bender](https://medium.com/@emilymenonbender/linguistics-nlp-and-interdisciplinarity-or-look-at-your-data-e49e03d37c9c)
18. [The State of NLP Literature: Part I, by Saif Mohammad](https://medium.com/@nlpscholar/state-of-nlp-cbf768492f90)
19. [From Word to Sense Embeddings:A Survey on Vector Representations of Meaning](https://arxiv.org/pdf/1805.04032.pdf)
20. [10 ML & NLP Research Highlights of 2019 by Sebastian Ruder](https://ruder.io/research-highlights-2019/index.html)
21. [Towards a Conversational Agent that Can Chat About…Anything](https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html?m=1)
22. [The Super Duper NLP Repo: a collection of Colab notebooks covering a wide array of NLP task implementations](https://notebooks.quantumstat.com/)
23. [The Big Bad NLP Database, a collection of nearly 300 well-organized, sortable, and searchable natural language processing datasets](https://datasets.quantumstat.com/)
24. [A Primer in BERTology: What we know about how BERT works](https://arxiv.org/abs/2002.12327)
25. [How Self-Attention with Relative Position Representations works](https://link.medium.com/wFxx3d96f7)
26. [Deep Learning Based Text Classification: A Comprehensive Review](https://arxiv.org/pdf/2004.03705.pdf)
27. [Teaching NLP is quite depressing, and I don't know how to do it well by Yoav Goldberg](https://twitter.com/yoavgo/status/1318567498653061122)
28. [The NLP index](https://index.quantumstat.com/)
29. [100 Must-Read NLP Papers](https://github.com/amanchadha/100-nlp-papers)

CC6205

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

CC6205

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README