MS_038_F21

KO | EN

gitlite — search

#hacktoberfest #llm #rust #deep-learning #linux #nodejs #java #windows #kubernetes #awesome-list #vue #pytorch #mcp #openai

(★ 9)

These materials for "Machine Learning for Artists" at Scripps College in the Fall of 2021

MS_038_F21 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MS_038_F21 | Machine Learning for Artists
## Professor Douglas Goodwin
### MW 09:35-10:50AM
### Scripps Campus, Steele Hall, 229

## [Syllabus](https://github.com/scrippscollege/MS038_F21/) | [Discord](https://discord.gg/2DmnS3jh)

## DESCRIPTION

Machine learning (ML) is a new branch of computer science that provides services for automatic translation and speech recognition (Apple's Siri, Amazon's Alexa, Google Assistant), product recommendations (Netflix, Amazon, etc.), transportation (Waymo, Tesla, the City of Copenhagen), and political campaigns (Facebook and Cambridge Analytica). ML is becoming a familiar presence in our lives; computer scientists and developers introduce new applications every day for chatting with humans, recommending the best course of action, and making predictions about the future. In spite of all the press, ML remains daunting to non-specialists. This class seeks to mend this divide. This class will introduce ML concepts to students without prior experience and provide templates to get students working in ML right away.

We will study and remake artworks by Mario Klingemann, Anna Ridler, Sougwen Chung, Memo Akten, Helena Sarin, Tom White, and others. We will try several techniques and frameworks such as image segm	entation, CycleGAN, pix2pix, and Tensorflow. Students will propose and work on a larger project in the last third of the class.

Prerequisite: Any experience with programming, especially with p5.js, ml5.js, and Python.

## QUESTIONS

-   What is machine intelligence and artificial intelligence?
-   What are examples of machine intelligence?
-   How does AI impact you?
-   Are you excited about AI, concerned about it, or both?
-   What topics in AI interest you?

## Required Instructional Materials

### Free [P5.js editor](https://editor.p5js.org) account

To work on ml5.js in an online context.

### [Free GitHub account](https://github.com), https://github.com

An online development platform. Our syllabus lives on GitHub. Essential.

### Free [Google Colab account](https://colab.research.google.com/) with Google Drive

To run ML code in the cloud. Integrate is with Google Drive for the full experience!

## Recommended Material

### *The Information: A History, a Theory, a Flood*, James Gleick

[Aesthetic Programing](http://aesthetic-programming.net), by Winnie Soon & Geoff Cox, http://aesthetic-programming.net.

## Assessment

### 40% Weekly assignments

### 40% Final project (proposal and completion)

### 20% Attendance + Participation

### Final Project Grading is based on the following factors:

1. Perseverance
2. Faithfulness to the proposal
3. Creative engagement
4. Application of critical thinking
5. Preparation (research, accumulation of needed materials, time management) 
6. Success of the finished piece

## Course and Institutional Policies

#### **COVID-19 Policies**:

Please wear your masks during class. Please consult https://www.scrippscollege.edu/scripps-strong/return-to-campus-plan/ for the latest information.

#### Attendance Policy:

You may miss up to four classes and still pass this class. Note that 60 percent of the students' grade is given for work on in-class activities, and that these cannot be repeated except in extraordinary circumstances.

#### Participation Policy:

You are expected to be attentive, ask questions, work alone and with a partner to complete your work.

#### Late Assignment and Missed Exam Policy:

Labs and in-class activities will not be repeated except in extraordinary circumstances.

#### Academic Integrity:

Students are expected to abide by the Scripps College academic integrity code. You must submit work that is your own and which is original work for this class. Also, all sources must be documented. Omission of sources is considered plagiarism, even if it is an oversight and/or unintentional. All plagiarism will be reported to the department and Dean’s office for further action. For this course, collaboration is allowed in on lab activities and assignments IFF all contributions are documented.

Permissible cooperation should never involve one student having possession of a copy of all or part of work done by someone else, in any form (e.g. email, Word doc, Box file, Google sheet, or a hard copy). Also, assignments that have been previously submitted in another course may not be submitted for this course, and I discourage you from finding solutions on Stack Overflow or other online forums to paste into your notebooks.

#### Accommodations for Students with Disabilities:

Scripps students seeking to register to receive academic accommodations must contact Academic Resources and Services Staff (ARS) at ars@scrippscollege.edu to formalize accommodations. Students will be required to submit documentation and meet with a staff member before being approved for accommodations. Once ARS has authorized academic accommodations, a formal notification will be sent out.

A student’s home campus is responsible for establishing and providing accommodations. If you are not a Scripps student, you must contact your home institution to establish accommodations. Below is a list of coordinators on the other campuses:

CMC - Julia Easley, julia.easley@claremontmckenna.edu

Harvey Mudd – Deborah Kahn, dkahn@hmc.edu

Pitzer- Gabriella Tempestoso, gabriella_tempestoso@pitzer.edu

Pomona - Jan Collins-Eaglin, Jan.Collins-Eaglin@pomona.edu

#### Inclusivity Statement:

This class is an example of Scripps College’s commitment to changing the norms in Computer Science. Creating this initiative at a liberal arts women's college is both a bold step towards correcting gender imbalance in this field.

Our community represents a wide variety of backgrounds and perspectives. We are committed to providing an atmosphere for learning that respects diversity.

#### Institutional Policies:

Students are responsible for reviewing Scripps College’s policies on incomplete grades, sexual misconduct, adverse weather, as well as student evaluation of instruction, and days of special concern/religious holiday.

## ASSIGNMENTS (updated Sept. 03)

| **W** | **Date** | **theme**                                    | **Reading (read before class)**                              | **videos (watch before class)**                              | **Assignment (complete before class)**                       |
| ----- | -------- | -------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| 1     | 08/30    | Introduction to Machine Learning with ml5.js |                                                              |                                                              |                                                              |
|       | 09/01    |                                              | [Awesome Machine Learning Art](https://github.com/vibertthio/awesome-machine-learning-art) | [AI4D workshop links](https://github.com/mmattyg/AI4D)       |                                                              |
| 2     | 09/06    | LABOR DAY                                    |                                                              |                                                              | Make a list of projects and people that interest you.        |
|       | 09/08    | Image Classification with Teachable Machine  | [What is a Teachable Machine](https://experiments.withgoogle.com/teachable-machine)? | [Teachable Machine 1: Image Classification](https://youtu.be/kwcillcWOg0) |                                                              |
| 3     | 9/13     | Tracking bodies                              | [Hello-ml5](https://learn.ml5js.org/#/tutorials/hello-ml5)   | [Nostalgia -> Art -> Creativity -> Evolution as  Data + Direction](https://youtu.be/tcAZ9KvBtkg?list=PLCpMvp7ftsnIbNwRnQJbDNRqO6qiN3EyH&t=2354), Alyosha Efros | Submit a project idea to Discord using Pose Estimation.      |
|       | 09/15    |                                              | [ML5js: PoseNet](https://learn.ml5js.org/#/reference/posenet) | [ml5.js Pose Estimation with PoseNet](https://www.youtube.com/watch?v=OIo-DIOkNVg), Dan Shiffman |                                                              |
| 4     | 09/20    | Tracking faces and hands                     | https://learn.ml5js.org/#/reference/face-api \| https://learn.ml5js.org/#/reference/handpose | [How Machine  Learning Can Benefit Human Creators](https://youtu.be/5PAgyGiCZEk), [Dr.Rebecca Fiebrink](https://www.doc.gold.ac.uk/~/mas01rf/homepage/) | Submit a project idea to Discord that uses hand or face tracking. |
|       | 09/22    |                                              |                                                              | Shiffman: [ml5.js Pose Estimation with PoseNet](https://youtu.be/OIo-DIOkNVg) |                                                              |
| 5     | 09/27    | Style Transfer                               | Ml5.js [StyleTransfer](https://learn.ml5js.org/#/reference/style-transfer?id=styletransfer) | [The Art Of  Deception - Encountering Perception as a Creative Material](https://youtu.be/_3NTUZ4-RVM), [Shiry  Ginosar](https://people.eecs.berkeley.edu/~shiry/) | Submit a project idea to Discord that combines style transfer with pose estimation or hand/face tracking. |
|       | 09/29    |                                              |                                                              | [Gene Kogan: Visualization, deepdream, style & texture synthesis](https://ml4a.github.io/classes/itp-F18/05/) |                                                              |
| 6     | 10/04    | pix2pix                                      | [Gene Kogan's pix2pix tutorial](https://ml4a.github.io/guides/Pix2Pix/) | [A Socratic  debate, Alyosha Efros and Phillip Isola](https://youtu.be/tcAZ9KvBtkg) | Submit a project idea to Discord that uses pix2pix with pose estimation or hand/face tracking. |
|       | 10/06    |                                              |                                                              |                                                              |                                                              |
| 7     | 10/11    | DIY Neural Networks                          |                                                              | [Neural  Abstractions](https://youtu.be/CU2NQyPeAH8), [Tom  White](http://drip.net/) | Submit a project idea to Discord that uses a DIY Neural Network. |
|       | 10/13    |                                              |                                                              |                                                              |                                                              |
| 8     | 10/18    | FALL BREAK                                   |                                                              |                                                              |                                                              |
|       | 10/20    |                                              |                                                              |                                                              |                                                              |

## Outline

| W    | Date         | theme                                         | Slides                                                       | Reading                                                      | videos                                                       | Demo                                                         | Assignment                                                   | Activity                                                     | Art                                                          |
| ---- | ------------ | --------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| 1    | 08/30        | Introduction to Machine Learning with ml5.js  | [intro_01.pdf](https://github.com/scrippscollege/MS038_F2021/blob/be0628521c663ddb5162f96e4e9fd8e023ed9e5a/01_intro/01.pdf) |                                                              | [**Hilary Mason   explains machine learning to 5 different people**](https://youtu.be/5q87K1WaoFI)  [**A Beginner's   Guide to Machine Learning with ml5.js**](https://youtu.be/jmznx0Q1fP0) | [Image classifier on an image](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/imageClassification-ml5/ImageClassification/)  [Image classifier with webcam](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/imageClassification-ml5/ImageClassification_Video/)  [Object Detection-YOLO-Webcam](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/ObjectDetector/ObjectDetector_YOLO_Video) | [**Intro to ml5**](https://www.youtube.com/watch?v=yNkAuWz5lnY), Dan Shiffman  [**Image classifier with ml5 and MobileNet**](https://www.youtube.com/watch?v=yNkAuWz5lnY), Dan Shiffman | Introductions, Discussion                                    | [Petra Cortright, VVEBCAM 2007](https://youtu.be/k50Mj8ZY-xY) |
|      | 09/01        |                                               |                                                              | [Awesome Machine Learning Art](https://github.com/vibertthio/awesome-machine-learning-art)  \| [AI4D workshop links](https://github.com/mmattyg/AI4D) | [Anna Ridler, Nicer Tuesdays](https://youtu.be/VlJfvoEXKbk)  | ml5.js  [examples](https://github.com/ml5js/ml5-examples)  tf.js  [examples](https://github.com/tensorflow/tfjs-examples)  [How to host p5   sketch on github pages](https://youtu.be/8HPYsDTk17A)  [ml5 image classifier webcam](https://editor.p5js.org/yining/sketches/TKI4SkqM5) (also works on the  mobile phone, [full screen](https://editor.p5js.org/yining/full/TKI4SkqM5)) | Build  on top of the [**image classifier example**](https://github.com/yining1023/machine-learning-for-the-web/tree/master/week1-intro/imageClassification-ml5/ImageClassification_Video)([**demo**](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/imageClassification-ml5/ImageClassification_Video/)) from the coding session. Publish  it on your blog / GitHub. Add your homework link to the list below.  Make something new using one example from the collected  ml5js [**examples**](https://github.com/ml5js/ml5-examples) | https://learn.ml5js.org/#/                                   | [Laws   of Ordered Form](http://annaridler.com/laws-of-ordered-form), Anna Ridler |
| 2    | 09/06        |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      | 09/08        | Image Classification with Teachable Machine   |                                                              | What  is a Teachable Machine? KNN image classifier?  How does Teachable Machine work? | [Lauren McCarthy   Explores Surveillance and Relationships](https://youtu.be/LBzQMZM-vEo) | Existing  projects about transfer learning using a webcam    | [**How to build a Teachable Machine with TensorFlow.js**](https://beta.observablehq.com/@nsthorat/how-to-build-a-teachable-machine-with-tensorflow-js), [**Nikhil   Thorat**](https://observablehq.com/@nsthorat)  Coding  train: ml5.js: KNN Classification [**parts 1- 3**](https://youtu.be/KTNqXwkLuM4), Dan Shiffman | Add  new outputs to the KNN Image Classifier example, mix it with videos, games,  or physical computing. Publish your project on GitHub or your own blog, or  record a video and put it on your blog. Add your project link below. |                                                              |
| 3    | 09/13,       | Tracking bodies                               | What  is PoseNet?  Can  you use PoseNet + KNN Image Classifier?  What is BodyPix body segmentation? | *[*The Shape of   Art History in the Eyes of the Machine**](https://youtu.be/Xw1UrWb3XI8), Ahmed Elgammal | [**Real-time Human Pose Estimation in the Browser with   TensorFlow.js**](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5), [***Dan Oved\***](http://www.danioved.com/), freelance creative  technologist at Google Creative Lab*  [**Introducing BodyPix: Real-time Person Segmentation in the   Browser with TensorFlow.js**](https://medium.com/tensorflow/introducing-bodypix-real-time-person-segmentation-in-the-browser-with-tensorflow-js-f1948126c2a0), [***Dan Oved\***](http://twitter.com/oveddan) |                                                              | [**Hour of Code   with p5.js and PoseNet**](https://youtu.be/EA3-k9mnLHs), Dan Shiffman  [**ml5.js Pose Estimation with PoseNet**](https://www.youtube.com/watch?v=OIo-DIOkNVg), Dan Shiffman | Make a  KNN Image Classifier                                 | [Body, Movement, Language: AI Sketches With Bill T. Jones](https://experiments.withgoogle.com/billtjonesai) |
|      | 09/15        |                                               | PoseNet  PoseNet  + KNN Image Classifier  BodyPix            |                                                              | [Maya Man   Interview : p5.js](https://youtu.be/hRZCrEjeBUw) \| Diversity with Code + Art Series |                                                              | Build  an interactive browser experiment with Webcam Poses data. | Maya Man’s PoseNet Sketchbook                                |                                                              |
| 4    | 09/20        | Tracking faces and hands                      | Face  detection  Face  recognition  Face landmark detection  |                                                              | [A Socratic   debate, Alyosha Efros and Phillip Isola](https://youtu.be/tcAZ9KvBtkg) | Just a dude who hacks: [face-api.js playground](https://justadudewhohacks.github.io/face-api.js/face_and_landmark_detection/) | [**face-api.js — JavaScript API for Face Recognition in the   Browser with tensorflow.js**](https://itnext.io/face-api-js-javascript-api-for-face-recognition-in-the-browser-with-tensorflow-js-bcc2a6c4cf07)  [**Introducing BodyPix: Real-time Person Segmentation in the   Browser with TensorFlow.js**](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html) |                                                              |                                                              |
|      | 09/22        |                                               |                                                              |                                                              | [How Machine   Learning Can Benefit Human Creators](https://youtu.be/5PAgyGiCZEk), [Dr.Rebecca Fiebrink](https://www.doc.gold.ac.uk/~/mas01rf/homepage/) |                                                              | Play with Just a dude who hacks: [face-api.js playground](https://justadudewhohacks.github.io/face-api.js/face_and_landmark_detection/), replace images | Detect  faces  Recognize  faces  Detect and track face landmark |                                                              |
| 5    | 09/27        | Style Transfer                                | What  is Style Transfer?  How  does it work?  What do Neural Networks see? | TBD                                                          | [The Art Of   Deception - Encountering Perception as a Creative Material](https://youtu.be/_3NTUZ4-RVM), [Shiry   Ginosar](https://people.eecs.berkeley.edu/~shiry/) |                                                              | TBD                                                          | TBD                                                          |                                                              |
|      | 09/29        |                                               |                                                              |                                                              | [Explorations in   AI for Creativity](https://youtu.be/vRGL_V3ZsBA), [Devi Parikh](https://www.cc.gatech.edu/~parikh/) | [**Style Transfer on one image**](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/styleTransfer-ml5/StyleTransfer_Image/)  [**Style Transfer on webcam images**](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/styleTransfer-ml5/StyleTransfer_Video/)  [**Photo Styles Transfer with Runway**](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/photostylestansfer/) (Need to open Runway, and run 'FastPhotoStyle' model on  localhost:8000) | Train  your own fast style transfer model and run the model in the browser with  ml5.js  Or  use any of the ml5's [**pre-trained Style Transfer models**](https://github.com/ml5js/ml5-data-and-models/tree/master/models/style-transfer) to  create a new sketch  Publish your project on GitHub or your own blog, or record  a video and put it on your blog. Add your project link below. | Train  a new Style Transfer model  Run a Style Transfer model in ml5.js |                                                              |
| 6    | 10/04        | pix2pix                                       | What  is pix2pix?  How  does it work?  Applications  of pix2pix  Running pix2pix with ml5: demo | Read this tutorial [**Pix2Pix with Tensorflow**](https://www.tensorflow.org/tutorials/generative/pix2pix), TF team | [Nostalgia -> Art -> Creativity -> Evolution as   Data + Direction](https://youtu.be/tcAZ9KvBtkg?list=PLCpMvp7ftsnIbNwRnQJbDNRqO6qiN3EyH&t=2354), Alyosha Efros |                                                              | TBD                                                          | TBD                                                          |                                                              |
|      | 10/06        |                                               |                                                              |                                                              | [AI+Creativity,   an Art Nerd's Perspective](https://youtu.be/4ti-DjN94Gw), Jason Bailey |                                                              | Try training Pix2Pix  with CMP Facade Database in [**Google Colab**](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/pix2pix.ipynb). You can find other  datasets [**here**](https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/) | Running  pix2pix with ml5.js  Setup  Spell.run training environment  Prepare  dataset for pix2pix  Training a new pix2pix model |                                                              |
| 7    | 10/11        | DIY Neural Networks                           |                                                              |                                                              | [Neural   Abstractions](https://youtu.be/CU2NQyPeAH8), [Tom   White](http://drip.net/) |                                                              |                                                              |                                                              |                                                              |
|      | 10/13        |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 8    | 10/18,10/20  |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 9    | 10/25, 10/27 | ML on Google. Colab, CycleGAN                 |                                                              |                                                              | [Efficient GANs](https://youtu.be/MQ7QEEB5oFQ),  Jun-Yan Zhu |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              | [Magenta:   Empowering creative agency with machine learning](https://youtu.be/oiPWOTr44qQ), [Jesse   Engel](https://jesseengel.github.io/about/) |                                                              |                                                              |                                                              |                                                              |
| 10   | 11/01, 11/03 | Embedding bias (Google) / Adversarial attacks |                                                              |                                                              | [Artificial   Biodiversity](https://youtu.be/xDc9zSPzqvQ), [Sofia Crepso](https://sofiacrepso.com/) & Feileacan McCormick |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              | ["Creative-Networks"](https://youtu.be/I0V2UTKIei0),   Joel Simon |                                                              |                                                              |                                                              |                                                              |
| 11   | 11/08, 11/10 | BigGAN, latent spaces, GLOW, RNNs             |                                                              |                                                              | [Sequence   Modeling: Recurrent and Recursive Nets presented](https://youtu.be/ZVN14xYm7JA), Ian Goodfellow |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 12   | 11/15, 11/17 | First Order Motion Model                      |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 13   | 11/22, 11/24 | Projects                                      |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 14   | 11/29, 12/01 | Projects                                      |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 15   | 12/06, 12/08 | Projects                                      |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
| 16   |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |
|      |              |                                               |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |                                                              |

## 1. TEACHABLE MACHINE + P5.JS

We begin by using the GPU in the browser.

## Homework

**WATCH**

[Intro to ml5](https://www.youtube.com/watch?v=yNkAuWz5lnY), Dan Shiffman

[Image classifier with ml5 and MobileNet](https://www.youtube.com/watch?v=yNkAuWz5lnY), Dan Shiffman

[ml4w-homework: How to push code to a Github Repo and host sketch on Github](https://github.com/yining1023/ml4w-homework)

[Video: How to host p5 sketch on GitHub pages](https://youtu.be/8HPYsDTk17A)

**CODE**

[Anna Ridler, Investigations into the pathways between words, definitions and data](http://annaridler.com/language-and-classification)

Build on top of the [image classifier example](https://github.com/yining1023/machine-learning-for-the-web/tree/master/week1-intro/imageClassification-ml5/ImageClassification_Video)([demo](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/imageClassification-ml5/ImageClassification_Video/)) from the coding session. Publish it on your blog / GitHub. Add your homework link to the list below.

Make something new using one example from the collected ml5js [examples](https://github.com/ml5js/ml5-examples)

### Shiffman's ml5.js videos

[Hilary Mason explains machine learning to 5 different people](https://youtu.be/5q87K1WaoFI)

[A Beginner's Guide to Machine Learning with ml5.js](https://youtu.be/jmznx0Q1fP0)

1. [ml5.js: Image Classification with MobileNet](https://youtu.be/yNkAuWz5lnY)
1. [ml5.js: Webcam Image Classification](https://youtu.be/D9BoBSkLvFo)
1. [ml5.js: Object Detection with COCO-SSD](https://youtu.be/QEzRxnuaZCk)
1. [ml5.js: Transfer Learning with Feature Extractor](https://youtu.be/kRpZ5OqUY6Y)
1. [ml5.js: Feature Extractor Classification](https://youtu.be/eeO-rWYFuG0)
1. [ml5.js: Feature Extractor Regression](https://youtu.be/aKgq0m1YjvQ)
1. [ml5.js: Save/Load Model](https://youtu.be/eU7gIy3xV30)

- Session 1 (M 08/30): Introduction to Machine Learning

- Session 2 (W 09/01): Coding session:

- Installing ml5.js
  - Running Image Classification example with ml5.js
  - Hosting p5 sketch on github or use p5 web editor
  - How to updating homework wiki

- Video:

- [Intro to ml5](https://www.youtube.com/watch?v=yNkAuWz5lnY)
  - [Image classifier with ml5 and MobileNet](https://www.youtube.com/watch?v=yNkAuWz5lnY)

- Coding:

- Build on top of the [image classifier example](https://github.com/yining1023/machine-learning-for-the-web/tree/master/week1-intro/imageClassification-ml5/ImageClassification_Video)([demo](https://yining1023.github.io/machine-learning-for-the-web/week1-intro/imageClassification-ml5/ImageClassification_Video/)) from the coding session. Publish it on your blog / GitHub. Add your homework link to the list below.
  - Or try any of the ml5js [examples](https://github.com/ml5js/ml5-examples), make something based on any of these examples.

Watch Dan Shiffman's videos before class on Wednesday.

1. [Teachable Machine 1: Image Classification](https://youtu.be/kwcillcWOg0)
1. [Teachable Machine 2: Snake Game](https://youtu.be/UPgxnGC8oBU)

We will start by using [Teachable Machine](https://teachablemachine.withgoogle.com), "a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone." This experiment from Google to bring a no-code and low-code approach to training AI models.

This teaching tool from Google demonstrates a machine learning workflow that you will use throughout the course:

1. [Gather images, sound, etc.](https://youtu.be/DFBbSTvtpy4)
2. [Train the model](https://youtu.be/CO67EQ0ZWgA)
3. Preview and Save/Export

It may be designed for High School STEM programs, but that doesn't mean that you can't make something interesting with Teachable Machine. Let's try it!

### Exercises

**Exercise 1.** Train teachable machine to learn the difference between red and green. Export your model for later.

**Exercise 2.** Train Teachable Machine to recognize your voices. Make a different class for each person--we can use the model to identify who is speaking. Save/Export your model to use in. **p5.js.**

**Exercise 3.** Let's do something interesting with your voice classifier. Watch this video showing a [collective game of PONG.](https://vimeo.com/78043173) Could you play PONG with your classifiers? Amy Goodchild [reproduced the experiment](https://www.amygoodchild.com/blog/collaborative-control-using-paddles) in 2018.

(Note that Carpenter returned to SIGGRAPH with an. airplane simulation. Could you fly an airplane collectively?)

Dan Shiffman's [video on using Teachable Machine](https://youtu.be/kwcillcWOg0) will help you with the next step (whatever that might be!). Let's try making a game!

### ML in the browser with ml5.js

Leverage your knowledge (and models) to make new projects and games.

by using a browser-based machine learning library called [ml5.js](https://ml5js.org). ml5.js provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies.

The library is supported by code examples, tutorials, and sample datasets with an emphasis on ethical computing. Bias in data, stereotypical harms, and responsible crowdsourcing are part of the documentation around data collection and usage. We're building friendly machine learning for the web - we're glad you're here!

ml5.js is heavily inspired by Processing and p5.js.

## 2. Image Classification

- Session 1 (M 09/06): NO CLASS: Labor Day

- Session 2 (W 09/08): Coding session:

- Make a KNN Image Classifier

- Reading:

- [How to build a Teachable Machine with TensorFlow.js](https://beta.observablehq.com/@nsthorat/how-to-build-a-teachable-machine-with-tensorflow-js)

- Video:

- Coding train: ml5.js: KNN Classification [part 1- 3](https://youtu.be/KTNqXwkLuM4)

- Coding:

- Add new outputs to the KNN Image Classifier example, mix it with videos, games, or physical computing. Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

- Object Detection Example [Andreas Refsgaard + Lasse Korsgaard](https://andreasrefsgaard.dk/project/an-algorithm-watching-a-movie-trailer)

## Homework

- Reading:

[How to build a Teachable Machine with TensorFlow.js](https://beta.observablehq.com/@nsthorat/how-to-build-a-teachable-machine-with-tensorflow-js), [Nikhil Thorat](https://observablehq.com/@nsthorat)

Coding train: ml5.js: KNN Classification [parts 1- 3](https://youtu.be/KTNqXwkLuM4), Dan Shiffman

- Coding:

Add new outputs to the KNN Image Classifier example, mix it with videos, games, or physical computing. Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

## 3. Tracking the body

- Session 1 (M 09/13): [Notes](https://github.com/yining1023/machine-learning-for-the-web/tree/master/week3-pose)
- Session 2 (W 09/15): Coding session:PosnetPoseNet + KNN Image ClassifierBody-pix
- Reading:
  - [Real-time Human Pose Estimation in the Browser with TensorFlow.js](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5)
  - [Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js](https://medium.com/tensorflow/introducing-bodypix-real-time-person-segmentation-in-the-browser-with-tensorflow-js-f1948126c2a0)
- Playlist:
  - [Hour of Code with p5.js and PoseNet](https://youtu.be/EA3-k9mnLHs)
  - [ml5.js Pose Estimation with PoseNet](https://www.youtube.com/watch?v=OIo-DIOkNVg)
- Coding:
  - Build an interactive browser experiment on Webcam Poses data
  - Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

#### Artist projects

- PoseNet Example [Maya Man](https://googlecreativelab.github.io/posenet-sketchbook)

## Homework

**Reading:**

[Real-time Human Pose Estimation in the Browser with TensorFlow.js](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5), [*Dan Oved*](http://www.danioved.com/), freelance creative technologist at Google Creative Lab*

[Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js](https://medium.com/tensorflow/introducing-bodypix-real-time-person-segmentation-in-the-browser-with-tensorflow-js-f1948126c2a0), [*Dan Oved*](http://twitter.com/oveddan)

**Playlist:**

[Hour of Code with p5.js and PoseNet](https://youtu.be/EA3-k9mnLHs), Dan Shiffman

[ml5.js Pose Estimation with PoseNet](https://www.youtube.com/watch?v=OIo-DIOkNVg), Dan Shiffman

**Coding:**

Build an interactive browser experiment on Webcam Poses data

Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

(sound classification)

- Session 1 (M 09/27): CSV and JSON Data
- Session 2 (W 09/29): Coding session:
  - Sound Classifer with Teachable Machine
  - Pose Classifier with Teachable Machine
  - Image Classifier with Teachable Machine
  - Classifiers with Arduino
- Reading:
  - [face-api.js — JavaScript API for Face Recognition in the Browser with tensorflow.js](https://itnext.io/face-api-js-javascript-api-for-face-recognition-in-the-browser-with-tensorflow-js-bcc2a6c4cf07)
  - [Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html)
- Coding:
  - Build an interactive browser experiment on face or hand data
  - Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

##Hands and faces

- Reading:

[face-api.js — JavaScript API for Face Recognition in the Browser with tensorflow.js](https://itnext.io/face-api-js-javascript-api-for-face-recognition-in-the-browser-with-tensorflow-js-bcc2a6c4cf07)

[Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html)

- Coding:

Build an interactive browser experiment on face or hand data

Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

## 4. Style Transfer

- Session 1 (M 10/04): Image Data

- Session 2 (W 10/06): Coding session:

- Training a new Style Transfer model
  - Running Style Transfer model in ml5.js

- Coding:

- Training a new Style Transfer model, Run this

Google Colab. Watch this: video1 | video2

- Notes:
      - Open the colab, make sure the GPU is enabled: Menu - Runtime - Change runtime type
      - Run through each cell, wait for each cell to finish running, make sure there is no error in each cell's output
      - Step 2 and 3 may take 1 and 2 hours to finish, keep the tab open and active while waiting(Power your computer while waiting)
      - Once step 2(download datasets) finishes, don't re-run it, becase it takes a long to finish
      - While running step 2(download dataset), it might notify you that "Disk is almost full", ignore that

- Running Style Transfer model in ml5.js, [p5 sketch](https://github.com/yining1023/machine-learning-for-the-web/tree/master/week5-styleTransfer/styleTransfer-ml5/StyleTransfer_Video)

- Watch these short videos:

- Introduction to Runway, [part1](https://youtu.be/ARnf4ilr9Hc), [part2](https://youtu.be/7btNir5L8Jc)
  - [Intro to ML with Runway](https://youtu.be/yoJWVSL1ST4)

- Generate videos/images/GPT using Runway models

- Or Build a sketch using Runway's model in p5.js or Processing

- Or use Photoshop or Unity Runway plugin to process images or build a game scene

- Add your blog/project link below

## Homework:

- Coding:

Train your own fast style transfer model and run the model in the browser with ml5.js

Or use any of the ml5's [pre-trained Style Transfer models](https://github.com/ml5js/ml5-data-and-models/tree/master/models/style-transfer) to create a new sketch

Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

## See demos live

[Style Transfer on one image](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/styleTransfer-ml5/StyleTransfer_Image/)

[Style Transfer on webcam images](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/styleTransfer-ml5/StyleTransfer_Video/)

[Photo Styles Transfer with Runway](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/photostylestansfer/) (Need to open Runway, and run 'FastPhotoStyle' model on localhost:8000)

[AdainStyleTransfer and bodyPix with Runway](https://yining1023.github.io/machine-learning-for-the-web/week5-styleTransfer/photostylestansfer/) (Need to host AdainStyleTransfer model in Runway, and change the model url and auth in the code)

## 5. pix2pix

## Homework:

Read this tutorial [Pix2Pix with Tensorflow](https://www.tensorflow.org/tutorials/generative/pix2pix), TF team

Try training Pix2Pix with CMP Facade Database in [Google Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/pix2pix.ipynb). You can find other datasets [here](https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/)

### (runwayml)

## 6. DIY Neural Networks

## 7. Embedding bias (Google)

Watch this video from Google. Could you intentionally build bias into a ML model?

## 8. Adversarial attacks on ML models (B0RK)

Reproduce B0RK's experiments and turn the pandas into vultures.

## 5.

## 4: [Tracking Faces and Hands](04_training)

- Session 1 (M 09/20): Demos

- Session 2 (W 09/22): Coding session:

- Face-api
  - Facemesh
  - Handpose

- Reading:

- [face-api.js — JavaScript API for Face Recognition in the Browser with tensorflow.js](https://itnext.io/face-api-js-javascript-api-for-face-recognition-in-the-browser-with-tensorflow-js-bcc2a6c4cf07)
  - [Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html)

- Coding:

- Build an interactive browser experiment on face or hand data
  - Publish your project on GitHub or your own blog, or record a video and put it on your blog. Add your project link below.

### 5: [Sound Classifier](05_training)

### 6: [Image Transformation, Style transfer](06_cnn)

## 7: [Recurrent Neural Networks](07_rnn)

- Session 1 (T 10/11): SketchRNN (Drawing)
- Session 2 (W 10/13): charRNN (Text)

### 8: FALL BREAK

- Session 1 (T 10/18): NO CLASS
- Session 2 (W 10/20): NO CLASS

### 9: [Pre-Trained Models on Google Colab](09_colab)

- Session 1 (M 10/25): Introduction to Google Colab
- Session 2 (W 10/27): Colab Model Workshop

### 9: [Generative Models with Colab](10_colab)

- Session 1 (M 11/01): Hosted Models and Networking
- Session 2 (W 11/03): Generative Adversarial Networks, Interactive Image Synthesis

### 10: [Model Training with Colab](11_runwayml)

- Session 1 (M 11/9): StyleGAN and Object Detection
- Session 2 (W 11/11): GPT-2

### 11: [Final Project Proposals](final)

- Session 1 (M 11/16): Project Proposals 1
- Session 2 (W 11/18): Project Proposals 2

### 12: [Final Project Development](final)

- Session 1 (M 11/23): Individual Meetings
- Session 2 (M 11/25): Individual Meetings

### 13: [Final Project Development](final)

- Session 1 (M 11/30): TBD 1
- Session 2 (W 12/2): TBD 2

### 14: Final Project Presentations

- Session 1 (M 12/7): Group 1
- Session 2 (W 12/9): Group 2

## CODE OF CONDUCT

Please read and review the [ITP/IMA Code of Conduct](https://github.com/ITPNYU/ITP-IMA-Code-of-Conduct/blob/main/README.md). The Code of Conduct will be reviewed and discussed as part of the course introduction.

The ITP/IMA Code of Conduct is an evolving work-in-progress document that establishes and communicates the commitment of the ITP/IMA community to uphold a key set of standards and obligations that aim to make ITP/IMA an inclusive and welcoming environment.

## COURSE DESCRIPTION

An introductory course designed to provide students with hands-on experience developing creative coding projects with machine learning. The history, theory, and application of machine learning algorithms and related datasets are explored in a laboratory context of experimentation and discussion. Examples and exercises will be demonstrated in JavaScript using the p5.js, ml5.js, and TensorFlow.js libraries. In addition, students will learn to work with open source pre-trained models in the cloud using Runway. Principles of data collection and ethics are introduced. Weekly assignments, team and independent projects, and project reports are required.

## COURSE OBJECTIVES

At the completion of this course, the student will:

- Develop an intuition for and high level understanding of core machine learning concepts and algorithms, including supervised learning, unsupervised learning, reinforcement learning, transfer learning, classification, and regression.
- Be able to apply machine learning algorithms to real-time interaction in media art projects using pre-trained models and “transfer learning” in JavaScript and related tools.
- Learn how to collect a custom dataset to train a machine learning model and
- Develop a vocabulary for critical discussions around the social impact and ethics of data collection and application of machine learning algorithms.
- Become familiar with the current landscape of new media art generated from machine learning algorithms. Understand how to use a machine learning model to generate media: words, sound, and images.

## EQUIPMENT

You will need a modern laptop (4 years old or younger is a good rule of thumb). Most required software is freely available. The department has all required commercial software installed on laptops available for checkout.

## COURSE TEXTS

There is no textbook for the class. Readings and videos will be assigned on the individual session notes pages.

## GRADING AND ATTENDANCE

Grades for the course will follow the standard A through F letter grading system and will be determined by the following breakdown:

- 25% Participation
- 50% Assignments (including reading responses and other written work)
- 25% Final project

At most two (2) unexcused absences will be tolerated without effect to your grade. Any more than two (2) unexcused absences will result a lowering of your final grade by one whole grade for each unexcused absence. For example, three (3) unexcused absences will result in your highest possible grade being a B instead of an A. Four (4) unexcused absences will result in your highest possible grade being a C and so on. Six (6) unexcused absences would result in an automatic F for the course. Two (2) late arrivals will count for one (1) absence.

## PARTICIPATION:

This class will be highly participatory. You are expected to contribute to discussions, engage in group work, give feedback to your peers, and otherwise fully participate in class.

## TEACHING STYLE

## COURSE SCHEDULE

The course will be two (2) times per week for one hour and thirty minutes (1:30) for a total of 14 weeks.

## ASSIGNMENTS

There will be regular assignments that are relevant the class material. These assignments must be documented (written description, photos, screenshots, screen recording, code, and video all qualify based on the assignment) on a web platform such as a blog or google doc. You are required to link to your assignment from the course repo (you may choose to use a privately shared google doc or password protected website if you prefer.) The due dates are specified on the assignment page.

It is expected that you will spend 6 to 8 hours a week on the class outside of class itself. This will include reviewing material, reading, watching video, completing assignments and so on. Please budget your time accordingly.

Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are simply not attempted or the assignment is turned in late (up to 1 week) then it may be marked partially complete. If it is more than a week late, not turned in, or an attempt isn’t made to meet the criteria specified it will be marked incomplete.

Responses to reading and other written assignments are also due in class one week after they are assigned and must also be submitted via the class website. Written assignments are expected to be 200 to 500 words in length unless otherwise specified. Grading will follow the same guidelines as above; on time and meeting the criteria specified will be marked as complete. Late (up to 1 week) or partially completed work will be given half credit. Work that is more than a week late, not turned in, or fails to meet the criteria specified will be given no credit.

## Readings

- *Artificial Intelligence, A Guide for Thinking Humans,* Melanie Mitchell, 2019
- [Jason's Machine Learning 101](https://docs.google.com/presentation/d/1kSuQyW5DTnkVaZEjGYCkfOxvzCqGEFzWBy4e9Uedd9k/edit#slide=id.g22aaaf9c33_0_23)

## LEARNING OBJECTIVES

* Develop an intuition for and high level understanding of core machine learning concepts and algorithms, including supervised learning, unsupervised learning, reinforcement learning, transfer learning, classification, and regression.
* Be able to apply machine learning algorithms to real-time interaction in media art projects using pre-trained models and “transfer learning” in JavaScript and related tools.
* Learn how to collect a custom dataset to train a machine learning model and
* Understand how to use a machine learning model to generate media: words, sound, and images.

## VIDEO PLAYLIST

[Beginner's Guide to Machine Learning in JavaScript with ml5.js](https://youtu.be/26uABexmOX4)
Dan Shiffman's playlist provides an introduction to developing creative coding projects with machine learning. The theory and application of machine learning algorithms is demonstrated in JavaScript using the p5.js and ml5.js libraries.

[Grant Sanderson](https://www.3blue1brown.com/lessons/neural-networks)
[source code](https://github.com/3b1b/videos/blob/master/_2017/nn/part1.py)

## Dan Shiffman, Coding Train, NYU

1. [ml5.js: KNN Classification Part 1](https://youtu.be/KTNqXwkLuM4)

1. [ml5.js: KNN Classification Part 2](https://youtu.be/Mwo5_bUVhlA)

1. [ml5.js: KNN Classification Part 3](https://youtu.be/JWsKay58Z2g)

1. [ml5.js: Train Your Own Neural Network](https://youtu.be/8HEgeAbYphA)

1. [ml5.js: Save Neural Network Training Data](https://youtu.be/q6cwxORPDo8)

1. [ml5: Save Neural Network Trained Model](https://youtu.be/wUrg9Hjkhg0)

1. [ml5: Neural Network Regression](https://youtu.be/fFzvwdkzr_c)

1. [ml5.js: Sound Classification](https://youtu.be/cO4UP2dX944)

1. [Teachable Machine 3: Sound Classifiication](https://youtu.be/TOrVsLklltM)

1. [Coding Challenge #151: Ukulele Tuner with Machine Learning Pitch Detection Model](https://youtu.be/F1OkDTUkKFo)

1. [ml5.js Pose Estimation with PoseNet](https://youtu.be/OIo-DIOkNVg)

1. [ml5.js: Pose Classification with PoseNet and ml5.neuralNetwork()](https://youtu.be/FYgYyq-xqAw)

1. [ml5.js: Pose Regression with PoseNet and ml5.neuralNetwork()](https://youtu.be/lob74HqHYJ0)

1. [ml5.js: Train a Neural Network with Pixels as Input](https://youtu.be/UaKab6h9Z0I)

1. [ml5.js: What is a Convolutional Neural Network Part 1 - Filters](https://youtu.be/qPKsVAI_W6M)

1. [ml5.js: What is a Convolutional Neural Network Part 2 - Max Pooling](https://youtu.be/pRWq_mtuppU)

1. [ml5.js: Training a Convolutional Neural Network for Image Classification](https://youtu.be/hWurN0XhzLY)

1. [Coding Challenge #158: Shape Classifier Neural Network with ml5.js](https://youtu.be/3MqJzMvHE3E)

1. [ml5.js: Classifying Drawings with DoodleNet](https://youtu.be/ABN_DWnM5GQ)

## READINGS

- [Machine Bias - Propublica](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)
- [Halt the Use of AI // Kate Crawford](https://www.nature.com/articles/d41586-019-02514-7)
- [cars and second order consequences // Ben Evans](https://www.ben-evans.com/benedictevans/2017/3/20/cars-and-second-order-consequences)
- [Perception Engines // Tom White](https://medium.com/artists-and-machine-intelligence/perception-engines-8a46bc598d57)

## PRESENTATIONS

- [Joey Lee, Aerial Bold](https://docs.google.com/presentation/d/1alYYj7clI37aKu7_TYaKxnPpYFX3-dRt6dA9cmXK1MU/edit#slide=id.g6b14100f50_1_73)

## PROJECTS USING ML

https://stamen.com/work/penny/ - Penny // Stamen
http://www.terrapattern.com/ - Terrapattern // Levin et al.
https://www.move-lab.com/project/opendatacam/ - OpenDataCam // Move Lab
http://type.aerial-bold.com/tw/ and https://www.kickstarter.com/projects/357538735/aerial-bold-kickstart-the-planetary-search-for-let // Aerial Bold - Groß & Lee
http://lauren-mccarthy.com/pplkpr // pplkpr - McCarthy & MacDonald
http://lauren-mccarthy.com/us - us+ // McCarthy & MacDonald
https://www.xujenna.com/ITP/allegoryOf/index.php and https://www.xujenna.com/ITP/NYQuotient/index.php // Jenna Xu
https://philippschmitt.com/ // Philipp Schmitt
https://www.fluate.net/en/travaux/vectoglyph // vectoglyph - Nicolas Boillot
https://kimalbrecht.com/vis_geisterstunde/#artificial-senses // Kim Albrecht
https://mayaontheinter.net/billtjones/ // Maya Man
https://pair.withgoogle.com/ // People + AI Guidebook // Google PAIR
https://pair-code.github.io/what-if-tool/ - What If Tool // Google PAIR
https://twitter.com/c_valenzuelab/status/979131716907536384 // Sidewalk orchestra // Cris Valenzuela
https://yossarian.co/ // Youssarian - J.Paul Neeley
http://cognimates.me/home/ // Cognimates // Stefania Druga + MIT
http://mimionuoha.com/a-peoples-guide-to-ai // People's guide to AI // Mimi O

MS_038_F21

MS_038_F21 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

MS_038_F21

MS_038_F21 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README