CS 4641\7641 A: Machine Learning (Spring 2021)

Course Information

• Lecture: Tuesdays and Thursdays, 9:30am-10:40am
• Q/A for Lecture: Tuesdays and Thursdays, 5:00pm-6:10pm - Offline Questions
• Location: BlueJeans
• Piazza: http://piazza.com/gatech/spring2021/mlspring2021aasynch/home

Course Overview

This course introduces techniques in machine learning with an emphasis on algorithms and their applications to real-world data. We will investigate the following question: how to computationally extract useful knowledge from data for decision making and task support? We will focus on machine learning methods, which are organized into three parts:

1. Basic math for data science and machine learning

   • Linear algebra
   • Probability and statistics
   • Information theory
   • Optimization

2. Unsupervised machine learning for data exploration

   • Clustering analysis
   • Dimensionality reduction
   • Kernel density estimation

3. Supervised learning for predictive data analysis

   • Tree-based models
   • Support vector machines
   • Linear classification and regression
   • Neural networks

Prerequisites for this course include (1) basic knowledge of probability, statistics, and linear algebra; (2) Basic programming experience in Python.

In addition to the technical content, this class includes the following learning objectives:

• Structuring a task into a machine learning work flow
• Collaborating effectively on team projects in a remote environment
• Conducting peer evaluation in a constructive format
• Communicating technical content in a concise and effective manner

Schedule

For all dates used in this course, their times are 23:59 Anywhere on Earth (11:59 pm AoE). For example, a due date of "January 8" is the same as "January 8, 23:59pm AoE". Convert the times to your local times using a Time Zone Converter.

Date	Class	Assignments	Project	Quizzes	Readings
Thu, Jan 14	*Course overview; Refer to Lecture1 video on Canvas Refer to Lecture1-QA video on Canvas	Piazza signup			GT Honor Code
Mon, Jan 18
Tue Jan 19	*Data analysis toolbox; *Broadcasting toolbox; *Modified in lecture-Data analysis toolbox; *Modified in lecture-Broadcasting toolbox; *QA notes; Project information; Refer to Lecture2 video on Canvas Refer to Lecture2-QA video on Canvas				Undergrad Canvas Access for previous ML projects; Grad Canvas Access for previous ML projects; Heilmeier catechism; Visual Information Theory by Chris Olah; GitHub Pages; YAML Configuration; NumPy Tutorial; Matplotlib Tutorial; Project Examples; seaborn: statistical data visualization; Overleaf for GT students;
Wed, Jan 20				Q0: Introduction,pre-test
Thu, Jan 21	*Foundations: Linear algebra; *Class notes; *QA notes; Refer to Lecture3 video on Canvas Refer to Lecture3-QA video on Canvas				Correlation vs Covariance Linear Algebra Review by Zico Kolter
Mon, Jan 25				Q1: Data analysis toolbox and Linear algebra
Tue, Jan 26	*Foundations: Probability and statistics; *Class Notes; *QA Notes; Refer to Lecture4 video on Canvas Refer to Lecture4-QA video on Canvas	A1 out			Probability Theory Review by Andrew Moore; The Differences Between Data, Information and Knowledge;
Wed, Jan 27			Project Seminar 1 by Ruijia and Kevin [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Jan 28	*Foundations: Information Theory; *Class Notes; *QA Notes; Refer to Lecture5 video on Canvas Refer to Lecture5-QA video on Canvas				The Differences Between Data, Information and Knowledge; More about Cross Entropy and KLD; Cross Entropy as loss function;
Mon, Feb 1				Q2: Probability and Information Theory
Tue, Feb 2	*Foundations: Optimization; *QA Notes; Refer to Lecture6 video on Canvas Refer to Lecture6-QA video on Canvas				KKT for inequality constrained optimization; Why Cross Entropy over MSE for Classification;
Wed, Feb 3			Project seminar 2 by Rohit and Dheeraj [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Feb 4	*Data analysis toolbox - part 2; Data analysis toolbox QA Notes; Refer to Lecture7 video on Canvas Refer to Lecture7-QA video on Canvas		Project team composition due		NumPy Tutorial; Matplotlib Tutorial;
Mon, Feb 8				Q3: Optimization and Data Analysis toolbox
Tue, Feb 9	*Clustering Analysis and K-Means; *Class Notes; *QA Notes; Refer to Lecture8 video on Canvas Refer to Lecture8-QA video on Canvas				Curse of dimensionality (Euclidean space example); Jupyter Notbook (Kmeans and DBSCAN);
Wed, Feb 10			Project seminar 3 by Shreeshaa and Rusty [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Feb 11	*Gaussian Mixture Model; *Class Notes; *QA Notes; Refer to Lecture9-part1 video on Canvas Refer to Lecture9-part1-QA video on Canvas				GitHub Student Application;
Mon, Feb 15				Q4: Clustering Analysis, Kmeans and GMM
Tue, Feb 16	*Gaussian Mixture Model; *Class Notes; *QA Notes; Refer to Lecture9-part2 video on Canvas Refer to Lecture9-part2-QA video on Canvas	A1 due | A2 out
Web, Feb 17			Project seminar 4 by Shalini [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Feb 18	*Hierarchical Clustering; *Hierarchical Class Notes; *QA Notes; Refer to Lecture10 video on Canvas Refer to Lecture10-QA video on Canvas				Understanding the concept of Hierarchical clustering Technique; Dendrogram Visualization ;
Mon, Feb 22				Q5: GMM and Hierarchical
Tue, Feb 23	*Density-Based Clustering; *Class Notes; *QA Notes; Refer to Lecture11 video on Canvas Refer to Lecture11-QA video on Canvas				Jupyter Notbook (Kmeans and DBSCAN);
Wed, Feb 24			Project proposal forum to discuss your project topic with the instructional Team before proposal submission [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Feb 25	*Evaluation of Clustering Algorithms; *Class Notes; *QA Notes (no notes for this lecture); Refer to Lecture12 video on Canvas Refer to Lecture12-QA video on Canvas
Mon, Mar 1				Q6: DBSCAN and Cluster evaluation
Tue, Mar 2	*Density Estimation *Class Notes; *Class QA; Refer to Lecture13 video on Canvas Refer to Lecture13-QA video on Canvas				KDE interactive visualization ; KDE sampling ; KDE SKLearn and sampling ; Jupyter Notebook Kernel Density Example;
Wed, Mar 3			Project proposal due Project seminar 5 by Daniel [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Mar 4	*Dimensionality reduction; *Class Notes; *QA Notes; Refer to Lecture14 video on Canvas Refer to Lecture14-QA video on Canvas				Image reconstruction using PCA ; Feature extraction using PCA ; PCA for images ; PCA as linear combination of features ; PCA and Linear Discriminant Analysis ;
Mon, Mar 8				Q7: Density estimation and dimensionality reduction
Tue, Mar 9	*Linear Regression; *Class Notes; *QA Notes; Refer to Lecture15 video on Canvas Refer to Lecture15-QA video on Canvas	A2 due | A3 out			Simple Linear Regression in Matrix Format; Adding Noise to Regression Predictors;
Wed, Mar 10
Thu, Mar 11	*Linear Regression [Contd]; *Regularization and Linear Regression; *Class Notes; *QA Notes (No QA Notes); Refer to Lecture16 video on Canvas Refer to Lecture16-QA video on Canvas (No QA Video)
Mon, Mar 15
Tue, Mar 16	No assignments or classes or assessments
Wed, Mar 17
Thu, Mar 18	*Regularization and Linear Regression [Contd]; *QA Notes; Refer to Lecture17 video on Canvas Refer to Lecture17-QA video on Canvas			Q8: Linear Regression
Fri, Mar 19			Project seminar 6 by James [Undergrad Canvas Access] [Grad Canvas Access]
Mon, Mar 22				Q9: Regularization and Linear Regression
Tue, Mar 23	*Naïve Bayes and Logistic Regression; *Class Notes; *QA Notes; Refer to Lecture18 video on Canvas Refer to Lecture18-QA video on Canvas
Wed, Mar 24
Thu, Mar 25	*Naïve Bayes and Logistic Regression; *QA Notes; Refer to Lecture19 video on Canvas Refer to Lecture19-QA video on Canvas
Mon, Mar 29				Q10: Naive Bayes and Logistic Regression
Tue, Mar 30	*Neural Networks(Forward pass and Back propagation); *Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture20 video on Canvas Refer to Lecture20-QA video on Canvas				NN Playground ; Interactive NN initialization ; The role of a hidden layer; Back propagation numerical example; More detailed introduction;
Wed, Mar 31			Project seminar 7 by Kunal and Yuzi [Undergrad Canvas Access] [Grad Canvas Access]
Thu, Apr 1	*Neural Networks and Deep learning (CNN); *Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture21 video on Canvas Refer to Lecture21-QA video on Canvas	A3 due | A4 out			CNN Live Demo; A guide to an efficient way to build CNN and optimize its hyper-parameters; Back Propagation in CNN; Transfer learning in CNN; Project Scoring Guidance;
Mon, Apr 5				Q11: NN and CNN
Tue, Apr 6	*Decision Tree; *Decision Tree Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture22 video on Canvas Refer to Lecture22-QA video on Canvas
Wed, Apr 7
Thu, Apr 8	*Decision Tree; *Decision Tree Class Notes; *Ensemble Learning and Random Forest; *Ensemble Learning Class Notes; *Ensemble Learning QA [No note, watch the QA video]; Refer to Lecture23 video on Canvas Refer to Lecture23-QA video on Canvas		Project midpoint report
Mon, Apr 12				Q12: Decision Tree, Ensemble Learning and Random Forest
Tue, Apr 13	*Support Vector Machine; *Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture24 video on Canvas Refer to Lecture24-QA video on Canvas				KKT and SVM;
Wed, Apr 14
Thu, Apr 15	*Support Vector Machine Class Notes; *Kernel Method \ SVM; *Kernel Method \ SVM Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture25 video on Canvas Refer to Lecture25-QA video on Canvas
Mon, Apr 19				Q13: SVM
Tue, Apr 20	*Kernel Method \ SVM Class Notes; *QA Notes [No note, watch the QA video]; Refer to Lecture26 video on Canvas Refer to Lecture26-QA video on Canvas
Wed, Apr 21
Thu, Apr 22	*Ethics in ML; Class Notes [No note, watch the QA video]; Refer to Lecture27 vsideo on Canvas Refer to Lecture27-QA video on Canvas	A4 due
Mon, Apr 26				Q14: Kernel Method
Tue, Apr 27	*ML Course review; Refer to Lecture28 video on Canvas Refer to Lecture28-QA video on Canvas		Final project due [Undergrad Canvas Access for Projects] [Grad Canvas Access for Projects]

Course policies

• Attendance: Our class will be offered in an Asynch mode. We will not meet in person and every lecture will be live-streamed and recorded. The recordings will be made available to all students after class time. My on-campus class was mandatory and it helped me and my students A LOT to work with each other for a better environment to facilitate learning. The class is in Asynch mode. Therefore, attendance is not required for this class, but I highly recommend students to come to the live-streaming lectures. Trust me it will be fun and you will give me a lot of energy to teach better. The fact that you need to listen to the lectures without fast-forwarding me can help you to learn the materials much better and you will have the chance to ask questions if you are confused anywhere in the lectures.
• Class deliverables: All class deliverables will be handled via Gradescope excep quizzes which will be on Canvas. The time span offered to complete the course objectives is plentiful and deadlines will not be extended under any circumstances. To ensure the class is fair for all students, you will receive zero credit for work submitted after the deadline. Regrade requests should be submitted directly on Gradescope within one week of grade publication. Should you find yourself in an impasse with the TA responsible for your grading, feel free to contact the head TA or course instructor.
• Piazza:
  • Piazza will be the main and only place for the course discussions and announcements. If you have questions, please ask it on Piazza first because 1) other students may have the same question; 2) you will get help much faster.
  • If it’s something you do not like to discuss publicly on Piazza, you can use private messaging on Piazza.
  • Anytime you want to send a private message to just me on Piazza, please make sure to add our HEAD TAs too in case I may miss your message.
  • Piazza GOOD questions
    • I don't understand this part of the lecture, can you explain it to me?
    • This certain part of the hw is not clear to me, would it be possible to explain that more?
    • I have a question about the project ...
    • I found an issue on the website, hw or the lectures, can you clarify ...
    • Any feedback, suggestions, ... would be greatly appreciated.
    • Usually, most of the questions are good.
  • Piazza BAD questions
    • Can you debug my code? [our team will not do that. You need to be specific about your question]
    • Can you find where the problem is in my code?
• Exceptional circumstances: Any request for exceptions to these policies should be made in advance when at all possible. Requests should be due to incapacitating illness, personal emergencies, or similarly serious events. Your request should be accompanied by a supporting letter issued by the Dean of Students.

Diversity and inclusion

Just as machine learning algorithms cannot accomplish complex tasks if trained on datasets of limited variability, our course cannot be successful without appreciating the diversity of our students. In this class we aim to create an environment where all voices are valued, respecting the diversity of gender, sexuality, age, socioeconomic status, ability, ethnicity, race, and culture. We always welcome suggestions that can help us achieve this goal. Additionally, if any of our class scheduled activities conflicts with religious events, please inform the instruction team so that we can make appropriate arrangements for you.

Students with disabilities: your access to this course is extremely important to us. The institute has policies regarding disability accommodation, which are administered through the Office of Disability Services: http://disabilityservices.gatech.edu. Please request your accommodation letter as early in the semester as possible, so that we have adequate time to arrange your approved academic accommodations.

Office hours and questions

Office hours will start on the second week of clases. Please follow the instruction on this Excel Sheet [Undergrad] [Grad] to signup for a 10-minutes slot with one of the TAs. If you require more than ten minutes, please advise the TAs. They’ll return to your BlueJeans meeting once they have completed their appointments with other students. You just need to add your name, question of interest and your BlueJeans meeting link. Please do not change the other part of the Excel Sheet [Undergrad] [Grad]. The TA meetings are designed to be one-on-one. Please do not join another student’s BlueJeans meeting. The sole exception to this policy being discussions about the project, in which your fellow team members can also join. In addition to the one-on-one meetings open office hours with the instructor will be held weekly where you can ask general questions about the topics covered in class. In-person office hours are only available by appointment and will likely be held outdoors, in line with the aforementioned Georgia Tech's and CDC guidelines with respect to preventing the spread of the coronavirus.

Grading

• Assignments (50%)

• There will be four assignments. Each one is designed to improve and test your understanding of the materials. Assignments will have both programming and written analysis components.
• You will need to submit all your assignments using Gradescope. Instructions on how to submit your code and written portions will follow with every assignment. Handwritten solutions WILL NOT BE ACCEPTED and you will not receive credit for a handwritten submission.
• You are required to use Markdown, Latex (watch the tutorial created by our own team [Undergrad Access] [Grad Access] and OverLeaf Latex Example in the Video), or a word processing software to generate your solutions to the written questions. Because handwritten solutions WILL NOT BE ACCEPTED.
• All assignments follow the “no-late” policy. Assignments received after the due date and time will receive zero credit.
• All students are expected to follow the Georgia Tech Academic Honor Code.
• You can easily export your Jupyter Notebook to a Python file and import that to your desired python IDE to debug your code for assignments.
• You are NOT allowed to share any assignment codes or answers with other students. Piazza is the best place to have discussion regarding assignments and course topics. Discussions are just for the better understanding of questions and should not directly answer the questions.
• We have 4 big assignments in total. Visit this course's Canvas site for the assignment documents. See the schedule table above for deliverable due dates. (Topics are subject to change)
  • [12.5%] HW1: Linear Algebra, Probability and Statistics, Maximum Likelihood Estimation, Optimization, Information Theory
  • [12.5%] HW2: KMeans, Expectation Maximization, Gaussian Mixture Model, Clustering Evaluation
  • HW3: Singular Value Decomposition, Principal Component Analysis, Linear Regression, Rgularization Naive Bayes
  • [12.5%] HW4: Decision Trees, Random Forest, Support Vector Machine, Neural Networks, CNN



• Project (30%)

• Proposal (5%)

• A project proposal should be written on your GitHub page. It is also a good starter to come up with the first draft of your project.
• You need to provide us the link to your GitHub page. Make sure your GitHub repository is private.
• It should be less than 500 words single spaced. References are not the part of the word count.
• A project proposal should include:
  • Introduction/Background
  • Problem definition
  • Methods
  • Potential results
  • Discussion
  • At least three references (preferably peer reviewed). You need to properly cite the references on your proposal.
• A checkpoint to make sure you are working on a proper machine learning related project.
• Your group needs to submit a presentation of your proposal. Please provide us a public link which includes a 3 minutes recorded video. I found that OBS Studio and GT subscribed Kaltura are good tools to record your screen. Please make your visuals are clearly visible in your video presentation.

• Midterm report (10%)

• A checkpoint to make sure that you have had major progress in your project. You will add information to your project Proposal and turn it into your midterm report.
• You need to provide us the link to your GitHub page. Make sure your GitHub repository is private.
• The midterm report does not have a word count limitation.
• A project midterm report is quite similar to your proposal with the exception of having actual results instead of potential ones:
  • Introduction/Background
  • Problem definition
  • Data Collection
  • Methods
  • Results
  • Discussion
• We need to see where you obtain your data and how you have done your data cleaning. Make sure to talk about the features and different feature selection approaches.
• You do not submit any video for the midterm report.

• Final report (15%)

• You need to provide us the link to your GitHub page. Make sure your GitHub repository is private.
• A final report should include:
  • Introduction/Background
  • Problem definition
  • Data Collection
  • Methods
  • Results
  • Discussion
  • Conclusions
• Your group needs to submit a presentation of your final report. Please provide us a public link which includes a 6 to 9 minutes recorded video. I found that OBS Studio and GT subscribed Kaltura are good tools to record your screen. Please make your visuals are clearly visible in your video presentation.



• General project guidance

• Your project will be graded based on the following criteria:
Was the motivation clear?
• What is the problem?
• Why is it important and why we should care?
Were the dataset and approach used effectively?
• How did you get your dataset?
• What are its characteristics (e.g. number of features, # of records, temporal or not, etc.)
• Why do you think your approach can effectively solve your problem?
• What is new in your approach?
Were the experiments, results, and conclusion satisfactory?
• How did you evaluate your approach?
• What are the results?
• How do you compare your method to other methods?
How was the presentation in general?
• Finished on time?
• Effective visualizations? (Are they relevant? Do they help you better understand the project's approaches and ideas?)
• Use of text (Succinct or verbose?)
• Undergrad students can ONLY team up with Undergrad Students, and Grad students can ONLY team up with Grad students. If you are in a Grad students team, you are required to have both unsupervised and supervised learning in your project. I highly recommend Undergrad students to use both unsupervised and supervised learning in your project. However, if you were to pick one, please go with supervised learning.
• In order for you to obtain hands-on experience applying the topics covered in this course, you are expected to complete a term project utilizing real-world data. The project will encompass both unsupervised and supervised learning.
• Each project needs to be completed in a team of five people (you will be forming your team on your own. In case you cannot find a team, we will randomly assign you a team). Team members need to clearly claim their contributions in the project report. Once your teams have been formed and you have selected a topic, you will be assigned a mentor, who will provide you with general guidance on your project. It is important to note that your team will lead the project effort: obtaining the data, researching data-driven approaches to accomplish your project goal and coordinate your own activities. The role of the mentor is solely to advise you, should you find yourself stuck and unable to make progress. We also accept a team of four, if you really cannot find the fifth team member.
• You will create a GitHub page page for your project, which you will use to publish your main deliverables. There will be three deliverables published to your GitHub: a proposal, a midterm checkpoint, and a final report.
• Seminars: To help you conduct your project successfully, We have project seminars where one or two TAs will present their research or industrial projects. Doing so, you will gain a good sense of what it is being done in both Academia and Industry. Besides that, students can ask general questions about their class project and how to improve that in each seminar. Seminars will be streamed online and recorded and they will be published on the course website. Similar to the class lectures, Please ensure that you join to these seminars and get yourself familiar with the practical and real-world application of ML. They will be held on Wednesdays from 11:00 am to 12:00 pm. Timing may change for some of the seminars. We will have Piazza post for each seminar, and its exact time. All the information will be updated for each seminar on the course webiste with a BJ Event link.
• Google colaboratory allows free access to run your Jupyter Notebook. I strongly suggest you use it for your project, especially for teams that are going to employ Deep Learning. Don't forget to take advatage of Google Cloud Platform and AWS Educate as well.



• Quizzes (14%)

• There will be 14 quizzes throughout the semester.
• We will consider your top 9 quizzes' scores. Each quiz will have 1.55% of your final score.
• All quizzes are mandatory to be taken even if they do not count toward your final grade.
• The topic of each quiz will coincide roughly with the content covered in class on that week.
• Quizzes will have a duration of seven-minutes for Undergrad students and six-minutes for Grad students. Each quiz will have five multiple choice questions. They will be available from 12:00 am AOE Monday until 11:59 pm AOE Monday.
• Quizzes measure your understanding of the topics and they will be more conceptual questions.
• Quizzes' answers will be released on Saturdays. Please do not ask any questions about a quiz that you just take on Piazza before we release the answers.
• Quizzes questions are selected randomly from our question bank, which means that students will not receive the same questions for their quiz.



• Class participation (6%)

• Piazza has statistics which give us many measurements regarding how much a student has been involved on Piazza's activities such as viewing posts, answering questions, asking questions and so on. We use this to account for your Class Participation score. At the end of the semester, we will define a minimum and maximum number of involvement considering all the students and your grade will be defined based on that.



• Bonus points (up to 8%)

• About bonus points: Bonus points will be counted to always be beneficial for your final grade. More information on bonus points for assignments will be provided as the semester progresses. If it becomes necessary to curve grades, bonus points will be applied after curving, not before.
• Undergrad and grad: You can obtain up to 5% bonus points would be answering the challenging questions we may have in some of the hws.
• Undergrad: You will notice that we have bonus points for all the hws, where grad students are required to answer those questions, but it will be optional for undergrad students. You will receive up to 3%, if you answer those questions. Note that these are different than the challenging questions. Challenging questions are bonus for both grad and undergrad.
• How does it work? For example, hw 1 may have 30 bonus points, hw 2 may have 20 bonus points and so on. If you receive all the bonus points for all your hws, we will add 5% to your final grade. If you are an undergrad and you answer all the challenging and Grad students questions, you will receive 8%.

COVID-19 Policy