COMPSCI 697L Deep Learning

Important Dates

Course project proposal: 11/02 (due 11:55pm on Moodle).
Course project milestone: 11/23 (due 11:55pm on Moodle).
Final course project write-up: 12/14 (due 11:55pm on Moodle).

Overview

The Course Project is an opportunity for you to apply what you have learned in class to a problem of your interest. There are two project options you can pick from:

Your are encouraged to select a topic and work on your own project. Potential projects usually fall into these two tracks:

• Applications. If you're coming to the class with a specific background and interests (e.g. biology, engineering, physics), we'd love to see you apply deep neural networks to problems related to your particular domain of interest. Pick a real-world problem and apply deep neural networks to solve it.
• Models. You can build a new model (algorithm) with deep neural networks, or a new variant of existing models, and apply it to tackle vision tasks. This track might be more challenging, and sometimes leads to a piece of publishable work.

Here you can find some sample project ideas professor described earlier in the class:

• Sample project ideas (Google Docs)

To inspire ideas, you might look at recent deep learning publications from top-tier vision conferences, as well as other resources below.

• Awesome Deep Vision
• CVPR: IEEE Conference on Computer Vision and Pattern Recognition
• ICCV: International Conference on Computer Vision
• ECCV: European Conference on Computer Vision
• NIPS: Neural Information Processing Systems
• ICLR: International Conference on Learning Representations
• Kaggle challenges: An online machine learning competition website. For example, a Yelp classification challenge.

For applications, this type of projects would involve careful data preparation, an appropriate loss function, details of training and cross-validation and good test set evaluations and model comparisons. Don't be afraid to think outside of the box. Some successful examples can be found below:

• Teaching Deep Convolutional Neural Networks to Play Go
• Playing Atari with Deep Reinforcement Learning
• Winning the Galaxy Challenge with convnets

Deep neural networks also run in real time on mobile phones and Raspberry Pi's - feel free to go the embedded way. You may find this TensorFlow demo on Android helpful.

For models, deep neural networks have been successfully used in a variety of computer vision and NLP tasks. This type of projects would involve understanding the state-of-the-art vision or NLP models, and building new models or improving existing models. The list below presents some papers on recent advances of deep neural networks in the computer vision community.

• Object recognition: [Krizhevsky et al.], [Russakovsky et al.], [Szegedy et al.], [Simonyan et al.], [He et al.]
• Object detection: [Girshick et al.], [Sermanet et al.], [Erhan et al.]
• Image segmentation: [Long et al.]
• Video classification: [Karpathy et al.], [Simonyan and Zisserman]
• Scene classification: [Zhou et al.]
• Face recognition: [Taigman et al.]
• Depth estimation: [Eigen et al.]
• Image-to-sentence generation: [Karpathy and Fei-Fei], [Donahue et al.], [Vinyals et al.]
• Visualization and optimization: [Szegedy et al.], [Nguyen et al.], [Zeiler and Fergus], [Goodfellow et al.], [Schaul et al.]

We also provide a list of popular computer vision datasets:

• Meta Pointer: A large collection organized by CV Datasets.
• Yet another Meta pointer
• ImageNet: a large-scale image dataset for visual recognition organized by WordNet hierarchy
• SUN Database: a benchmark for scene recognition and object detection with annotated scene categories and segmented objects
• Places Database: a scene-centric database with 205 scene categories and 2.5 millions of labelled images
• NYU Depth Dataset v2: a RGB-D dataset of segmented indoor scenes
• Microsoft COCO: a new benchmark for image recognition, segmentation and captioning
• Flickr100M: 100 million creative commons Flickr images
• Labeled Faces in the Wild: a dataset of 13,000 labeled face photographs
• Human Pose Dataset: a benchmark for articulated human pose estimation
• YouTube Faces DB: a face video dataset for unconstrained face recognition in videos
• UCF101: an action recognition data set of realistic action videos with 101 action categories
• HMDB-51: a large human motion dataset of 51 action classes

If you are unable to come up with a project idea, you can fall back to working on the Tiny ImageNet Challenge which we will run similar to the ImageNet challenge. The goal of the challenge will be for you to do as well as possible on the Image Classification problem.

• Tiny ImageNet dataset (237 MB)

Tiny Imagenet has 200 classes. Each class has 500 training images, 50 validation images, and 50 test images. We have released the training and validation sets with images and annotations. We provide both class labels and bounding boxes as annotations; however, you are asked only to predict the class label of each image without localizing the objects. The test set is released without labels.

We use test set error rate, the fraction of test images that are incorrectly classified by the model, to measure the performance. To submit your predictions on the test set, name your submission file as .txt and upload it to Moodle. Your submission should be a two-column file with 10,000 lines. Each line contains a pair of test image filename and its predicted class id.

This file illustrates a submission of random guessing, giving us a chance accuracy 0.005 (1/200). Note that, the class ids correspond to synsets in ImageNet. For example, you can browse images and metadata of class id n01910747 using this link.

Grading Policy

  Final Project: 40%
  milestone: 5%
  write-up: 10%
   •  clarity, structure, language, references: 3%
   •  background literature survey, good understanding of the problem: 3%
   •  good insights and discussions of methodology, analysis, results, etc.: 4%
  technical: 12%
   •  correctness: 4%
   •  depth: 4%
   •  innovation: 4%
  evaluation and results: 10%
   •  sound evaluation metric: 3%
   •  thoroughness in analysis and experimentation: 3%
   •  results and performance: 4%
  poster: 3% (+2% bonus for best few posters)

Project Proposal

The project proposal should be one paragraph (200-400 words). If you work on your own project, your proposal should contain:

• Who are the (1~3) group members? What will each person do? (This need to be a separate detailed paragraph)
• What is the problem that you will be investigating? Why is it interesting?
• What data will you use? If you are collecting new datasets, how do you plan to collect them?
• What method or algorithm are you proposing? If there are existing implementations, will you use them and how? How do you plan to improve or modify such implementations?
• What reading will you examine to provide context and background?
• How will you evaluate your results? Qualitatively, what kind of results do you expect (e.g. plots or figures)? Quantitatively, what kind of analysis will you use to evaluate and/or compare your results (e.g. what performance metrics or statistical tests)?

If you choose to work on Tiny ImageNet Challenge, emphasize the last three bullet points on the list above. Each group should submit a plain-text proposal to Moodel. If your proposed project is joint with another class' project (with the consent of the other class' instructor), make this clear in the proposal.

Project Milestone

Your project milestone report should be between 2 - 3 pages using the provided template. The following is a suggested structure for your report:

• Title, Author(s)
• Introduction: this section introduces your problem, and the overall plan for approaching your problem
• Problem statement: Describe your problem precisely specifying the dataset to be used, expected results and evaluation
• Technical Approach: Describe the methods you intend to apply to solve the given problem
• Intermediate/Preliminary Results: State and evaluate your results upto the milestone

Submission: Please upload a PDF file named <your ID>_milestone.pdf to Moodle. One submission for each group is sufficient.

Final Submission

Your final write-up should be between 4 - 8 pages using the provided template. After the class, we will post all the final reports online so that you can read about each others' work. If you do not want your writeup to be posted online, then please let us know at least a week in advance of the final writeup submission deadline.

Submit your final submission through Moodle. You will submit one or two files:

1. A pdf file of your final report
2. (OPTIONAL) zip file (or pdf file) with Supplementary Materials

Report. The following is a suggested structure for the report:

• Title, Author(s)
• Abstract: It should not be more than 300 words
• Introduction: this section introduces your problem, and the overall plan for approaching your problem
• Background/Related Work: This section discusses relevant literature for your project
• Approach: This section details the framework of your project. Be specific, which means you might want to include equations, figures, plots, etc
• Experiment: This section begins with what kind of experiments you're doing, what kind of dataset(s) you're using, and what is the way you measure or evaluate your results. It then shows in details the results of your experiments. By details, we mean both quantitative evaluations (show numbers, figures, tables, etc) as well as qualitative results (show images, example results, etc).
• Conclusion: What have you learned? Suggest future ideas.
• References: This is absolutely necessary.

Supplementary Material is not counted toward your 4-8 page limit.
Examples of things to put in your supplementary material:

• Source code (if your project proposed an algorithm, or code that is relevant and important for your project.).
• Cool videos, interactive visualizations, demos, etc.

Examples of things to not put in your supplementary material:

• All of Caffe source code.
• Various ordinary data preprocessing scripts.
• Any code that is larger than 1MB.
• Model checkpoints.
• A computer virus.

Example Project Reports

Stanford CS231n 2015 (Winter) projects.

Computing Resources

Amazon offers free AWS credits for students (annually renewable). Using your link provided by the GitHub Student Developer Pack will get you the most free credits.

• UMass is an "AWS member institution", so you are in the higher allowance tier. Use your .edu email and the full school name "University of Massachusetts Amherst" when you register to get the full benefits (a total of $75-$150).
• To get GPUs, use g2 (up to 4 NVIDIA GRID K520 GPUs) or p2 (up to 16 NVIDIA K80 GPUs) instances in EC2. Check the pricing first and make your plan accordingly!

Collaboration Policy

You can work in teams of 1~3 people. We do expect that projects done with 3 people have more impressive writeup and results than projects done with 2 people.

Honor Code

You may consult any papers, books, online references, or publicly available implementations for ideas and code that you may want to incorporate into your strategy or algorithm, so long as you clearly cite your sources in your code and your writeup. However, under no circumstances may you look at another group’s code or incorporate their code into your project.

If you are doing a similar project for another class, you must make this clear and write down the exact portion of the project that is being counted for COMPSCI 697L.