TS-LSTM-and-Temporal-Inception

KO | EN

gitlite — search

#python #typescript #golang #go #rust #linux #cli #kubernetes #awesome-list #vue #security #pytorch #mcp #ios

TS-LSTM-and-Temporal-Inception

(★ 9)

Code for TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

TS-LSTM-and-Temporal-Inception Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Activity Recognition with RNN and Temporal-ConvNet
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

[Chih-Yao Ma](http://shallowdown.wixsite.com/chih-yao-ma), [Min-Hung Chen](https://www.linkedin.com/in/chensteven) (equal contribution)

Codes for the paper: 
[TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition](http://arxiv.org/abs/1703.10667)

Project: 
[Activity Recognition with RNN and Temporal-ConvNet](https://ghassanalregib.squarespace.com/human-action-recognition?p)

---
## Abstract
In this work, we demonstrate a strong baseline two-stream ConvNet using ResNet-101. We use this baseline to thoroughly examine the use of both RNNs and Temporal-ConvNets for extracting spatiotemporal information. Building upon our experimental results, we then propose and investigate two different networks to further integrate spatiotemporal information: 1) temporal segment RNN and 2) Inception-style Temporal-ConvNet.

Our analysis identifies specific limitations for each method that could form the basis of future work. Our experimental results on UCF101 and HMDB51 datasets achieve state-of-the-art performances, 94.1% and 69.0%, respectively, without requiring extensive temporal augmentation.

---
#### How we tackle Activity Recognition problem?

<img src="https://github.com/chihyaoma/Activity-Recognition-with-CNN-and-RNN/blob/master/figures/overview_image.png?raw=true">

---
### Demo
The GIFs demonstrate the top-3 predictions results of our TS-LSTM and Temporal-Inception methods. The text on the top is the ground truth, three texts are the predictions for each of the method, and the bar right next to the predictions are how confident the model makes predictions.

---
## Dataset
We are currently using [UCF101](http://crcv.ucf.edu/data/UCF101.php) and [HMDB51](http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/) dataset for our project.
You can directly download the videos here:

|      | UCF101 | HMDB51 |
|:----:|:------:|:------:|
| RGB      |[link](https://www.dropbox.com/sh/fibb0k5ji33nucn/AADg9VyZJwTlIYntHDNGi4QWa?dl=0)|[link](https://www.dropbox.com/sh/kma8z2pcy69pwmo/AAB2B4DmLz6fb0O78rOlFHDDa?dl=0)|
| TV-L1       |[link](https://www.dropbox.com/sh/1drtfrzmn9sajv2/AAD1h73Emkdt1vlbqPZk59Z8a?dl=0)|[link](https://www.dropbox.com/sh/1glhoe7fg3eyil3/AADVD7PRsUSfDbs4WGre3u9la?dl=0)|

---
#### Prerequisites
* Linux (tested on Ubuntu 14.04)
* [Torch](http://torch.ch/docs/getting-started.html#_)
* [CUDA](https://developer.nvidia.com/cuda-downloads) and [cuDNN](https://developer.nvidia.com/cudnn)
* NVIDIA GPU is strongly recommended
* [torch-pastalog](https://github.com/Kaixhin/torch-pastalog) to visualize training status

---
## Usage
We proposed two different methods to train the models for activity recognition: TS-LSTM and Temporal-Inception.

#### Inputs
Our models takes the **feature vectors** generated by the first stage two-stream ConvNet as input for training. You can generate the features using our codes under "/CNN-Pred-Feat/". You can also download the feature vectors generated by us. (please refer to the Dropbox link below.) We followed the training/testing splits from [UCF101](http://crcv.ucf.edu/data/UCF101.php) and [HMDB51](http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/). If you would like to compare with our results, please use the same training and testing list, as it will affect your overall performance a lot.

* Features for training:

|                 | UCF101          | HMDB51      |
|:-------------:|:-------------:|:---------:|
| RGB      | [sp1](https://www.dropbox.com/s/lxz10lijnh1gb6i/data_feat_train_RGB_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/0pqodpr7btj93m5/data_feat_train_RGB_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/nnljsen7xwxbfm1/data_feat_train_RGB_centerCrop_25f_sp3.t7?dl=0) | [sp1](https://www.dropbox.com/s/qvko5we5ccr16a3/data_feat_train_RGB_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/xna1c9travtgp7p/data_feat_train_RGB_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/nb5rpnc47rs6eia/data_feat_train_RGB_centerCrop_25f_sp3.t7?dl=0) |
| TV-L1       | [sp1](https://www.dropbox.com/s/fug14kobliewgb2/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/ju1v4bymtwbgrdp/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/343oko45z0xauz7/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp3.t7?dl=0)      |  [sp1](https://www.dropbox.com/s/nnzru9rkwaozw32/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/vk53sr16x3mo6pz/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/zt09zfnw2q97qop/data_feat_train_FlowMap-TVL1-crop20_centerCrop_25f_sp3.t7?dl=0)  |

* Features for testing:

|                 | UCF101          | HMDB51      |
|:-------------:|:-------------:|:---------:|
| RGB      | [sp1](https://www.dropbox.com/s/x5slrzhos937bnv/data_feat_test_RGB_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/83hmoaezad8j8mj/data_feat_test_RGB_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/t0lqrehejzjomqu/data_feat_test_RGB_centerCrop_25f_sp3.t7?dl=0) | [sp1](https://www.dropbox.com/s/7vxs1n7id9x98jt/data_feat_test_RGB_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/x630x0x0mf1s1fx/data_feat_test_RGB_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/ot65vzgvz9f37dy/data_feat_test_RGB_centerCrop_25f_sp3.t7?dl=0) |
| TV-L1       | [sp1](https://www.dropbox.com/s/p48731hdg8m0fjo/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/9w0250idyysy99d/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/7xx5aqu0j79y4qt/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp3.t7?dl=0)      | [sp1](https://www.dropbox.com/s/zj4neexynd1lt0g/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp1.t7?dl=0) [sp2](https://www.dropbox.com/s/fvfp1943ctuq6ya/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp2.t7?dl=0) [sp3](https://www.dropbox.com/s/egfwm4rbdqay46q/data_feat_test_FlowMap-TVL1-crop20_centerCrop_25f_sp3.t7?dl=0)   |

#### Train with RNN
We use the [RNN library](https://github.com/Element-Research/rnn) provided by Element-Research. Simply install it by:
```bash
$ luarocks install rnn
```
After you downloaded the feature vectors, please modify the code in *./RNN/data-ucf101.lua* to the director where you put your feature vector files.

To start the training process, go to *./RNN* and simply execute:
```bash
$ th main.lua -pastalogName 'model_RNN' -nGPU 1 -dataset 'ucf101' -split '1' -fcSize '{0}' -hiddenSize '{512}' -lstm -spatFeatDir '<path/to/feature/>' -tempFeatDir '<path/to/feature/>'
```
The training and testing loss will be reported, and the results will be saved into log files. The learning rate and best testing accuracy will be reported each epoch if there is any update.

#### Train with Temporal-ConvNet
To start the training process, go to *./Temporal-ConvNet* and simply execute:
```bash
$ th run.lua -o <output_folder_name> --dataset <dataset-name>
```
For more details and hyper-parameter tuning, please refer to the readme file in the folder *./Temporal-ConvNet/*.

You also need to modify the code in *./Temporal-ConvNet/data-2Stream.lua* to the director where you put your feature vector files.

The training and testing performance will be plotted, and the results will be saved into log files. The best testing accuracy will be reported each epoch if there is any update.

---
## Can I train with frame-level features?
To standardize the comparison, the above features are equally sampled across each video. If you would like to train with frame-level features extracted at 25fps for all videos in UCF101. Please refer to [Temporal Augmentation using frame-level features with RNN](https://github.com/chihyaoma/temporal-augmentation).

---
#### Citation
```
@article{ma2018ts,
  title={TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition},
  author={Ma, Chih-Yao and Chen, Min-Hung and Kira, Zsolt and AlRegib, Ghassan},
  journal={Signal Processing: Image Communication},
  year={2018},
  publisher={Elsevier}
}
```

---
## Acknowledgment
This work was initialized as a class project for deep learning class in Georgia Tech 2016 Spring. We were teamed up with Hao Yan and Casey Battaglino to work on this class project, who have been a great help and provide valuable discussions as we go long this class project.

#### Please contact us if you have any questions.

[Chih-Yao Ma](http://shallowdown.wix.com/chih-yao-ma/me) at <cyma@gatech.edu> or [[LinkedIn]](https://www.linkedin.com/in/chih-yao-ma-9b5b3063)
 
[Min-Hung Chen](https://www.linkedin.com/in/chensteven) at <cmhungsteve@gatech.edu>

TS-LSTM-and-Temporal-Inception

TS-LSTM-and-Temporal-Inception Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

TS-LSTM-and-Temporal-Inception

TS-LSTM-and-Temporal-Inception Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README