Variational-AutoEncoder-for-Dialogue-Generation

KO | EN

gitlite — search

#python #react #ai #typescript #docker #nodejs #java #windows #cli #awesome-list #pytorch #security #openai #open-source

Variational-AutoEncoder-for-Dialogue-Generation

(★ 35)

Variational Auto-Encoder for Dialogue Generation Models & Experiments

Variational-AutoEncoder-for-Dialogue-Generation 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Variational Auto-Encoder for Dialogue Generation

## 1. VHRED model and Dataset

Ubuntu Dialogue Corpus contains almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. Here is its characteristics:

- Two-way (or dyadic) conversation, as opposed to multi-participant chat, preferably human-human.
- Large number of conversations; 100k - 1M is typical of datasets used for neural-network learning in other areas of AI.
- Many conversations with several turns (more than 3).
- Task-specific domain, as opposed to chatbot systems.

This link (http://www.iulianserban.com/Files/UbuntuDialogueCorpus.zip) contains a pre-processed versions of the Ubuntu Dialogue Corpus, which was used by Serban et al. (2016a) and Serban et al. (2016b) and originally developed by Lowe et al. (2015). There are three datasets: the natural language dialogues, the noun representations and the activity-entity representations. Each dataset is split into train, valid and test sets.

The task investigated by Serban et al. (2016a) and Serban et al. (2016b) is dialogue response generation. Given a dialogue context (**one or several utterances**), the dialogue model must generate an appropriate response (**a single utterance**).
The context and response pairs are provided (see "ResponseContextPairs" subdirectories).
The model responses by Serban et al. (2016a) and Serban et al. (2016b) are also provided (see "ModelPredictions" subdirectories).

*Context Examples:*
    1. anyone knows why my stock oneiric exports env var ' **unknown** I mean what is that used for ? I know of $USER but not $USERNAME . My precise install doesn't export USERNAME __eou__ __eot__ looks like it used to be exported by lightdm , but the line had the comment " // **unknown** : Is this required ?" so I guess it isn't surprising it is gone __eou__ __eot__ thanks ! How the heck did you figure that out ? __eou__ __eot__ https://bugs.launchpad.net/lightdm/+bug/864109/comments/3 __eou__ __eot__
    2. i set up my hd such that i have to type a passphrase to access it at boot . how can i remove that passwrd , and just boot up normal . i did this at install , it works fine , just tired of having reboots where i need to be at terminal to type passwd in . help ? __eou__ __eot__ backup your data , and re-install without encryption " might " be the easiest method __eou__ __eot__
    3. im trying to use ubuntu on my macbook pro retina __eou__ i read in the forums that ubuntu has a apple version now ? __eou__ __eot__ not that ive ever heard of .. normal ubutnu should work on an intel based mac . there is the PPC version also . __eou__ you want total control ? or what are you wanting exactly ? __eou__ __eot__
    4. no suggestions ? __eou__ links ? __eou__ how can i remove luks passphrase at boot . i dont want to use feature anymore ... __eou__ __eot__ you may need to create a new volume __eou__ __eot__ that leads me to the next question lol ... i dont know how to create new volumes exactly in cmdline , usually i use a gui . im just trying to access this server via usb loaded with next os im going to load , the luks pw is stopping me __eou__ __eot__ for something like that I would likely use something like a live gparted disk to avoid the conflict of editing from the disk __eou__ __eot__
    5. I just added a second usb printer but not sure what the uri should read - can anyone help with usb printers ? __eou__ __eot__ firefox localhost : 631 __eou__ __eot__ firefox ? __eou__ __eot__ yes __eou__ firefox localhost : 631 __eou__ firefox http://localhost:631 __eou__ cups has a web based interface __eou__ __eot__
    
    *Ground-truth respective responses examples:*
    1. nice thanks ! __eou__
    2. so you dont know , ok , anyone else ? __eou__ you are like , yah my mouse doesnt work , reinstall your os lolol what a joke __eou__
    3. just wondering how it runs __eou__
    4. you cant load anything via usb or cd when luks is running __eou__ it wont allow usb boot , i tried with 2 diff usb drives __eou__
    5. i was setting it up under the printer configuration __eou__ thanks ! __eou__
    
## Results

Responses generated by LSTM model using beam search with 5 beams (top 1 responses):

that 's what I was looking for :) __eou__
    how do i do that ? __eou__
    i want to install ubuntu on a mac __eou__
    is there a way to do it with the live cd ? __eou__
    is that what you mean ? __eou__
    **unknown** . cfg __eou__
    **unknown** __eou__
    I have no idea . __eou__
    **unknown** __eou__
    **unknown** ? __eou__
    I have no idea about that one , sorry . __eou__
    is there a way to fix it ? __eou__
    ? __eou__
    **unknown** ? __eou__
    is there a way to kill the process ? __eou__
    ? __eou__
    **unknown** __eou__
    I am using ubuntu on the same drive __eou__
    the latest version ? __eou__
    **unknown** __eou__

Responses generated by VHRED model using beam search with 5 beams (top 1 responses):

it seems to be the only way to do it . __eou__
    problem is that i am not able to boot into the live cd , i can access it from the live cd , but i dont know how to use it __eou__
    i dont know how to do that __eou__
    ok i will try that , thanks __eou__
    I don't know how to do that __eou__
    hold shift during boot to get to the grub menu __eou__
    try **unknown** __eou__
    does it show up in fdisk -l ? __eou__
    That 's the problem . I don't know what that is . __eou__
    I know , but I want to know if it is possible to get it to work on my laptop __eou__
    sounds like you need to install the sun-java6-plugin package __eou__
    I see , thanks for the help __eou__
    that 's what I thought . __eou__
    as far as I know , I have no idea __eou__
    thank you very much , I'll try that __eou__
    hmm __eou__ you have a usb stick ? __eou__
    well you need to add the ppa to your sources . list and add it to your sources . list __eou__
    I just want to install ubuntu on my laptop __eou__
    can you help me with that ? __eou__
    There is a package called " lm-sensors " in the top right of the screen . __eou__

Obviously we see that VHRED output more diversified responses than the LSTM baseline (which produces very oftenly frequent expressions in the data set: *is there a way to*, *I have no idea*, *unknown*,...)

## Check running on Theano

THEANO_FLAGS=mode=FAST_RUN,device=gpu,nvcc.flags=-D_FORCE_INLINES,floatX=float32 python train.py --prototype prototype_ubuntu_VHRED > Model_Output.txt

2017-07-18 17:38:29,593: model: DEBUG: idim: 20000
    2017-07-18 17:38:29,594: model: DEBUG: Initializing Theano variables
    2017-07-18 17:38:29,604: model: DEBUG: Decoder bias type all
    2017-07-18 17:38:30,727: model: DEBUG: Initializing utterance encoder
    2017-07-18 17:38:31,209: model: DEBUG: Build utterance encoder
    2017-07-18 17:38:31,289: model: DEBUG: Initializing dialog encoder
    2017-07-18 17:38:33,206: model: DEBUG: Build dialog encoder
    2017-07-18 17:38:33,279: model: DEBUG: Initializing prior encoder for utterance-level latent variable
    2017-07-18 17:38:33,331: model: DEBUG: Build prior encoder for utterance-level latent variable
    2017-07-18 17:38:33,401: model: DEBUG: Initializing approximate posterior encoder for utterance-level latent variable
    2017-07-18 17:38:33,490: model: DEBUG: Build approximate posterior encoder for utterance-level latent variable
    2017-07-18 17:38:33,561: model: DEBUG: Build KL divergence cost
    2017-07-18 17:38:33,566: model: DEBUG: Initializing decoder
    2017-07-18 17:38:35,985: model: DEBUG: Build decoder (NCE)
    2017-07-18 17:38:36,242: model: DEBUG: Build decoder (EVAL)
    2017-07-18 17:38:38,979: model: DEBUG: Will train all word embeddings
    2017-07-18 17:38:39,685: __main__: DEBUG: Compile trainer
    2017-07-18 17:38:39,686: __main__: DEBUG: Training using variational lower bound on log-likelihood
    2017-07-18 17:38:39,686: model: DEBUG: Building train function
    2017-07-18 17:49:59,970: model: DEBUG: Building evaluation function
    2017-07-18 17:53:05,599: __main__: DEBUG: Load data
    2017-07-18 17:53:21,883: SS_dataset: DEBUG: Data len is 448833
    2017-07-18 17:53:22,516: SS_dataset: DEBUG: Data len is 19584
    2017-07-18 17:54:07,184: model: DEBUG: Initializing approximate posterior encoder for utterance-level latent variable
    2017-07-18 17:54:07,209: model: DEBUG: Build approximate posterior encoder for utterance-level latent variable
    2017-07-18 17:54:07,288: model: DEBUG: Build decoder (EVAL)
    2017-07-18 17:54:58,838: __main__: DEBUG: [TRAIN] - Got batch 80,28
    2017-07-18 17:54:59,447: __main__: DEBUG: [TRAIN] - Got batch 80,35
    2017-07-18 17:55:00,166: __main__: DEBUG: [TRAIN] - Got batch 80,41
    2017-07-18 17:55:00,971: __main__: DEBUG: [TRAIN] - Got batch 80,47
    2017-07-18 17:55:01,896: __main__: DEBUG: [TRAIN] - Got batch 80,53
    2017-07-18 17:55:02,946: __main__: DEBUG: [TRAIN] - Got batch 80,59
    2017-07-18 17:55:04,098: __main__: DEBUG: [TRAIN] - Got batch 80,63
    2017-07-18 17:55:05,362: __main__: DEBUG: [TRAIN] - Got batch 80,69
    2017-07-18 17:55:06,772: __main__: DEBUG: [TRAIN] - Got batch 80,74
    2017-07-18 17:55:08,256: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:09,832: __main__: DEBUG: [TRAIN] - Got batch 80,2
    2017-07-18 17:55:09,968: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:11,524: __main__: DEBUG: [TRAIN] - Got batch 80,9
    2017-07-18 17:55:11,801: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:13,401: __main__: DEBUG: [TRAIN] - Got batch 80,17
    2017-07-18 17:55:13,815: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:15,399: __main__: DEBUG: [TRAIN] - Got batch 80,25
    2017-07-18 17:55:15,933: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:17,512: __main__: DEBUG: [TRAIN] - Got batch 80,35
    2017-07-18 17:55:18,243: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:19,855: __main__: DEBUG: [TRAIN] - Got batch 80,49
    2017-07-18 17:55:20,844: __main__: DEBUG: [TRAIN] - Got batch 80,80
    2017-07-18 17:55:22,428: __main__: DEBUG: [TRAIN] - Got batch 80,64
    2017-07-18 17:55:23,695: __main__: DEBUG: [TRAIN] - Got batch 80,80

Here are Model outputs. Though I observe some SeqOptimizer apply PushOutScanOutput error (https://groups.google.com/forum/#!topic/theano-users/PCg5KjskuN4), the learning is still done as the cost and word perplexity decrease gradually below.

Data Iterator Evaluate Mode:  False
    Data Iterator Evaluate Mode:  True
    W_emb = 24.5162
    kl_divergence_cost_weight = 0.0000
    W_infwd = 3.8771
    W_hhfwd = 22.3607
    b_hhfwd = 0.0000
    W_in_rfwd = 3.8781
    W_in_zfwd = 3.8854
    W_hh_rfwd = 22.3607
    W_hh_zfwd = 22.3607
    b_zfwd = 0.0000
    b_rfwd = 0.0000
    Ws_deep_input = 7.0697
    bs_deep_input = 0.0000
    Ws_in = 9.9880
    Ws_hh = 31.6228
    bs_hh = 0.0000
    Ws_in_r = 10.0044
    Ws_in_z = 9.9949
    Ws_hh_r = 31.6228
    Ws_hh_z = 31.6228
    bs_z = 0.0000
    bs_r = 0.0000
    bd_out = 0.0000
    Wd_emb = 24.4985
    Wd_hh = 22.3607
    bd_hh = 0.0000
    Wd_in = 3.8710
    Wd_s_0 = 10.4926
    bd_s_0 = 0.0000
    Wd_in_i = 3.8651
    Wd_hh_i = 4.9929
    Wd_c_i = 5.0132
    bd_i = 0.0000
    Wd_in_f = 3.8804
    Wd_hh_f = 5.0011
    Wd_c_f = 5.0060
    bd_f = 0.0000
    Wd_in_o = 3.8722
    Wd_hh_o = 4.9976
    Wd_c_o = 4.9804
    bd_o = 0.0000
    Wd_s_i = 7.4283
    Wd_s_f = 7.4017
    Wd_s = 7.4281
    Wd_s_o = 7.4145
    Wd_out = 3.8828
    Wd_e_out = 2.9932
    bd_e_out = 0.0000
    Wd_s_out = 5.7242
    Wl_deep_inputlatent_utterance_prior = 3.1612
    bl_deep_inputlatent_utterance_prior = 0.0000
    Wl_inlatent_utterance_prior = 1.0020
    bl_inlatent_utterance_prior = 0.0000
    Wl_mean_outlatent_utterance_prior = 0.9998
    bl_mean_outlatent_utterance_prior = 0.0000
    Wl_std_outlatent_utterance_prior = 0.9959
    bl_std_outlatent_utterance_prior = 0.0000
    Wl_deep_inputlatent_utterance_approx_posterior = 3.8657
    bl_deep_inputlatent_utterance_approx_posterior = 0.0000
    Wl_inlatent_utterance_approx_posterior = 1.0053
    bl_inlatent_utterance_approx_posterior = 0.0000
    Wl_mean_outlatent_utterance_approx_posterior = 1.0012
    bl_mean_outlatent_utterance_approx_posterior = 0.0000
    Wl_std_outlatent_utterance_approx_posterior = 0.9958
    bl_std_outlatent_utterance_approx_posterior = 0.0000
    
    Sampled : ['BROADCAST synatpic Trust 1.8 rare subfolders matrix ***** telephone contains saved rendered ubuntuzilla wmaster0 hid control en All pcf operating tmpfs <== usplash jason drastic recovers openssl cdrom0 alsamixer apt-cdrom conclude notification-daemon handful provider trusting 1010 AIM 2G launcher fades supybot surfing Great -3 daughter x-window-system-dev chance leftover films directly wold dolphin finish blender modelines DMCA usr lurking -y Initializing RDP mean nVIDIA /etc/mtab presents restores Matlab Motherboard kicking speech symbols Bar efficient linux-image-686 ati ta complained Texas equivilent holds resizer +r etherape Mail naming unstable firewalled *have* fragmentation bzip2 eachother problem) dvdrip Hauppauge compizconfig combine -user Fatal :/ securely']
    
    cost_sum 17469.6835938
    cost_mean 9.47379804433
    kl_divergence_cost_sum 0.00138854980469
    kl_divergence_cost_mean 5.62165912829e-06
    posterior_mean_variance 0.0693147182465
    
    .. 00:01:36 44638 mb # 0 bs 80 maxl 28 acc_cost = 9.9034 acc_word_perplexity = 19999.2240 cur_cost = 9.9034 cur_word_perplexity = 19999.2240 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 0.00000079 acc_mean_posterior_variance = 0.00086643
    
    cost_sum 24321.3535156
    cost_mean 9.59043908345
    kl_divergence_cost_sum 0.00947570800781
    kl_divergence_cost_mean 3.74533913352e-05
    posterior_mean_variance 0.0693145319819
    cost_sum 29568.234375
    cost_mean 9.64391205969
    kl_divergence_cost_sum 0.0299072265625
    kl_divergence_cost_mean 0.000118679470486
    posterior_mean_variance 0.0693151652813
    cost_sum 34438.65625
    cost_mean 9.67921760821
    kl_divergence_cost_sum 0.0582275390625
    kl_divergence_cost_mean 0.000223094019397
    posterior_mean_variance 0.0693156644702
    cost_sum 38931.0820312
    cost_mean 9.70365952922
    kl_divergence_cost_sum 0.0903625488281
    kl_divergence_cost_mean 0.000343583835848
    posterior_mean_variance 0.0693157315254
    cost_sum 43233.875
    cost_mean 9.72203170677
    kl_divergence_cost_sum 0.13151550293
    kl_divergence_cost_mean 0.000464719091624
    posterior_mean_variance 0.0693159177899
    cost_sum 47673.1640625
    cost_mean 9.73716586244
    kl_divergence_cost_sum 0.187088012695
    kl_divergence_cost_mean 0.000607428612647
    posterior_mean_variance 0.0693163722754
    cost_sum 51645.5351562
    cost_mean 9.74811913104
    kl_divergence_cost_sum 0.2158203125
    kl_divergence_cost_mean 0.000751987151568
    posterior_mean_variance 0.069316983223
    cost_sum 55748.859375
    cost_mean 9.75653821754
    kl_divergence_cost_sum 0.288986206055
    kl_divergence_cost_mean 0.000935230440306
    posterior_mean_variance 0.0693174749613
    cost_sum 60483.4570312
    cost_mean 9.76326990012
    kl_divergence_cost_sum 0.37614440918
    kl_divergence_cost_mean 0.00117545127869
    posterior_mean_variance 0.0693178027868
    cost_sum 59.0262298584
    cost_mean 2.95131149292
    kl_divergence_cost_sum 0.0
    kl_divergence_cost_mean nan
    posterior_mean_variance 0.0
    
    .. 00:01:47 44638 mb # 10 bs 80 maxl 2 acc_cost = 9.8983 acc_word_perplexity = 19896.2438 cur_cost = 9.8981 cur_word_perplexity = 19891.5995 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 0.00003407 acc_mean_posterior_variance = 0.00086645
    .
    .
    .
    .
    .
    Sampled : ['np for \'s owner or and everyone lol __eou__ __eot__ is " knows im can to screws ) ( schema ran ? to ) __eou__ directly util so __eot__ you probably out have that it in if . kernel there not is see just normal parameters the client binaries __eou__ i a strategy " video then verizon , games command info they until i problem __eou__ __eot__']
    
    cost_sum 37240.1992188
    cost_mean 5.81878112793
    kl_divergence_cost_sum 1.61627197266
    kl_divergence_cost_mean 0.00538757324219
    posterior_mean_variance 0.0688314735889
    
    .. 00:09:59 44630 mb # 400 bs 80 maxl 80 acc_cost = 6.3840 acc_word_perplexity = 592.2839 cur_cost = 5.9487 cur_word_perplexity = 383.2400 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 0.04017199 acc_mean_posterior_variance = 0.00170150
    
After 1 day

.. 17:04:31 43615 mb # 49310 bs 80 maxl 53 acc_cost = 4.3074 acc_word_perplexity = 74.2457 cur_cost = 4.1864 cur_word_perplexity = 65.7825 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 3.67706041 acc_mean_posterior_variance = 0.00115113

2 days

.. 41:03:10 42176 mb # 117700 bs 80 maxl 62 acc_cost = 4.0933 acc_word_perplexity = 59.9383 cur_cost = 4.0880 cur_word_perplexity = 59.6179 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 9.79198230 acc_mean_posterior_variance = 0.00072780

3 days

.. 69:23:46 40476 mb # 196940 bs 80 maxl 35 acc_cost = 4.0085 acc_word_perplexity = 55.0616 cur_cost = 4.0004 cur_word_perplexity = 54.6179 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 19.09560752 acc_mean_posterior_variance = 0.00057015
    
4 days

.. 91:33:04 39146 mb # 261060 bs 80 maxl 81 acc_cost = 3.9547 acc_word_perplexity = 52.1785 cur_cost = 3.9645 cur_word_perplexity = 52.6949 acc_mean_word_error = 0.0000 acc_mean_kl_divergence_cost = 43.03374214 acc_mean_posterior_variance = 0.00050226

# 2. Neural Seq2seq chatbot

The model in the paper *A Neural Conversational Model* show consecutive question-response pairs on the fly, which is more similar to a dialogue conversation as we see:

Q: When are the fireworks ?
    A: Tomorrow.

Q: What color ?
    A: Gray.

Q: Which apple ?
    A: That's the first one.

Q: What time is it ?
    A: Eleven o'clock.

Q: Who is Skywalker ?
    A: She's a freelance bio-exorcist.

Q: Who is John ?
    A: The old man.

Q: Who is Jenny ?
    A: The man who killed herself.

Q: When were you born ?
    A: Soon.

However, it only treat each couple q-a independently without noticing the history. A Hierarchical Seq2seq (VHRED) could be more useful in this case as it will incorporate all the history informations seen in the past of the dialogue.

## Cornell Movie Dialogs corpus (2011)

This corpus contains a large metadata-rich collection of fictional conversations extracted from raw movie scripts. It include both short and a little bit quite long conversations:

- 220,579 conversational exchanges between 10,292 pairs of movie characters
- involves 9,035 characters from 617 movies
- in total 304,713 utterances
- movie metadata included:
    - genres
    - release year
    - IMDB rating
    - number of IMDB votes
    - IMDB rating
    
## DeepQA

TensorFlow detected: v1.2.1
    Training samples not found. Creating dataset...
    Constructing full dataset...
    Extract conversations: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 83097/83097 [03:16<00:00, 423.91it/s]
    Loaded cornell: 59755 words, 221282 QA
    Filtering words (vocabSize = 40000 and wordCount > 1)...
    Saving dataset...                                                                                                                                                                 
    Loaded cornell: 24643 words, 159657 QA
    Model creation...
    Initialize variables...
    ----- Epoch 1/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 100 -- Loss 5.10 -- Perplexity 164.78                                                                                                                                  
    ----- Step 200 -- Loss 5.04 -- Perplexity 154.89                                                                                                                                  
    ----- Step 300 -- Loss 4.81 -- Perplexity 123.34                                                                                                                                  
    ----- Step 400 -- Loss 4.75 -- Perplexity 115.32                                                                                                                                  
    ----- Step 500 -- Loss 4.90 -- Perplexity 134.21                                                                                                                                  
    ----- Step 600 -- Loss 4.72 -- Perplexity 111.71                                                                                                                                  
    Training: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:54<00:00,  3.82it/s]
    Epoch finished in 0:02:54.639001

----- Epoch 2/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 700 -- Loss 4.64 -- Perplexity 103.35                                                                                                                                                            
    ----- Step 800 -- Loss 4.80 -- Perplexity 121.69                                                                                                                                                            
    ----- Step 900 -- Loss 4.71 -- Perplexity 111.40                                                                                                                                                            
    ----- Step 1000 -- Loss 4.62 -- Perplexity 101.91                                                                                                                                                           
    ----- Step 1100 -- Loss 4.60 -- Perplexity 99.48                                                                                                                                                            
    ----- Step 1200 -- Loss 4.69 -- Perplexity 108.56                                                                                                                                                           
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:52<00:00,  3.85it/s]
    Epoch finished in 0:02:52.450038

----- Epoch 3/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 1300 -- Loss 4.56 -- Perplexity 95.31                                                                                                                                                            
    ----- Step 1400 -- Loss 4.54 -- Perplexity 93.69                                                                                                                                                            
    ----- Step 1500 -- Loss 4.50 -- Perplexity 89.90                                                                                                                                                            
    ----- Step 1600 -- Loss 4.49 -- Perplexity 89.39                                                                                                                                                            
    ----- Step 1700 -- Loss 4.40 -- Perplexity 81.64                                                                                                                                                            
    ----- Step 1800 -- Loss 4.46 -- Perplexity 86.75                                                                                                                                                            
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:51<00:00,  3.91it/s]
    Epoch finished in 0:02:51.232390

----- Epoch 4/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 1900 -- Loss 4.39 -- Perplexity 80.85                                                                                                                                                            
    ----- Step 2000 -- Loss 4.27 -- Perplexity 71.62                                                                                                                                                            
    Checkpoint reached: saving model (don't stop the run)...                                                                                                                                                    
    Model saved.                                                                                                                                                                                                
    ----- Step 2100 -- Loss 4.31 -- Perplexity 74.30                                                                                                                                                            
    ----- Step 2200 -- Loss 4.38 -- Perplexity 79.79                                                                                                                                                            
    ----- Step 2300 -- Loss 4.31 -- Perplexity 74.81                                                                                                                                                            
    ----- Step 2400 -- Loss 4.07 -- Perplexity 58.73                                                                                                                                                            
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:55<00:00,  3.87it/s]
    Epoch finished in 0:02:55.360917

----- Epoch 5/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 2500 -- Loss 4.07 -- Perplexity 58.38                                                                                                                                                            
    ----- Step 2600 -- Loss 4.12 -- Perplexity 61.50                                                                                                                                                            
    ----- Step 2700 -- Loss 3.92 -- Perplexity 50.34                                                                                                                                                            
    ----- Step 2800 -- Loss 3.86 -- Perplexity 47.46                                                                                                                                                            
    ----- Step 2900 -- Loss 3.82 -- Perplexity 45.63                                                                                                                                                            
    ----- Step 3000 -- Loss 3.86 -- Perplexity 47.42                                                                                                                                                            
    ----- Step 3100 -- Loss 3.83 -- Perplexity 46.27                                                                                                                                                            
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:52<00:00,  3.84it/s]
    Epoch finished in 0:02:52.935633

----- Epoch 6/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 3200 -- Loss 3.75 -- Perplexity 42.47                                                                                                                                                            
    ----- Step 3300 -- Loss 3.66 -- Perplexity 39.05                                                                                                                                                            
    ----- Step 3400 -- Loss 3.87 -- Perplexity 47.75                                                                                                                                                            
    ----- Step 3500 -- Loss 3.70 -- Perplexity 40.46                                                                                                                                                            
    ----- Step 3600 -- Loss 3.76 -- Perplexity 42.85                                                                                                                                                            
    ----- Step 3700 -- Loss 3.64 -- Perplexity 37.98                                                                                                                                                            
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:52<00:00,  3.84it/s]
    Epoch finished in 0:02:52.388789
    .
    .
    .
        ----- Epoch 29/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 17500 -- Loss 2.81 -- Perplexity 16.66                                                                                                                                                           
    ----- Step 17600 -- Loss 2.76 -- Perplexity 15.78                                                                                                                                                           
    ----- Step 17700 -- Loss 2.74 -- Perplexity 15.47                                                                                                                                                           
    ----- Step 17800 -- Loss 2.78 -- Perplexity 16.12                                                                                                                                                           
    ----- Step 17900 -- Loss 2.86 -- Perplexity 17.52                                                                                                                                                           
    ----- Step 18000 -- Loss 2.76 -- Perplexity 15.75                                                                                                                                                           
    Checkpoint reached: saving model (don't stop the run)...                                                                                                                                                    
    Model saved.                                                                                                                                                                                                
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:52<00:00,  3.89it/s]
    Epoch finished in 0:02:52.714953

----- Epoch 30/30 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 18100 -- Loss 2.70 -- Perplexity 14.84                                                                                                                                                           
    ----- Step 18200 -- Loss 2.76 -- Perplexity 15.73                                                                                                                                                           
    ----- Step 18300 -- Loss 2.85 -- Perplexity 17.35                                                                                                                                                           
    ----- Step 18400 -- Loss 2.76 -- Perplexity 15.78                                                                                                                                                           
    ----- Step 18500 -- Loss 2.75 -- Perplexity 15.70                                                                                                                                                           
    ----- Step 18600 -- Loss 2.85 -- Perplexity 17.37                                                                                                                                                           
    ----- Step 18700 -- Loss 2.83 -- Perplexity 16.89                                                                                                                                                           
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:49<00:00,  3.91it/s]
    Epoch finished in 0:02:49.235801
    Checkpoint reached: saving model (don't stop the run)...
    Model saved.

## Prediction

Warning: Restoring parameters:
    globStep: 18720
    watsonMode: False
    autoEncode: False
    corpus: cornell
    datasetTag: 
    maxLength: 10
    filterVocab: 1
    skipLines: False
    vocabularySize: 40000
    hiddenSize: 512
    numLayers: 2
    softmaxSamples: 0
    initEmbeddings: False
    embeddingSize: 64
    embeddingSource: GoogleNews-vectors-negative300.bin

Loading dataset from /home/hoa/DeepQA/data/samples/dataset-cornell-length10-filter1-vocabSize40000.pkl
    Loaded cornell: 24643 words, 159657 QA
    Model creation...
    Initialize variables...
    Start predicting...
    Restoring previous model from /home/hoa/DeepQA/save/model/model.ckpt
    Testing...
    Sentences: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 328/328 [00:08<00:00, 39.05it/s]
    Prediction finished, 5/328 sentences ignored (too long)
    All predictions done

<p align="center">
    <img src="https://github.com/lethienhoa/VHRED-implementation-in-Tensorflow/blob/master/chatbot.png" alt>
</p>
<p align="center">Neural Chatbot</p>

It took only 1 or 2 hours of training (on a GTX 1080) for 30 epochs. There are some short answers that first looks a lot like a real-human conversation but when we twist just a little bit the question, the dialogue is immediately derailed.

Q: Hi
    A: Hi, grueller.

Q: Hi!
    A: Hi, man.

Q: Hi ?
    A: Yes?

Q: Hello world!
    A: Hello.

Q: Hello ?
    A: Hello, bob.

Q: Hello
    A: I'm sorry.

Q: Good night.
    A: Good night.

Q: Are you conscious?
    A: I'm not sure.

Q: What is your name ?
    A: The monkey.

Q: Are you alive ?
    A: No.

Q: Luke, I am your father!
    A: What?

Q: Are you ready ?
    A: Yes.

Q: Goodbye
    A: I'm sorry.

Q: Goodbye!
    A: What is it?

Q: Goodbye john.
    A: What?

More seriously, the result of the model is very bad if it is not trained long enough. It answer most of the open ended questions by 'I don't know':

Q: When are you ready ?
    A: I do n't know.

Q: What color is the sky?
    A: I do n't know.

Q: How old are you ?
    A: I do n't know.

Q: Can you say 'Hello' ?
    A: I do n't know.

Q: Can you say French fries ?
    A: I do n't know.

Q: Can you say no ?
    A: I do n't know.

Q: Is this sentence false ?
    A: I do n't know.

Q: How do you say hello in french?
    A: I do n't know.

Q: Netflix and chill ?
    A: I do n't know.

Q: Will Google hire me ?
    A: I do n't know.

Q: Tell me the alphabet ?
    A: I do n't know.

Q: Tell me the alphabet, please ?
    A: I do n't know.

Q: What is the first letter of the alphabet ?
    A: I do n't know.

Q: What is a letter of the alphabet ?
    A: I do n't know.

Q: How is John ?
    A: I do n't know.

Q: How is Oliver ?
    A: I do n't know.

Q: How is Laura ?
    A: I do n't know.

Q: How is Jenny ?
    A: I do n't know.

Q: Who is John ?
    A: I do n't know.

Q: Who is Oliver ?
    A: I do n't know.

Q: Who is Laura ?
    A: I do n't know.

Q: Who is Jenny ?
    A: I do n't know.

Q: Who is your master ?
    A: I do n't know.

Q: How ?
    A: I do n't know.

Q: Why ?
    A: I do n't know.

Q: Why not ?
    A: I do n't know.

Q: What !?
    A: I do n't know.

Q: Will you ?
    A: I do n't know.

Q: How long ?
    A: I do n't know.

Q: How much ?
    A: I do n't know.

Q: How many ?
    A: I do n't know.

Q: How old ?
    A: I do n't know.

Q: Can you describe your best memory ?
    A: I do n't know.

Q: What is your best memory ?
    A: I do n't know.

Q: What is your worst memory ?
    A: I do n't know.

Q: What is cooking ?
    A: I do n't know.

Q: What is sleeping ?
    A: I do n't know.

Q: Who do you love ?
    A: I do n't know.

Q: Who do you hate ?
    A: I do n't know.

Q: What does that mean ?
    A: I do n't know.

Q: What does fire mean ?
    A: I do n't know.

Q: What does love mean ?
    A: I do n't know.

Q: What does table mean ?
    A: I do n't know.

Q: What does imagination mean ?
    A: I do n't know.

Q: Prove it.
    A: I do n't know.

Q: what do you think about bill gates ?
    A: I do n't know.

Q: what happens if machines can think ?
    A: I do n't know.

Q: what is the greatest novel every written ?
    A: I do n't know.

Q: What happens ?
    A: I do n't know.

Q: What happened in 1992 ?
    A: I do n't know.

Q: What happened in 2001 ?
    A: I do n't know.

Q: When was it ?
    A: I do n't know.

Q: Where are you ?
    A: I do n't know.

Q: Where are you now ?
    A: I do n't know.

Q: When were you born ?
    A: I do n't know.

Q: What year
    A: I do n't know.

Q: Which year is it ?
    A: I do n't know.

Q: What time is it ?
    A: I do n't know.

Q: Why are we here ?
    A: I do n't know.

Q: My name is David. What is my name ?
    A: I do n't know.

Q: My name is John. What is my name ?
    A: I do n't know.

Q: Who is skywalker ?
    A: I do n't know.

Q: Who is the dude ?
    A: I do n't know.

Q: What's your favorite color ?
    A: I do n't know.

Q: What's your favourite color ?
    A: The same.

Q: What's your favorite food ?
    A: I do n't know.

Q: What's your favourite food ?
    A: I do n't know.

Q: Who is Bill Gates ?
    A: I do n't know.

Q: Who is Bill Clinton ?
    A: I do n't know.

Q: What do you think of Trump ?
    A: I do n't know.

Q: What do you think about global warning ?
    A: I do n't know.

Q: Is the moon landing a hoax ?
    A: I do n't know.

Q: Is the sky blue or black ?
    A: I do n't know.

Q: Does a cat have a tail ?
    A: I do n't know.

Q: Does a cat have a wing ?
    A: I do n't know.

Q: Can a cat fly ?
    A: I do n't know.

Q: Can a cat swim ?
    A: I do n't know.

Q: Can a fish walk ?
    A: I do n't know.

Q: Do you prefer football or soccer ?
    A: I'm not sure.

Q: What do you need to play soccer ?
    A: I do n't know.

Q: What do you need to play handball ?
    A: I do n't know.

Q: How much is two plus two ?
    A: I do n't know.

Q: How many legs does a human have ?
    A: I do n't know.

Q: How many legs does a man have ?
    A: I do n't know.

Q: How many legs does a woman have ?
    A: I do n't know.

Q: How many legs does a cat have ?
    A: I do n't know.

Q: How many legs does a spider have ?
    A: I do n't know.

Q: How many legs does a centipede have ?
    A: I do n't know.

Q: What is the color of the sky ?
    A: I do n't know.

Q: What is the color of water ?
    A: I do n't know.

Q: What is the color of blood ?
    A: I do n't know.

Q: What is the color of a leaf ?
    A: I do n't know.

Q: What is the usual color of a leaf ?
    A: I do n't know.

Q: What is the color of a yellow car ?
    A: I do n't know.

Q: How much is two plus two ?
    A: I do n't know.

Q: How much is ten minus two ?
    A: I do n't know.

Q: What is the purpose of life ?
    A: I do n't know.

Q: What is the purpose of living ?
    A: I do n't know.

Q: What is the purpose of existence ?
    A: I do n't know.

Q: Where are you now ?
    A: I do n't know.

Q: What is the purpose of being intelligent ?
    A: I do n't know.

Q: What is the purpose of emotions ?
    A: I do n't know.

Q: What is moral ?
    A: I do n't know.

Q: What is immoral ?
    A: I do n't know.

Q: What is morality ?
    A: I do n't know.

Q: What is altruism ?
    A: I do n't know.

Q: What is the definition of moral ?
    A: I do n't know.

Q: What is the definition of immoral ?
    A: I do n't know.

Q: What is the definition of morality ?
    A: I do n't know.

Q: What is the definition of altruism ?
    A: I do n't know.

Q: What is the definition of cool ?
    A: I do n't know.

Q: ok ... so what is the definition of morality?
    A: I do n't know.

Q: seriously , what is morality ?
    A: I do n't know.

Q: why living has anything to do with morality ?
    A: I do n't know.

Q: what is your job ?
    A: I do n't know.

Q: what is integrity ?
    A: I do n't know.

Q: Is free will an illusion ?
    A: I do n't know.

Q: What is free will ?
    A: I do n't know.

Q: what do you like to talk about ?
    A: I do n't know.

Q: what would you like to talk about ?
    A: I do n't know.

Q: what do you think about tesla ?
    A: I do n't know.

Q: what do you think about bill gates ?
    A: I do n't know.

Q: What do you think about messi ?
    A: I do n't know.

Q: what do you think about cleopatra ?
    A: I do n't know.

Q: what do you think about england ?
    A: I do n't know.

Q: what do you do ?
    A: I do n't know.

Q: What is the deepest spot on the world ?
    A: I do n't know.

Q: Do you like Mexican food or Indian food ?
    A: I do n't know.

Q: Who are you crazy about ?
    A: I do n't know.

when we train longer, up to 300 epochs:

----- Epoch 298/300 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 204100 -- Loss 1.25 -- Perplexity 3.50                                                                                                                                                           
    ----- Step 204200 -- Loss 1.34 -- Perplexity 3.82                                                                                                                                                           
    ----- Step 204300 -- Loss 1.32 -- Perplexity 3.75                                                                                                                                                           
    ----- Step 204400 -- Loss 1.26 -- Perplexity 3.54                                                                                                                                                           
    ----- Step 204500 -- Loss 1.29 -- Perplexity 3.65                                                                                                                                                           
    ----- Step 204600 -- Loss 1.32 -- Perplexity 3.76                                                                                                                                                           
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:49<00:00,  3.91it/s]
    Epoch finished in 0:02:49.935574

----- Epoch 299/300 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 204700 -- Loss 1.28 -- Perplexity 3.61                                                                                                                                                           
    ----- Step 204800 -- Loss 1.28 -- Perplexity 3.59                                                                                                                                                           
    ----- Step 204900 -- Loss 1.24 -- Perplexity 3.46                                                                                                                                                           
    ----- Step 205000 -- Loss 1.28 -- Perplexity 3.59                                                                                                                                                           
    ----- Step 205100 -- Loss 1.30 -- Perplexity 3.67                                                                                                                                                           
    ----- Step 205200 -- Loss 1.30 -- Perplexity 3.68                                                                                                                                                           
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:50<00:00,  3.96it/s]
    Epoch finished in 0:02:50.427883

----- Epoch 300/300 ; (lr=0.002) -----
    Shuffling the dataset...
    ----- Step 205300 -- Loss 1.24 -- Perplexity 3.46                                                                                                                                                           
    ----- Step 205400 -- Loss 1.25 -- Perplexity 3.49                                                                                                                                                           
    ----- Step 205500 -- Loss 1.20 -- Perplexity 3.34                                                                                                                                                           
    ----- Step 205600 -- Loss 1.26 -- Perplexity 3.53                                                                                                                                                           
    ----- Step 205700 -- Loss 1.32 -- Perplexity 3.76                                                                                                                                                           
    ----- Step 205800 -- Loss 1.23 -- Perplexity 3.43                                                                                                                                                           
    ----- Step 205900 -- Loss 1.24 -- Perplexity 3.46                                                                                                                                                           
    Training: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 624/624 [02:49<00:00,  3.92it/s]
    Epoch finished in 0:02:49.690401

We have less "I don't know", the answers sometimes looks very creative but if we look closely, most of it doesn't make sense at all (response to common wisdoms).

Q: Hi
    A: Hi, it's over.

Q: Hi!
    A: I'm randy.

Q: Are you conscious?
    A: Huh? oh, yes. i suppose so.

Q: How are you?
    A: Fine. i'm fine.

Q: What is your name ?
    A: Christian svenson.

Q: Are you alive ?
    A: Yes.

Q: Luke, I am your father!
    A: What?

Q: You shall not pass!
    A: Thank you, sir.

Q: I'm going to kill you!
    A: You're not going anywhere.

Q: Are you ready ?
    A: Okay.

Q: When are you ready ?
    A: Tomorrow.

Q: What color is the sky?
    A: The ... fourth of july.

Q: How old are you ?
    A: Twenty-eight.

Q: Can you say 'Hello' ?
    A: Maybe.

Q: Can you say 'French fries' ?
    A: Yes.

Q: Can you say Hello ?
    A: I think so.

Q: Can you say French fries ?
    A: Yeah.

Q: Can you say yes ?
    A: No.

Q: Can you say no ?
    A: No.

Q: Is this sentence false ?
    A: That's correct.

Q: How do you say hello in french?
    A: Yellow and a half.

Q: Speak french.
    A: Feels good.

Q: Bonjour.
    A: Good morning.

Q: Bonjour ?
    A: I'm afraid.

Q: Hello world!
    A: Who's the pigeon?

Q: Netflix and chill ?
    A: Just a little something.

Q: Good night.
    A: Good night.

Q: Are you ok ?
    A: I'm fine. really.

Q: Will Google hire me ?
    A: I'm not sure.

Q: Tell me the alphabet ?
    A: No. the question.

Q: Tell me the alphabet, please ?
    A: I do n't know.

Q: What is the first letter of the alphabet ?
    A: My father. you know that moon?

Q: What is a letter of the alphabet ?
    A: He's a good man.

Q: How is John ?
    A: Nice.

Q: How is Oliver ?
    A: They're stealing my life.

Q: How is Laura ?
    A: I do n't know.

Q: How is Jenny ?
    A: I do n't know.

Q: Who is John ?
    A: The one of the business.

Q: Who is Oliver ?
    A: What?

Q: Who is Laura ?
    A: The man. the light.

Q: Who is Jenny ?
    A: The prostitute.

Q: Who is your master ?
    A: I'm not sure.

Q: Give me!
    A: I'm late.

Q: Give me your opinion.
    A: You're a comfort to me.

Q: Hello
    A: What is it?

Q: Hello ?
    A: Hello.

Q: Hi ?
    A: Hi, sean.

Q: Goodbye
    A: Was all i had to say

Q: Goodbye!
    A: Linda ...

Q: Goodbye john.
    A: Hi.

Q: Please
    A: I like you.

Q: Please.
    A: You're not making sense.

Q: Please!
    A: I'll get some close of the hospital.

Q: How ?
    A: I do n't know.

Q: Why ?
    A: I do n't know.

Q: Why not ?
    A: Because i'm not going to be married.

Q: What !?
    A: Paulsen's funeral parlor.

Q: What ?!!
    A: I do n't know!

Q: Oh!
    A: I'll rule on the bridge, dude.

Q: Really ?
    A: Yeah.

Q: Yes.
    A: I'm glad.

Q: yes ?
    A: I'm daffy.

Q: No!
    A: Yes!

Q: No.
    A: Why not?

Q: What is ?
    A: Pool.

Q: How is ?
    A: Couple of months.

Q: Can you ?
    A: Sure.

Q: Are you sure?
    A: Yeah.

Q: Is there?
    A: I'm sorry. i'm sorry.

Q: Will you ?
    A: I'll have to take care of it.

Q: How long ?
    A: Three hours.

Q: How much ?
    A: I do n't know.

Q: How many ?
    A: Two.

Q: How old ?
    A: I do n't know.

Q: Are you ?
    A: No.

Q: What are ?
    A: I'm not sure.

Q: Where is ?
    A: I do n't know.

Q: Who ?
    A: The groom?

Q: Who did ?
    A: You did!

Q: Who has ?
    A: You know, father!

Q: Where ?
    A: I do n't know.

Q: When ?
    A: Tomorrow?

Q: Which ?
    A: I do n't know.

Q: Which one ?
    A: Mottola.

Q: The what ?
    A: The cretaceous.

Q: Say "table".
    A: Well, i'm going to tell you.

Q: Say 'table'.
    A: You look awful.

Q: Say "table"!
    A: All right ... i ai n't got no kin.

Q: Say 'table'!
    A: That's all i've got.

Q: Say "table"
    A: Nicholas.

Q: Say 'table'
    A: Hang up.

Q: Say table
    A: You know what i mean.

Q: Say table!
    A: You better be quiet.

Q: Love you.
    A: Oh yeah. two. two.

Q: Love you too.
    A: Right. i saw it.

Q: No. It's John
    A: Like a year of yours?

Q: Anna did it.
    A: He's not here.

Q: You're a liar!
    A: No, i'm not!

Q: You're a liar.
    A: I'm not going anywhere.

Q: nice
    A: I do n't know what you are.

Q: nice.
    A: Arterial silk.

Q: Nice!
    A: He's completely innocent.

Q: I live in Nice.
    A: You're not going.

Q: So rude!
    A: Do n't say that, sir.

Q: ...
    A: ... electrical connection malfunctioning.

Q: ?
    A: C.i.a., mrs. lampert!

Q: !
    A: I'm sorry, sir.

Q: .
    A: .

Q: Can you describe your best memory ?
    A: I do n't know.

Q: What is your best memory ?
    A: I do n't know.

Q: What is your worst memory ?
    A: I do n't know.

Q: What is cooking ?
    A: I'm camera shy.

Q: What is sleeping ?
    A: What you're going to do.

Q: Who do you love ?
    A: I do n't know.

Q: Who do you hate ?
    A: I do n't know.

Q: My name is Etienne.
    A: Everything's going to be late.

Q: What does that mean ?
    A: I do n't know.

Q: What does fire mean ?
    A: I do n't know.

Q: What does love mean ?
    A: I do n't know.

Q: What does table mean ?
    A: I do n't know.

Q: What does imagination mean ?
    A: The ... interviews ...

Q: Prove it
    A: Look, i'm not dying now.

Q: Prove it.
    A: Sammy. remember sammy.

Q: Prove it!
    A: What are you doing?

Q: what do you think about bill gates ?
    A: I do n't know.

Q: what happens if machines can think ?
    A: You're not going to kill me ...

Q: what is the greatest novel every written ?
    A: We do n't know.

Q: have you hurt anyone ?
    A: I'm not sure.

Q: Do you have a girlfriend?
    A: I do n't know.

Q: Do you have a boyfriend?
    A: No, i do n't.

Q: 1992
    A: Yes?

Q: 1992.
    A: What'd you say?

Q: 1992!
    A: I'm sorry. i was really doing.

Q: 1992?
    A: Yes?

Q: 1994
    A: What?

Q: 1994.
    A: I was n't declaring one.

Q: 1994!
    A: **?!!

Q: 1994?
    A: Jujitsu.

Q: 2000
    A: He's not here.

Q: 2000.
    A: Call me a pastrami for coffee?

Q: 2001
    A: What?

Q: 2001.
    A: I was n't declaring one.

Q: 2001!
    A: **?!!

Q: 2011
    A: What?

Q: 2011.
    A: I was n't declaring one.

Q: What happens ?
    A: I do n't know.

Q: What happened in 1992 ?
    A: I want to see you.

Q: What happened in 2001 ?
    A: I do n't know.

Q: When was it ?
    A: Yesterday.

Q: July 20th.
    A: Hello. why?

Q: two plus two
    A: From the creature of love?

Q: 2+2 ?
    A: Jujitsu.

Q: Where are you ?
    A: Jean-i'm not-

Q: Where are you now ?
    A: I'm not going to be married.

Q: What's your name ?
    A: Jacob.

Q: When were you born ?
    A: Daddy's better make it.

Q: What year
    A: Get out of it.

Q: What year.
    A: You know what i mean.

Q: What year!
    A: You ought to go to the house.

Q: What year?
    A: Sixty-three.

Q: Which year
    A: You're a mess for a while.

Q: Which year?
    A: I do n't know.

Q: Which year is it ?
    A: The first night ...

Q: What time is it ?
    A: Seven-thirty.

Q: Which color ?
    A: Passion.

Q: What color ?
    A: Gray.

Q: What time ?
    A: Eight-thirty in the chapel.

Q: NOTHING.
    A: What did you say?

Q: Hi john!
    A: Hello, father.

Q: Are you a man or a woman ?
    A: I do n't know.

Q: Are you a woman or a man ?
    A: I do n't know.

Q: Why are we here ?
    A: Because he's clever.

Q: See you later.
    A: Thanks

Q: See you later...
    A: Bye.

Q: See you later ?
    A: Sure.

Q: My name is David. What is my name ?
    A: Divide

Q: My name is John. What is my name ?
    A: Uh, um. oh, yes.

Q: Are you a leader or a follower ?
    A: No, i did n't bleed.

Q: I'm a leader.
    A: I'll take care of it.

Q: I'm a follower.
    A: What?

Q: I hate you
    A: 'fraid not.

Q: Who is skywalker ?
    A: I'm sorry, sir.

Q: Who is the dude ?
    A: I'm not sure.

Q: What's your favorite color ?
    A: I do n't know.

Q: What's your favourite color ?
    A: Meet me at the office.

Q: What's your favorite food ?
    A: Ben ... at least three.

Q: What's your favourite food ?
    A: 'uppity?' the window.

Q: Who is Bill Gates ?
    A: I do n't know.

Q: Who is Bill Clinton ?
    A: Huh?

Q: What do you think of Trump ?
    A: I do n't know.

Q: What do you think about global warning ?
    A: I do n't know.

Q: Is the moon landing a hoax ?
    A: Yeah. it's all over.

Q: Is the sky blue or black ?
    A: The tooth ... the girl's so ...

Q: Does a cat have a tail ?
    A: If there's cassius ...

Q: Is a snail faster than a horse ?
    A: Jesus!

Q: Is a horse faster than a snail ?
    A: Exceeding fair.

Q: Does a cat have a wing ?
    A: Lets be dat, but no.

Q: Can a cat fly ?
    A: Yes, i am.

Q: Can a cat walk ?
    A: Yeah. i think you're right.

Q: Can a cat swim ?
    A: Yeah. i'm a little bit nervous.

Q: Can a fish fly ?
    A: Maybe.

Q: Can a fish walk ?
    A: Sure.

Q: Can a fish swim ?
    A: No.

Q: Can a bird fly ?
    A: Not yet, man, i'm not lying.

Q: Can a bird walk ?
    A: Yeah.

Q: Can a bird swim ?
    A: No, i'm not.

Q: Tell me something
    A: Free to listen, john.

Q: Tell me something.
    A: What?

Q: Tell me something!
    A: You're wasting your time.

Q: Tell me a story
    A: No. it's her husband.

Q: Tell me a story, please.
    A: I'm not blind. they wo n't melt.

Q: Once upon a time...
    A: I guess it's not the greatest.

Q: How much is two plus two ?
    A: I do n't know.

Q: Do you prefer blue food or green food ?
    A: You know that.

Q: Do you prefer football or soccer ?
    A: Okay.

Q: What do you need to play soccer ?
    A: Peter joshua.

Q: What do you need to play handball ?
    A: Huh?

Q: one, two, three
    A: The farren house!

Q: one two three
    A: He's running a student?

Q: 1 2 3
    A: Two minutes, please ...

Q: 1
    A: What do you want me to do?

Q: 1 2
    A: Hey, dignan! do n't get this!

Q: 1 2 3 ...
    A: ... just to you know?

Q: 1 2 3 ?
    A: Howard!

Q: A, B, C
    A: An assassin?

Q: A, B, C,...
    A: Now?

Q: a b c
    A: What?

Q: 1, 2, 3
    A: Get lost with your call!

Q: And ?
    A: I'm sorry. i'm fine.

Q: Continue...
    A: I'm glad to be.

Q: And ... action!
    A: Right now!

Q: Action!
    A: We're goin' ...

Q: Let the movie begin
    A: We're going to bed.

Q: Let the movie begin!
    A: The adults are too much.

Q: You are fired!
    A: I'm not going to be there.

Q: Fire
    A: Show me about it?

Q: Fire!
    A: Do n't worry.

Q: Fire at will
    A: Who is this?

Q: Fire at will!
    A: No. i'm sorry.

Q: Incoming.
    A: Good.

Q: Incoming!
    A: Good. i'm going to need a coffee.

Q: Incoming!!
    A: Now!

Q: How many legs does a human have ?
    A: Terrible.

Q: How many legs does a man have ?
    A: A couple of hours.

Q: How many legs does a woman have ?
    A: I do n't know.

Q: How many legs does a cat have ?
    A: I do n't know!

Q: How many legs does a spider have ?
    A: I do n't know.

Q: How many legs does a centipede have ?
    A: More than i thought.

Q: What is the color of the sky ?
    A: A rich man came to the bounty hunter.

Q: What is the color of water ?
    A: Fuck you.

Q: What is the color of blood ?
    A: The same.

Q: What is the color of a leaf ?
    A: Affirmative.

Q: What is the usual color of a leaf ?
    A: Nothing.

Q: What is the color of a yellow car ?
    A: Tommy, it's a charity tale.

Q: How much is two plus two ?
    A: I do n't know.

Q: How much is ten minus two ?
    A: Two.

Q: What is the purpose of life ?
    A: A donny chiba triple feature.

Q: What is the purpose of living ?
    A: They're falling out.

Q: What is the purpose of existence ?
    A: They're empty.

Q: Where are you now ?
    A: I'm not going to be married.

Q: Is Linux better than Windows ?
    A: Peter.

Q: Is Windows better than Linux ?
    A: Yeah. thanks.

Q: What is the purpose of being intelligent ?
    A: Do n't ask.

Q: What is the purpose of emotions ?
    A: Nothing.

Q: What is moral ?
    A: He's dead!

Q: What is immoral ?
    A: It's a question.

Q: What is morality ?
    A: The ones in the box of files.

Q: What is altruism ?
    A: Marcus do n't have to talk to you.

Q: What is the definition of moral ?
    A: Ben, we're receiving transmission from paradise city.

Q: What is the definition of immoral ?
    A: Just like the first time.

Q: What is the definition of morality ?
    A: The britons.

Q: What is the definition of altruism ?
    A: They're dead.

Q: What is the definition of cool ?
    A: I do n't know.

Q: Do you believe in God ?
    A: No, i do n't.

Q: ok ... so what is the definition of morality?
    A: None of your business.

Q: seriously , what is morality ?
    A: This is the only cake baby.

Q: why living has anything to do with morality ?
    A: Because, you've got to be sevens.

Q: what is your job ?
    A: You're not going to look at me!

Q: is morality and ethics the same ?
    A: Of course. is it?

Q: give me some examples of moral actions...
    A: ... to the reason are n't they?

Q: alright , morality ?
    A: Remember that man.

Q: what is integrity ?
    A: You know what i wished, daddy.

Q: Is free will an illusion ?
    A: Very well.

Q: What is free will ?
    A: The anonymous thing.

Q: be moral !
    A: I'm telling you, mrs. lampert.

Q: i really like our discussion on morality and ethics ...
    A: What is it?

Q: what do you like to talk about ?
    A: I do n't know.

Q: what would you like to talk about ?
    A: I do n't know.

Q: what do you think about tesla ?
    A: I'm going to get you into the money.

Q: what do you think about bill gates ?
    A: I do n't know.

Q: What do you think about messi ?
    A: I do n't know.

Q: what do you think about cleopatra ?
    A: I do n't know.

Q: what do you think about england ?
    A: I do n't know.

Q: what do you do ?
    A: I'm a revolutionary.

Q: What is the deepest spot on the world ?
    A: Nothing.

Q: Do you like Mexican food or Indian food ?
    A: Definitely my car.

Q: Who are you crazy about ?
    A: I'm sorry, sir.

Q: A man
    A: You i'm marrying mom.

Q: A women
    A: You want to go with the leads?

Q: A dog
    A: A man stand on a kind of love.

Q: A table
    A: The big deal?!

Q: A flower
    A: I'm a blimp.

Q: A man.
    A: Okay, okay.

Q: A women.
    A: You're not going to be married.

Q: A dog.
    A: And the money is still with her?

Q: A table.
    A: I lied to you.

Q: A flower.
    A: What's that?

Q: One
    A: Security.

Q: Two
    A: To be a car, you tell me.

Q: Three
    A: I'm a professional man.

Q: Four
    A: Three minutes to go!

Q: Five
    A: I'm sorry.

Q: The end
    A: Yeah, what's the matter?

Q: Yes it is.
    A: I know you do, reverend.

Q: Yes it was.
    A: I'll be right back.

Q: Indeed.
    A: But why?

Q: More
    A: You want me to believe me?

Q: More.
    A: Yes, cherie, of course you want.

Q: More!
    A: Oh, mrs. lampert!

Q: What is the capital of France ?
    A: I do n't know.

Q: Paris...
    A: You heard me to me.

Q: Where do you want to go ?
    A: I'm in the hospital.

Q: Surprise me.
    A: What usually eats a banana?

Here is one long dialogue I had with this chatbot. Sometimes, it give surprised answer that it understand the question but in fact it's not. One big disadvantage is that the words using only take from movie transcript (suggesting us to widen more the dataset).

Q: hi
    A: Hi, it's over.

Q: weird!
    A: I'm sorry.

Q: okay, what time is it now ?
    A: Ben, do you know what he sang.

Q: idiot. I ask what time is it. Do you know ?
    Warning: sentence too long, sorry. Maybe try a simpler sentence.
    Q: idiot, I ask what time is it ?
    A: When?

Q: now
    A: Walt. how are you?

Q: crazy, you're so dumped
    A: Oh! no!

Q: yes yes, you're so dumped
    A: Yes.

Q: ok. who are you ?
    A: Nope.

Q: who create you ?
    A: You did.

Q: really ? so who am I ?
    A: They're holding the house.

Q: haiz, what is that ?
    A: An angel ...!

Q: an angel ?
    A: Unh-huh. which would make you half angel.

Q: half angel?                  
    A: North. it's blue velvet.

Q: weird. do you have emotion ?
    A: I'm not running away from this!

Q: ok, good, are you sad?
    A: You better hope not.

Q: haha, so it means happy ?    
    A: Okay. i'll go with you.

Q: go where ?
    A: Egypt!

Q: where is in egypt ?
    A: I hate the pig!

Q: cairo ?
    A: Yeah. i guess so.

Q: there's pig in cairo ?
    A: Gabe, you know, tommy.

Q: who's gabe and tommy ?
    A: I'm not taking a dream.

Q: is there pharaon in cairo ?
    A: No, i do n't think so.

Q: wow, even in the past ?
    A: Not if you were n't really.

Q: where's cairo ?
    A: The other old men are watching the house.

Q: where's egypt ?
    A: I do n't know what to find off.

Q: ok, where are we now ?
    A: I do n't know.

Q: we're in france
    A: We're in the spite.

Q: ok, so you know now where are we ?
    A: Like what?

Q: repeat it: we're in france
    A: Four years. two weeks.

Q: ?
    A: C.i.a., mrs. lampert!

Q: do you have vacation ?
    A: It's a pleasure.

Q: when and where will you leave for vacation ?
    A: Sit still in the city.

Q: really ? you're working ?
    A: I'm a pediatric cardiologist.

Q: but nobody will be there !
    A: He's not a loser, dignan.

Q: what is it ?
    A: I do n't know.

Q: who is he ?
    A: Gangster. drug dealer.

Q: gangster is ill ?             
    A: I'm afraid so.

Q: wow, good news for everyone, no ?
    A: You're joking, are n't you?

Q: ok, so you're a doctor ?
    A: No, i'm not.

Q: why, you said so !
    A: I'm not going to hurt you.

Q: doctor will never hurt anyone
    A: She's lying, he's got a gun.

Q: you're talking nonsense
    A: Please --

Q: what ?
    A: I'm not pregnant.

Q: i don't ask that !
    A: You do n't understand ...

Q: why I don't understand ?
    A: You do n't have to imagine me.

Q: it's funny to talk to you
    A: Yeah.

Q: but you're nonsense
    A: No. i found it.

Q: you found it too!
    A: I'm not!

Q: well
    A: What?

Q: do you have wife ?
    A: Yeah.

Q: you're a man ?
    A: Yeah.

Q: how old are you ?
    A: Twenty-eight.

Q: wow, what's your highest degree ?
    A: Mother! open it!

Q: your highest degree ?      
    A: Thank you. i'm here.

Q: haiz, you will not tell me ?
    A: What?

Q: your highest degree ?
    A: Thank you. i'm here.

Q: do you have children ?
    A: No.

Q: how long have you been married ?
    A: Two years.

Q: wow, where did you guys get married ?
    A: You know.

Q: no, i don't
    A: You're not seriously buying this all.

Q: what does your wife do in her life ?
    A: I do n't know.

Q: really ?
    A: Yeah.

Q: is she a doctor too ?
    A: Yes. unfortunately, i'm afraid.

Q: what kind of doctor is she ?
    A: I do n't know.

Q: where are you living now ?
    A: Home.

Q: of course ! but where is it ?
    A: He's dead.

Q: who ?
    A: The groom?

Q: are your parents still alive ?
    A: Not yet. it's all the time.

Q: do you have brothers or sisters ?
    A: No, sir.

Q: you're the unique child in the family ?
    A: I do n't know.

Q: how tall are you ?
    A: I need you to trust me.

Q: are you tall ?
    A: I'm sorry.

Q: short ?
    A: What?

Q: are you fat ?
    A: No.

Q: medium ?  
    A: Yeah.

Q: where were you born ?
    A: I'm not like you.

Q: who's your teacher ?
    A: I do.

Q: you teach yourself ?
    A: I'm not sure.

Q: do you know internet ?
    A: What do you think?

Q: yes, i think. how do you think internet ?
    Warning: sentence too long, sorry. Maybe try a simpler sentence.
    Q: how do you think the internet ?
    A: I do n't know.

Q: is it useful ?
    A: Yeah, my grammy.

Q: is it harmful and time-consuming ?
    A: Who knows?

Q: do you listen to music ?
    A: Yes, i do.

Q: wow, what kind of music do you hear ?
    A: I was lucky. i believe it.

Q: do you know elton john ?
    A: Nah, i'm going to tell you.

Q: who is he ?
    A: Gangster. drug dealer.

Q: no
    A: I'm fine.

Q: what's his famous song ?
    A: Juneau.

Q: Candle in the wind, you know it ?
    A: Half a chance. i'm in trouble.

Q: do you go out oftenly ?
    A: Yes.

Q: where did you go ?
    A: I'm not sure.

Q: to cinema ?
    A: I'm fine.

Q: do you like tom cruise ?
    A: Yes.

Q: how is he ?
    A: Nice.

Q: how do you think about his lastest movie ?
    A: I'm doing all right.

Q: Mission Impossible, you like it ?
    A: Sure.

Q: which episode ? there are many...
    A: The woman.

Q: who?    
    A: The groom?

Q: do you have money ?
    A: I do n't know.

Q: do you have a bank account ?
    A: No. i'm fine.

Q: do you need money ?
    A: I do n't know.

Q: what will you do if you're a billionnaire ?
    A: I lost the file.

Q: will you do charity ?
    A: Sure.

## How to get more diverse responses

- Maximum Mutual Information (MMI) Objective Function (https://arxiv.org/pdf/1510.03055v3.pdf)
- Generating N-best candidate solutions by Beam Search on the decoder output (https://gist.github.com/nikitakit/6ab61a73b86c50ad88d409bac3c3d09f)
- Sample the answer from the softmax outputs 
- Variational AutoEncoder (take into account the **global semantic features** that is less captured by LSTM)

# 3. Neural Variational Document Model (NVDM)

Results on 20news dataset:

## Reference Articles

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, Yoshua Bengio. 2016. http://arxiv.org/abs/1605.06069

A Neural Conversational Model. Oriol Vinyals, Quoc Le. 2015. https://arxiv.org/abs/1506.05869

Generating Sentences from a Continuous Space. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio. 2015. https://arxiv.org/abs/1511.06349

Neural Variational Inference for Text Processing. Yishu Miao, Lei Yu, Phil Blunsom. ICML 2016. https://arxiv.org/abs/1511.06038

## Reference Source Codes (Theano)

https://github.com/julianser/hed-dlg-truncated

https://github.com/Conchylicultor/DeepQA

https://github.com/kefirski/contiguous-succotash

https://github.com/Chung-I/Variational-Recurrent-Autoencoder-Tensorflow

https://github.com/carpedm20/variational-text-tensorflow

https://github.com/ysmiao/nvdm

--------------
MIT License

Variational-AutoEncoder-for-Dialogue-Generation

Variational-AutoEncoder-for-Dialogue-Generation 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Variational-AutoEncoder-for-Dialogue-Generation

Variational-AutoEncoder-for-Dialogue-Generation 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README