Getting Start with Recommender System

NVIDIA

Resource

NVIDIA

Getting Start with Recommender System

The Variational Autoencoder for collaborative filtering focuses on providing recommendations.

Back to File Browser

VAE-model.ipynb

Prepare dataset

After downloading the dataset in extracted folder, go back to the main workspace and run prepare_dataset.py . prepare_dataset.py uses ml-20m dataset and divides it to train, validation and test data. Train data are used in training process and we will test the model using validation and test data.

In [1]:

%%bash
python prepare_dataset.py

Preprocessing seed:  0

In this model the metrics of accuracy is recall. In information retrieval, recall is the fraction of the relevant documents that are successfully retrieved. For example, for a text search on a set of documents, recall is the number of correct results divided by the number of results that should have been returned.

In the field of machine learning, a confusion matrix also known as the error matrix is used to show the performance of the model on a set of test data. This matrix has 4 entries including true positive, true negetive, false positive and false negetive. Recall is true positive/ (true positive+ false negetive) which means the true positive output divided by total actual positive in the test dataset. In the next sections including train, test and inference cells, you can see the recall as the metrics function of this model.

For this model, the performance of the model is the number correct predict rate over all correct rates. The model predicts the rate of a movie for a user and we have all correct rates for movies so we find the recall by dividing the correct predicted model (True positive) by whole correct rates ( True positive+ False negetive).

Training the model

The training can be started by running the main.py script with the train argument. The resulting checkpoints, containing the trained model weights, are then stored in the directory specified by the --checkpoint_dir directory (by default no checkpoints are saved).

Additionally, a command-line argument called --results_dir (by default None) specifies where to save the following statistics in a JSON format:

a complete list of command-line arguments saved as <results_dir>/args.json, and
a dictionary of validation metrics and performance metrics recorded during training

When you run the training command in the next cell you can see the details of trianing process in the each epoch. Also, you can change the hyperparameters of the model for training by changing the arguments of main.py. In the last cell of this notebook see more details about main.py arguments.

After each 50 epochs we have inference and you can see the recall of the model inference after each 50 epochs. recall shows the performance of this model that is the percentage of correct predicts rate over all correct rates (true positive /true positive+ false negetive)

In [2]:

%%bash
mpirun --allow-run-as-root -np 1 -H localhost:8 python main.py --train --amp --checkpoint_dir ./checkpoints

DLL 2020-11-12 21:34:48.556530 - PARAMETER train : True  test : False  inference_benchmark : False  amp : True  epochs : 400  batch_size_train : 24576  batch_size_validation : 10000  validation_step : 50  warm_up_epochs : 5  total_anneal_steps : 15000  anneal_cap : 0.1  lam : 1.0  lr : 0.004  beta1 : 0.9  beta2 : 0.9  top_results : 100  xla : False  trace : False  activation : tanh  log_path : ./vae_cf.log  seed : 0  data_dir : /data  checkpoint_dir : ./checkpoints  world_size : 1  local_batch_size : 24576 
DLL 2020-11-12 21:34:55.632787 - (1,) train_epoch_time : 1.3805668354034424  train_throughput : 71205.53491441265 
DLL 2020-11-12 21:34:56.116540 - (2,) train_epoch_time : 0.48351335525512695  train_throughput : 203311.8608443187 
DLL 2020-11-12 21:34:56.594167 - (3,) train_epoch_time : 0.47730565071105957  train_throughput : 205956.0783610103 
DLL 2020-11-12 21:34:57.067740 - (4,) train_epoch_time : 0.47342419624328613  train_throughput : 207644.6467672365 
DLL 2020-11-12 21:34:57.542833 - (5,) train_epoch_time : 0.4748830795288086  train_throughput : 207006.74384427382 
DLL 2020-11-12 21:34:58.021251 - (6,) train_epoch_time : 0.4781332015991211  train_throughput : 205599.61046675136 
DLL 2020-11-12 21:34:58.496483 - (7,) train_epoch_time : 0.474776029586792  train_throughput : 207053.41860994147 
DLL 2020-11-12 21:34:58.971247 - (8,) train_epoch_time : 0.4745321273803711  train_throughput : 207159.84087880817 
DLL 2020-11-12 21:34:59.453350 - (9,) train_epoch_time : 0.48178935050964355  train_throughput : 204039.37923495538 
DLL 2020-11-12 21:34:59.928088 - (10,) train_epoch_time : 0.4745934009552002  train_throughput : 207133.09498645877 
DLL 2020-11-12 21:35:00.400375 - (11,) train_epoch_time : 0.47209787368774414  train_throughput : 208228.00838332184 
DLL 2020-11-12 21:35:00.888168 - (12,) train_epoch_time : 0.4876265525817871  train_throughput : 201596.89721472247 
DLL 2020-11-12 21:35:01.356910 - (13,) train_epoch_time : 0.4685966968536377  train_throughput : 209783.80910504892 
DLL 2020-11-12 21:35:01.825493 - (14,) train_epoch_time : 0.46843576431274414  train_throughput : 209855.88097489672 
DLL 2020-11-12 21:35:02.291843 - (15,) train_epoch_time : 0.46618008613586426  train_throughput : 210871.2982891125 
DLL 2020-11-12 21:35:02.760501 - (16,) train_epoch_time : 0.46848249435424805  train_throughput : 209834.94833782708 
DLL 2020-11-12 21:35:03.229495 - (17,) train_epoch_time : 0.4688446521759033  train_throughput : 209672.86188244255 
DLL 2020-11-12 21:35:03.706421 - (18,) train_epoch_time : 0.4767630100250244  train_throughput : 206190.492829635 
DLL 2020-11-12 21:35:04.519359 - (19,) train_epoch_time : 0.8127946853637695  train_throughput : 120945.67271438746 
DLL 2020-11-12 21:35:04.987256 - (20,) train_epoch_time : 0.4677314758300781  train_throughput : 210171.8722810795 
DLL 2020-11-12 21:35:05.453060 - (21,) train_epoch_time : 0.4656500816345215  train_throughput : 211111.3127156212 
DLL 2020-11-12 21:35:05.941785 - (22,) train_epoch_time : 0.48844337463378906  train_throughput : 201259.76746783295 
DLL 2020-11-12 21:35:06.410180 - (23,) train_epoch_time : 0.46825551986694336  train_throughput : 209936.66028311523 
DLL 2020-11-12 21:35:06.877457 - (24,) train_epoch_time : 0.46712684631347656  train_throughput : 210443.9099910579 
DLL 2020-11-12 21:35:07.348372 - (25,) train_epoch_time : 0.47074365615844727  train_throughput : 208827.030835041 
DLL 2020-11-12 21:35:07.819000 - (26,) train_epoch_time : 0.4704627990722656  train_throughput : 208951.6964866333 
DLL 2020-11-12 21:35:08.289277 - (27,) train_epoch_time : 0.47008275985717773  train_throughput : 209120.62384476105 
DLL 2020-11-12 21:35:08.759090 - (28,) train_epoch_time : 0.4696533679962158  train_throughput : 209311.81739293324 
DLL 2020-11-12 21:35:09.226803 - (29,) train_epoch_time : 0.46756649017333984  train_throughput : 210246.03359311738 
DLL 2020-11-12 21:35:09.696231 - (30,) train_epoch_time : 0.469280481338501  train_throughput : 209478.1349516462 
DLL 2020-11-12 21:35:10.168149 - (31,) train_epoch_time : 0.4717249870300293  train_throughput : 208392.6073514145 
DLL 2020-11-12 21:35:10.642627 - (32,) train_epoch_time : 0.47432923316955566  train_throughput : 207248.45344891457 
DLL 2020-11-12 21:35:11.106497 - (33,) train_epoch_time : 0.4636847972869873  train_throughput : 212006.08813395482 
DLL 2020-11-12 21:35:11.571760 - (34,) train_epoch_time : 0.465071439743042  train_throughput : 211373.9774136942 
DLL 2020-11-12 21:35:12.038391 - (35,) train_epoch_time : 0.4664182662963867  train_throughput : 210763.61520012267 
DLL 2020-11-12 21:35:12.511295 - (36,) train_epoch_time : 0.4726989269256592  train_throughput : 207963.23917921682 
DLL 2020-11-12 21:35:12.985156 - (37,) train_epoch_time : 0.47369813919067383  train_throughput : 207524.56441554756 
DLL 2020-11-12 21:35:13.454346 - (38,) train_epoch_time : 0.4690372943878174  train_throughput : 209586.745395812 
DLL 2020-11-12 21:35:13.920250 - (39,) train_epoch_time : 0.46574831008911133  train_throughput : 211066.78837158112 
DLL 2020-11-12 21:35:14.732242 - (40,) train_epoch_time : 0.8118412494659424  train_throughput : 121087.71273283762 
DLL 2020-11-12 21:35:15.199895 - (41,) train_epoch_time : 0.46747493743896484  train_throughput : 210287.20927490346 
DLL 2020-11-12 21:35:15.667515 - (42,) train_epoch_time : 0.46744275093078613  train_throughput : 210301.68893250372 
DLL 2020-11-12 21:35:16.130162 - (43,) train_epoch_time : 0.4624931812286377  train_throughput : 212552.32290960528 
DLL 2020-11-12 21:35:16.596183 - (44,) train_epoch_time : 0.4658694267272949  train_throughput : 211011.91527115175 
DLL 2020-11-12 21:35:17.064915 - (45,) train_epoch_time : 0.4685835838317871  train_throughput : 209789.67977522945 
DLL 2020-11-12 21:35:17.536714 - (46,) train_epoch_time : 0.47162890434265137  train_throughput : 208435.0621746021 
DLL 2020-11-12 21:35:18.008361 - (47,) train_epoch_time : 0.47149038314819336  train_throughput : 208496.29921105356 
DLL 2020-11-12 21:35:18.495584 - (48,) train_epoch_time : 0.48704028129577637  train_throughput : 201839.5680506365 
DLL 2020-11-12 21:35:18.963122 - (49,) train_epoch_time : 0.467388391494751  train_throughput : 210326.14799356653 
DLL 2020-11-12 21:35:19.430675 - (50,) train_epoch_time : 0.4674108028411865  train_throughput : 210316.06330545386 
DLL 2020-11-12 21:35:19.586096 - (50,) inference_throughput : 74204.47388347522 
DLL 2020-11-12 21:35:19.661624 - (50,) valid_time : 0.2307882308959961 
DLL 2020-11-12 21:35:19.661736 - (50,) ndcg@100 : 0.40235925027568464  recall@20 : 0.378038386439412  recall@50 : 0.5135686857887355 
DLL 2020-11-12 21:35:20.128822 - (51,) train_epoch_time : 0.46701788902282715  train_throughput : 210493.0074642067 
DLL 2020-11-12 21:35:20.599327 - (52,) train_epoch_time : 0.47036194801330566  train_throughput : 208996.49815469162 
DLL 2020-11-12 21:35:21.067014 - (53,) train_epoch_time : 0.46752023696899414  train_throughput : 210266.83387508529 
DLL 2020-11-12 21:35:21.534192 - (54,) train_epoch_time : 0.46701931953430176  train_throughput : 210492.36271001791 
DLL 2020-11-12 21:35:22.001720 - (55,) train_epoch_time : 0.4673483371734619  train_throughput : 210344.17410051316 
DLL 2020-11-12 21:35:22.469362 - (56,) train_epoch_time : 0.46749162673950195  train_throughput : 210279.70208924715 
DLL 2020-11-12 21:35:22.940873 - (57,) train_epoch_time : 0.4713630676269531  train_throughput : 208552.61421923683 
DLL 2020-11-12 21:35:23.406237 - (58,) train_epoch_time : 0.46520256996154785  train_throughput : 211314.39580853024 
DLL 2020-11-12 21:35:23.873619 - (59,) train_epoch_time : 0.4671745300292969  train_throughput : 210422.4303363355 
DLL 2020-11-12 21:35:24.730542 - (60,) train_epoch_time : 0.8567581176757812  train_throughput : 114739.50228412157 
DLL 2020-11-12 21:35:25.191645 - (61,) train_epoch_time : 0.4609556198120117  train_throughput : 213261.31144705563 
DLL 2020-11-12 21:35:25.659172 - (62,) train_epoch_time : 0.467379093170166  train_throughput : 210330.33235016552 
DLL 2020-11-12 21:35:26.124587 - (63,) train_epoch_time : 0.4652571678161621  train_throughput : 211289.5980978052 
DLL 2020-11-12 21:35:26.594097 - (64,) train_epoch_time : 0.4693570137023926  train_throughput : 209443.97788914706 
DLL 2020-11-12 21:35:27.062954 - (65,) train_epoch_time : 0.46870970726013184  train_throughput : 209733.2282163333 
DLL 2020-11-12 21:35:27.531914 - (66,) train_epoch_time : 0.46880412101745605  train_throughput : 209690.9894619711 
DLL 2020-11-12 21:35:28.004273 - (67,) train_epoch_time : 0.4721343517303467  train_throughput : 208211.92027167944 
DLL 2020-11-12 21:35:28.472404 - (68,) train_epoch_time : 0.46798181533813477  train_throughput : 210059.44414522088 
DLL 2020-11-12 21:35:28.940662 - (69,) train_epoch_time : 0.4680445194244385  train_throughput : 210031.30240876644 
DLL 2020-11-12 21:35:29.408140 - (70,) train_epoch_time : 0.4672994613647461  train_throughput : 210366.1744289274 
DLL 2020-11-12 21:35:29.878320 - (71,) train_epoch_time : 0.4700286388397217  train_throughput : 209144.7028476096 
DLL 2020-11-12 21:35:30.343025 - (72,) train_epoch_time : 0.46455979347229004  train_throughput : 211606.77566441963 
DLL 2020-11-12 21:35:30.815767 - (73,) train_epoch_time : 0.47258996963500977  train_throughput : 208011.18584027936 
DLL 2020-11-12 21:35:31.282571 - (74,) train_epoch_time : 0.4666450023651123  train_throughput : 210661.20820272923 
DLL 2020-11-12 21:35:31.748567 - (75,) train_epoch_time : 0.4658374786376953  train_throughput : 211026.38690103302 
DLL 2020-11-12 21:35:32.215956 - (76,) train_epoch_time : 0.46721744537353516  train_throughput : 210403.1023957315 
DLL 2020-11-12 21:35:32.720894 - (77,) train_epoch_time : 0.5047900676727295  train_throughput : 194742.34200609 
DLL 2020-11-12 21:35:33.189258 - (78,) train_epoch_time : 0.4682142734527588  train_throughput : 209955.1542396935 
DLL 2020-11-12 21:35:33.656073 - (79,) train_epoch_time : 0.4666602611541748  train_throughput : 210654.32003331094 
DLL 2020-11-12 21:35:34.126422 - (80,) train_epoch_time : 0.4701523780822754  train_throughput : 209089.658125258 
DLL 2020-11-12 21:35:34.941748 - (81,) train_epoch_time : 0.8151710033416748  train_throughput : 120593.10205713533 
DLL 2020-11-12 21:35:35.406881 - (82,) train_epoch_time : 0.4649806022644043  train_throughput : 211415.27091941115 
DLL 2020-11-12 21:35:35.872285 - (83,) train_epoch_time : 0.46524810791015625  train_throughput : 211293.71259900625 
DLL 2020-11-12 21:35:36.337684 - (84,) train_epoch_time : 0.4652442932128906  train_throughput : 211295.4450685055 
DLL 2020-11-12 21:35:36.805833 - (85,) train_epoch_time : 0.46799445152282715  train_throughput : 210053.77239008798 
DLL 2020-11-12 21:35:37.271459 - (86,) train_epoch_time : 0.46544528007507324  train_throughput : 211204.20424962568 
DLL 2020-11-12 21:35:37.743104 - (87,) train_epoch_time : 0.47136569023132324  train_throughput : 208551.45386537828 
DLL 2020-11-12 21:35:38.210776 - (88,) train_epoch_time : 0.4674966335296631  train_throughput : 210277.45003807076 
DLL 2020-11-12 21:35:38.685337 - (89,) train_epoch_time : 0.47441649436950684  train_throughput : 207210.33346583933 
DLL 2020-11-12 21:35:39.151049 - (90,) train_epoch_time : 0.465564489364624  train_throughput : 211150.12473172022 
DLL 2020-11-12 21:35:39.617490 - (91,) train_epoch_time : 0.4662914276123047  train_throughput : 210820.94625538407 
DLL 2020-11-12 21:35:40.086154 - (92,) train_epoch_time : 0.46849489212036133  train_throughput : 209829.39548195683 
DLL 2020-11-12 21:35:40.553156 - (93,) train_epoch_time : 0.46684885025024414  train_throughput : 210569.2237376322 
DLL 2020-11-12 21:35:41.038538 - (94,) train_epoch_time : 0.48522233963012695  train_throughput : 202595.78335765563 
DLL 2020-11-12 21:35:41.504073 - (95,) train_epoch_time : 0.4653744697570801  train_throughput : 211236.3405996756 
DLL 2020-11-12 21:35:41.974773 - (96,) train_epoch_time : 0.470489501953125  train_throughput : 208939.83732243627 
DLL 2020-11-12 21:35:42.443392 - (97,) train_epoch_time : 0.46846723556518555  train_throughput : 209841.78302544565 
DLL 2020-11-12 21:35:42.916085 - (98,) train_epoch_time : 0.47254443168640137  train_throughput : 208031.23136839396 
DLL 2020-11-12 21:35:43.383851 - (99,) train_epoch_time : 0.46759843826293945  train_throughput : 210231.66879082218 
DLL 2020-11-12 21:35:43.852293 - (100,) train_epoch_time : 0.4682939052581787  train_throughput : 209919.45207103062 
DLL 2020-11-12 21:35:43.936103 - (100,) inference_throughput : 149693.924166286 
DLL 2020-11-12 21:35:44.011842 - (100,) valid_time : 0.15939712524414062 
DLL 2020-11-12 21:35:44.011978 - (100,) ndcg@100 : 0.41972841503818137  recall@20 : 0.39676701455149593  recall@50 : 0.5351118806433364 
DLL 2020-11-12 21:35:44.858258 - (101,) train_epoch_time : 0.8461921215057373  train_throughput : 116172.19955330616 
DLL 2020-11-12 21:35:45.326087 - (102,) train_epoch_time : 0.46767258644104004  train_throughput : 210198.33714883198 
DLL 2020-11-12 21:35:45.793779 - (103,) train_epoch_time : 0.4674389362335205  train_throughput : 210303.40517224232 
DLL 2020-11-12 21:35:46.260653 - (104,) train_epoch_time : 0.46672844886779785  train_throughput : 210623.5440296567 
DLL 2020-11-12 21:35:46.730094 - (105,) train_epoch_time : 0.469282865524292  train_throughput : 209477.07070057385 
DLL 2020-11-12 21:35:47.196371 - (106,) train_epoch_time : 0.4661214351654053  train_throughput : 210897.8317315881 
DLL 2020-11-12 21:35:47.675108 - (107,) train_epoch_time : 0.47858428955078125  train_throughput : 205405.8232715331 
DLL 2020-11-12 21:35:48.141641 - (108,) train_epoch_time : 0.4663350582122803  train_throughput : 210801.22171567692 
DLL 2020-11-12 21:35:48.614495 - (109,) train_epoch_time : 0.4727027416229248  train_throughput : 207961.56092197399 
DLL 2020-11-12 21:35:49.080257 - (110,) train_epoch_time : 0.46560144424438477  train_throughput : 211133.36570408536 
DLL 2020-11-12 21:35:49.559185 - (111,) train_epoch_time : 0.4787778854370117  train_throughput : 205322.76654814903 
DLL 2020-11-12 21:35:50.022987 - (112,) train_epoch_time : 0.4636528491973877  train_throughput : 212020.69645462208 
DLL 2020-11-12 21:35:50.492584 - (113,) train_epoch_time : 0.46944332122802734  train_throughput : 209405.47144827698 
DLL 2020-11-12 21:35:50.963435 - (114,) train_epoch_time : 0.47068142890930176  train_throughput : 208854.63917239604 
DLL 2020-11-12 21:35:51.438190 - (115,) train_epoch_time : 0.4745805263519287  train_throughput : 207138.7141728229 
DLL 2020-11-12 21:35:51.905616 - (116,) train_epoch_time : 0.4672577381134033  train_throughput : 210384.9588385878 
DLL 2020-11-12 21:35:52.372196 - (117,) train_epoch_time : 0.4664309024810791  train_throughput : 210757.90535552634 
DLL 2020-11-12 21:35:52.841688 - (118,) train_epoch_time : 0.4693481922149658  train_throughput : 209447.91442804973 
DLL 2020-11-12 21:35:53.307094 - (119,) train_epoch_time : 0.46526598930358887  train_throughput : 211285.59202692128 
DLL 2020-11-12 21:35:53.780240 - (120,) train_epoch_time : 0.472980260848999  train_throughput : 207839.54032995887 
DLL 2020-11-12 21:35:54.246806 - (121,) train_epoch_time : 0.46641016006469727  train_throughput : 210767.27828219678 
DLL 2020-11-12 21:35:55.224332 - (122,) train_epoch_time : 0.9773805141448975  train_throughput : 100579.04631545207 
DLL 2020-11-12 21:35:55.685901 - (123,) train_epoch_time : 0.4614126682281494  train_throughput : 213050.06725864916 
DLL 2020-11-12 21:35:56.151790 - (124,) train_epoch_time : 0.4657306671142578  train_throughput : 211074.78407876255 
DLL 2020-11-12 21:35:56.621653 - (125,) train_epoch_time : 0.4696831703186035  train_throughput : 209298.53614579537 
DLL 2020-11-12 21:35:57.086275 - (126,) train_epoch_time : 0.46448850631713867  train_throughput : 211639.25191483856 
DLL 2020-11-12 21:35:57.553881 - (127,) train_epoch_time : 0.4674520492553711  train_throughput : 210297.50571549233 
DLL 2020-11-12 21:35:58.020319 - (128,) train_epoch_time : 0.4662923812866211  train_throughput : 210820.51507844473 
DLL 2020-11-12 21:35:58.488498 - (129,) train_epoch_time : 0.46802759170532227  train_throughput : 210038.89886452205 
DLL 2020-11-12 21:35:58.958663 - (130,) train_epoch_time : 0.46999168395996094  train_throughput : 209161.1476435711 
DLL 2020-11-12 21:35:59.422494 - (131,) train_epoch_time : 0.4636878967285156  train_throughput : 212004.67101593543 
DLL 2020-11-12 21:35:59.887966 - (132,) train_epoch_time : 0.465334415435791  train_throughput : 211254.5230679686 
DLL 2020-11-12 21:36:00.357103 - (133,) train_epoch_time : 0.46895718574523926  train_throughput : 209622.54761867237 
DLL 2020-11-12 21:36:00.822646 - (134,) train_epoch_time : 0.4653961658477783  train_throughput : 211226.49306946213 
DLL 2020-11-12 21:36:01.289706 - (135,) train_epoch_time : 0.4669070243835449  train_throughput : 210542.98793167717 
DLL 2020-11-12 21:36:01.758111 - (136,) train_epoch_time : 0.46825408935546875  train_throughput : 209937.30163747453 
DLL 2020-11-12 21:36:02.224597 - (137,) train_epoch_time : 0.4663369655609131  train_throughput : 210800.35952491846 
DLL 2020-11-12 21:36:02.691775 - (138,) train_epoch_time : 0.4670243263244629  train_throughput : 210490.10610146198 
DLL 2020-11-12 21:36:03.160028 - (139,) train_epoch_time : 0.4680368900299072  train_throughput : 210034.72609545465 
DLL 2020-11-12 21:36:03.627495 - (140,) train_epoch_time : 0.4673173427581787  train_throughput : 210358.12499445173 
DLL 2020-11-12 21:36:04.094711 - (141,) train_epoch_time : 0.46704912185668945  train_throughput : 210478.93122934477 
DLL 2020-11-12 21:36:04.940525 - (142,) train_epoch_time : 0.8456618785858154  train_throughput : 116245.04129758332 
DLL 2020-11-12 21:36:05.417958 - (143,) train_epoch_time : 0.4772801399230957  train_throughput : 205967.08678437732 
DLL 2020-11-12 21:36:05.896479 - (144,) train_epoch_time : 0.4783358573913574  train_throughput : 205512.5044066499 
DLL 2020-11-12 21:36:06.359335 - (145,) train_epoch_time : 0.46270203590393066  train_throughput : 212456.38093628475 
DLL 2020-11-12 21:36:06.830923 - (146,) train_epoch_time : 0.47139716148376465  train_throughput : 208537.53062614842 
DLL 2020-11-12 21:36:07.297664 - (147,) train_epoch_time : 0.466594934463501  train_throughput : 210683.81317305053 
DLL 2020-11-12 21:36:07.765436 - (148,) train_epoch_time : 0.4676215648651123  train_throughput : 210221.27161384496 
DLL 2020-11-12 21:36:08.231697 - (149,) train_epoch_time : 0.4660971164703369  train_throughput : 210908.83536125932 
DLL 2020-11-12 21:36:08.699257 - (150,) train_epoch_time : 0.46740150451660156  train_throughput : 210320.24726079663 
DLL 2020-11-12 21:36:08.777187 - (150,) inference_throughput : 153495.8207076252 
DLL 2020-11-12 21:36:08.854533 - (150,) valid_time : 0.15512585639953613 
DLL 2020-11-12 21:36:08.854653 - (150,) ndcg@100 : 0.4233379522310202  recall@20 : 0.3985709752360643  recall@50 : 0.536773034842413 
DLL 2020-11-12 21:36:09.335620 - (151,) train_epoch_time : 0.48089170455932617  train_throughput : 204420.2448659052 
DLL 2020-11-12 21:36:09.802769 - (152,) train_epoch_time : 0.4669923782348633  train_throughput : 210504.5062439118 
DLL 2020-11-12 21:36:10.269912 - (153,) train_epoch_time : 0.4669928550720215  train_throughput : 210504.29130192832 
DLL 2020-11-12 21:36:10.738061 - (154,) train_epoch_time : 0.4679872989654541  train_throughput : 210056.98277990363 
DLL 2020-11-12 21:36:11.206864 - (155,) train_epoch_time : 0.4686393737792969  train_throughput : 209764.70501664618 
DLL 2020-11-12 21:36:11.677671 - (156,) train_epoch_time : 0.4706604480743408  train_throughput : 208863.9493762452 
DLL 2020-11-12 21:36:12.144924 - (157,) train_epoch_time : 0.46707797050476074  train_throughput : 210465.93118867298 
DLL 2020-11-12 21:36:12.606089 - (158,) train_epoch_time : 0.4609804153442383  train_throughput : 213249.84040068436 
DLL 2020-11-12 21:36:13.075813 - (159,) train_epoch_time : 0.4695711135864258  train_throughput : 209348.48238254522 
DLL 2020-11-12 21:36:13.544656 - (160,) train_epoch_time : 0.46869659423828125  train_throughput : 209739.09605586575 
DLL 2020-11-12 21:36:14.012567 - (161,) train_epoch_time : 0.46772193908691406  train_throughput : 210176.15763739648 
DLL 2020-11-12 21:36:14.861623 - (162,) train_epoch_time : 0.8489086627960205  train_throughput : 115800.44392080956 
DLL 2020-11-12 21:36:15.327655 - (163,) train_epoch_time : 0.46589207649230957  train_throughput : 211001.6567358872 
DLL 2020-11-12 21:36:15.801089 - (164,) train_epoch_time : 0.47327208518981934  train_throughput : 207711.38437326672 
DLL 2020-11-12 21:36:16.266072 - (165,) train_epoch_time : 0.46481800079345703  train_throughput : 211489.2276809254 
DLL 2020-11-12 21:36:16.726115 - (166,) train_epoch_time : 0.4598982334136963  train_throughput : 213751.63646599994 
DLL 2020-11-12 21:36:17.190320 - (167,) train_epoch_time : 0.46405982971191406  train_throughput : 211834.75428378838 
DLL 2020-11-12 21:36:17.658259 - (168,) train_epoch_time : 0.4677884578704834  train_throughput : 210146.2709180769 
DLL 2020-11-12 21:36:18.126584 - (169,) train_epoch_time : 0.4681413173675537  train_throughput : 209987.87407354216 
DLL 2020-11-12 21:36:18.596071 - (170,) train_epoch_time : 0.46930813789367676  train_throughput : 209465.7903040051 
DLL 2020-11-12 21:36:19.074421 - (171,) train_epoch_time : 0.47820401191711426  train_throughput : 205569.16619310746 
DLL 2020-11-12 21:36:19.544936 - (172,) train_epoch_time : 0.470304012298584  train_throughput : 209022.24397266956 
DLL 2020-11-12 21:36:20.009069 - (173,) train_epoch_time : 0.46396374702453613  train_throughput : 211878.62334166665 
DLL 2020-11-12 21:36:20.477125 - (174,) train_epoch_time : 0.46788978576660156  train_throughput : 210100.76088525084 
DLL 2020-11-12 21:36:20.947258 - (175,) train_epoch_time : 0.46996307373046875  train_throughput : 209173.88087468952 
DLL 2020-11-12 21:36:21.414362 - (176,) train_epoch_time : 0.4669654369354248  train_throughput : 210516.65117903394 
DLL 2020-11-12 21:36:21.883249 - (177,) train_epoch_time : 0.46873903274536133  train_throughput : 209720.1067814697 
DLL 2020-11-12 21:36:22.352610 - (178,) train_epoch_time : 0.4691941738128662  train_throughput : 209516.66812300967 
DLL 2020-11-12 21:36:22.818080 - (179,) train_epoch_time : 0.46526336669921875  train_throughput : 211286.78300509977 
DLL 2020-11-12 21:36:23.289778 - (180,) train_epoch_time : 0.47151851654052734  train_throughput : 208483.85917321808 
DLL 2020-11-12 21:36:23.757508 - (181,) train_epoch_time : 0.46758127212524414  train_throughput : 210239.38694804857 
DLL 2020-11-12 21:36:24.226415 - (182,) train_epoch_time : 0.46871423721313477  train_throughput : 209731.20122079627 
DLL 2020-11-12 21:36:25.071743 - (183,) train_epoch_time : 0.8451719284057617  train_throughput : 116312.42909999356 
DLL 2020-11-12 21:36:25.540420 - (184,) train_epoch_time : 0.4685177803039551  train_throughput : 209819.14482781081 
DLL 2020-11-12 21:36:26.008110 - (185,) train_epoch_time : 0.46752142906188965  train_throughput : 210266.2977336739 
DLL 2020-11-12 21:36:26.476459 - (186,) train_epoch_time : 0.46819162368774414  train_throughput : 209965.31126657425 
DLL 2020-11-12 21:36:26.944176 - (187,) train_epoch_time : 0.4675755500793457  train_throughput : 210241.9597930606 
DLL 2020-11-12 21:36:27.417439 - (188,) train_epoch_time : 0.47312140464782715  train_throughput : 207777.5366624421 
DLL 2020-11-12 21:36:27.887701 - (189,) train_epoch_time : 0.47006773948669434  train_throughput : 209127.30600773887 
DLL 2020-11-12 21:36:28.353941 - (190,) train_epoch_time : 0.46606945991516113  train_throughput : 210921.35068857402 
DLL 2020-11-12 21:36:28.819674 - (191,) train_epoch_time : 0.46558332443237305  train_throughput : 211141.58270133418 
DLL 2020-11-12 21:36:29.287439 - (192,) train_epoch_time : 0.46761059761047363  train_throughput : 210226.20210564314 
DLL 2020-11-12 21:36:29.754942 - (193,) train_epoch_time : 0.46735525131225586  train_throughput : 210341.06223045255 
DLL 2020-11-12 21:36:30.218867 - (194,) train_epoch_time : 0.4637715816497803  train_throughput : 211966.41598931522 
DLL 2020-11-12 21:36:30.688001 - (195,) train_epoch_time : 0.4689791202545166  train_throughput : 209612.74341307578 
DLL 2020-11-12 21:36:31.154344 - (196,) train_epoch_time : 0.46619629859924316  train_throughput : 210863.9650193902 
DLL 2020-11-12 21:36:31.619830 - (197,) train_epoch_time : 0.46533846855163574  train_throughput : 211252.68303299925 
DLL 2020-11-12 21:36:32.086884 - (198,) train_epoch_time : 0.46689677238464355  train_throughput : 210547.61098030084 
DLL 2020-11-12 21:36:32.554705 - (199,) train_epoch_time : 0.4676523208618164  train_throughput : 210207.44603349722 
DLL 2020-11-12 21:36:33.021506 - (200,) train_epoch_time : 0.46663975715637207  train_throughput : 210663.57611500748 
DLL 2020-11-12 21:36:33.107578 - (200,) inference_throughput : 153333.65016578868 
DLL 2020-11-12 21:36:33.183137 - (200,) valid_time : 0.1613175868988037 
DLL 2020-11-12 21:36:33.183264 - (200,) ndcg@100 : 0.42552455162334535  recall@20 : 0.4002976536299098  recall@50 : 0.539919019847044 
DLL 2020-11-12 21:36:33.646837 - (201,) train_epoch_time : 0.4634983539581299  train_throughput : 212091.3680933596 
DLL 2020-11-12 21:36:34.113207 - (202,) train_epoch_time : 0.4662168025970459  train_throughput : 210854.6913204344 
DLL 2020-11-12 21:36:34.964270 - (203,) train_epoch_time : 0.8509061336517334  train_throughput : 115528.60663739762 
DLL 2020-11-12 21:36:35.432030 - (204,) train_epoch_time : 0.467498779296875  train_throughput : 210276.48488804753 
DLL 2020-11-12 21:36:35.898609 - (205,) train_epoch_time : 0.4663364887237549  train_throughput : 210800.5750719469 
DLL 2020-11-12 21:36:36.370988 - (206,) train_epoch_time : 0.472217321395874  train_throughput : 208175.3369601383 
DLL 2020-11-12 21:36:36.840367 - (207,) train_epoch_time : 0.4692230224609375  train_throughput : 209503.7866736041 
DLL 2020-11-12 21:36:37.311617 - (208,) train_epoch_time : 0.47110676765441895  train_throughput : 208666.07476144566 
DLL 2020-11-12 21:36:37.777223 - (209,) train_epoch_time : 0.4654560089111328  train_throughput : 211199.33595866134 
DLL 2020-11-12 21:36:38.244638 - (210,) train_epoch_time : 0.467254638671875  train_throughput : 210386.35438573576 
DLL 2020-11-12 21:36:38.710385 - (211,) train_epoch_time : 0.4655921459197998  train_throughput : 211137.58224120317 
DLL 2020-11-12 21:36:39.179992 - (212,) train_epoch_time : 0.4694530963897705  train_throughput : 209401.1111141583 
DLL 2020-11-12 21:36:39.653471 - (213,) train_epoch_time : 0.4733312129974365  train_throughput : 207685.4373863834 
DLL 2020-11-12 21:36:40.120354 - (214,) train_epoch_time : 0.466717004776001  train_throughput : 210628.7086050799 
DLL 2020-11-12 21:36:40.584144 - (215,) train_epoch_time : 0.46363162994384766  train_throughput : 212030.40010860778 
DLL 2020-11-12 21:36:41.052043 - (216,) train_epoch_time : 0.4677300453186035  train_throughput : 210172.5150733866 
DLL 2020-11-12 21:36:41.519982 - (217,) train_epoch_time : 0.467785120010376  train_throughput : 210147.77040753138 
DLL 2020-11-12 21:36:41.990316 - (218,) train_epoch_time : 0.4701848030090332  train_throughput : 209075.23886541135 
DLL 2020-11-12 21:36:42.457227 - (219,) train_epoch_time : 0.46675872802734375  train_throughput : 210609.88064531947 
DLL 2020-11-12 21:36:42.923782 - (220,) train_epoch_time : 0.46639537811279297  train_throughput : 210773.95834790237 
DLL 2020-11-12 21:36:43.390856 - (221,) train_epoch_time : 0.4669017791748047  train_throughput : 210545.35318700442 
DLL 2020-11-12 21:36:43.856902 - (222,) train_epoch_time : 0.4658973217010498  train_throughput : 210999.28121732856 
DLL 2020-11-12 21:36:44.325019 - (223,) train_epoch_time : 0.4679732322692871  train_throughput : 210063.29683282538 
DLL 2020-11-12 21:36:45.176941 - (224,) train_epoch_time : 0.8517093658447266  train_throughput : 115419.6536309096 
DLL 2020-11-12 21:36:45.647333 - (225,) train_epoch_time : 0.47023725509643555  train_throughput : 209051.91780230164 
DLL 2020-11-12 21:36:46.119638 - (226,) train_epoch_time : 0.4721362590789795  train_throughput : 208211.07913161907 
DLL 2020-11-12 21:36:46.585606 - (227,) train_epoch_time : 0.4658033847808838  train_throughput : 211041.83269566126 
DLL 2020-11-12 21:36:47.053205 - (228,) train_epoch_time : 0.4674370288848877  train_throughput : 210304.26330261613 
DLL 2020-11-12 21:36:47.527829 - (229,) train_epoch_time : 0.47446465492248535  train_throughput : 207189.30057300098 
DLL 2020-11-12 21:36:47.996390 - (230,) train_epoch_time : 0.4683876037597656  train_throughput : 209877.45877753798 
DLL 2020-11-12 21:36:48.464505 - (231,) train_epoch_time : 0.467970609664917  train_throughput : 210064.47407111534 
DLL 2020-11-12 21:36:48.932444 - (232,) train_epoch_time : 0.46778297424316406  train_throughput : 210148.7343763379 
DLL 2020-11-12 21:36:49.402222 - (233,) train_epoch_time : 0.4696321487426758  train_throughput : 209321.27466823705 
DLL 2020-11-12 21:36:49.869354 - (234,) train_epoch_time : 0.4669787883758545  train_throughput : 210510.63227496884 
DLL 2020-11-12 21:36:50.336551 - (235,) train_epoch_time : 0.46705007553100586  train_throughput : 210478.50145026672 
DLL 2020-11-12 21:36:50.807747 - (236,) train_epoch_time : 0.47104477882385254  train_throughput : 208693.534923483 
DLL 2020-11-12 21:36:51.272823 - (237,) train_epoch_time : 0.46492767333984375  train_throughput : 211439.33914242111 
DLL 2020-11-12 21:36:51.741502 - (238,) train_epoch_time : 0.46851253509521484  train_throughput : 209821.49384759125 
DLL 2020-11-12 21:36:52.214054 - (239,) train_epoch_time : 0.4723820686340332  train_throughput : 208102.73405225016 
DLL 2020-11-12 21:36:52.681037 - (240,) train_epoch_time : 0.4668307304382324  train_throughput : 210577.3968815595 
DLL 2020-11-12 21:36:53.161078 - (241,) train_epoch_time : 0.4798319339752197  train_throughput : 204871.73328709044 
DLL 2020-11-12 21:36:53.627420 - (242,) train_epoch_time : 0.4661874771118164  train_throughput : 210867.9551175964 
DLL 2020-11-12 21:36:54.095411 - (243,) train_epoch_time : 0.4677915573120117  train_throughput : 210144.87855417267 
DLL 2020-11-12 21:36:54.563886 - (244,) train_epoch_time : 0.4683208465576172  train_throughput : 209907.37594233002 
DLL 2020-11-12 21:36:55.429057 - (245,) train_epoch_time : 0.8649983406066895  train_throughput : 113646.46079095584 
DLL 2020-11-12 21:36:55.897066 - (246,) train_epoch_time : 0.46785879135131836  train_throughput : 210114.67950846488 
DLL 2020-11-12 21:36:56.365436 - (247,) train_epoch_time : 0.46822071075439453  train_throughput : 209952.2676850692 
DLL 2020-11-12 21:36:56.828869 - (248,) train_epoch_time : 0.4632837772369385  train_throughput : 212189.60134173685 
DLL 2020-11-12 21:36:57.297572 - (249,) train_epoch_time : 0.4685385227203369  train_throughput : 209809.8560375495 
DLL 2020-11-12 21:36:57.767051 - (250,) train_epoch_time : 0.46930837631225586  train_throughput : 209465.68389095427 
DLL 2020-11-12 21:36:57.844917 - (250,) inference_throughput : 153389.72575436748 
DLL 2020-11-12 21:36:57.920367 - (250,) valid_time : 0.15316271781921387 
DLL 2020-11-12 21:36:57.920493 - (250,) ndcg@100 : 0.4281573218663558  recall@20 : 0.40165849490636646  recall@50 : 0.5419149665277968 
DLL 2020-11-12 21:36:58.423793 - (251,) train_epoch_time : 0.503192663192749  train_throughput : 195360.55906750064 
DLL 2020-11-12 21:36:58.890669 - (252,) train_epoch_time : 0.46671414375305176  train_throughput : 210629.99978851018 
DLL 2020-11-12 21:36:59.363180 - (253,) train_epoch_time : 0.4722259044647217  train_throughput : 208171.5532133497 
DLL 2020-11-12 21:36:59.830050 - (254,) train_epoch_time : 0.4667210578918457  train_throughput : 210626.87945565165 
DLL 2020-11-12 21:37:00.298707 - (255,) train_epoch_time : 0.4685060977935791  train_throughput : 209824.37680739033 
DLL 2020-11-12 21:37:00.768413 - (256,) train_epoch_time : 0.4695403575897217  train_throughput : 209362.19520004874 
DLL 2020-11-12 21:37:01.232501 - (257,) train_epoch_time : 0.4639456272125244  train_throughput : 211886.898451074 
DLL 2020-11-12 21:37:01.695027 - (258,) train_epoch_time : 0.46237659454345703  train_throughput : 212605.91725466496 
DLL 2020-11-12 21:37:02.163412 - (259,) train_epoch_time : 0.4682278633117676  train_throughput : 209949.06049524157 
DLL 2020-11-12 21:37:02.631663 - (260,) train_epoch_time : 0.4680979251861572  train_throughput : 210007.33972684373 
DLL 2020-11-12 21:37:03.093889 - (261,) train_epoch_time : 0.46207237243652344  train_throughput : 212745.89407204685 
DLL 2020-11-12 21:37:03.561074 - (262,) train_epoch_time : 0.46700191497802734  train_throughput : 210500.20748764006 
DLL 2020-11-12 21:37:04.027736 - (263,) train_epoch_time : 0.4665091037750244  train_throughput : 210722.57583939334 
DLL 2020-11-12 21:37:04.496004 - (264,) train_epoch_time : 0.46808314323425293  train_throughput : 210013.97170759385 
DLL 2020-11-12 21:37:05.372149 - (265,) train_epoch_time : 0.8759877681732178  train_throughput : 112220.74505104433 
DLL 2020-11-12 21:37:05.838798 - (266,) train_epoch_time : 0.4664740562438965  train_throughput : 210738.40802970968 
DLL 2020-11-12 21:37:06.305016 - (267,) train_epoch_time : 0.4660656452178955  train_throughput : 210923.07705718325 
DLL 2020-11-12 21:37:06.772762 - (268,) train_epoch_time : 0.467573881149292  train_throughput : 210242.71021805095 
DLL 2020-11-12 21:37:07.239804 - (269,) train_epoch_time : 0.4668867588043213  train_throughput : 210552.12671216612 
DLL 2020-11-12 21:37:07.709432 - (270,) train_epoch_time : 0.4694390296936035  train_throughput : 209407.3857986663 
DLL 2020-11-12 21:37:08.175486 - (271,) train_epoch_time : 0.46590423583984375  train_throughput : 210996.1499336794 
DLL 2020-11-12 21:37:08.643002 - (272,) train_epoch_time : 0.4673423767089844  train_throughput : 210346.85682102016 
DLL 2020-11-12 21:37:09.108396 - (273,) train_epoch_time : 0.4652211666107178  train_throughput : 211305.94877308677 
DLL 2020-11-12 21:37:09.577851 - (274,) train_epoch_time : 0.4692342281341553  train_throughput : 209498.7835625125 
DLL 2020-11-12 21:37:10.048261 - (275,) train_epoch_time : 0.47025036811828613  train_throughput : 209046.08834941464 
DLL 2020-11-12 21:37:10.516539 - (276,) train_epoch_time : 0.4681224822998047  train_throughput : 209996.32300728105 
DLL 2020-11-12 21:37:10.985088 - (277,) train_epoch_time : 0.46840691566467285  train_throughput : 209868.80575942373 
DLL 2020-11-12 21:37:11.460654 - (278,) train_epoch_time : 0.4754161834716797  train_throughput : 206774.6185713426 
DLL 2020-11-12 21:37:11.927127 - (279,) train_epoch_time : 0.4663271903991699  train_throughput : 210804.77832710778 
DLL 2020-11-12 21:37:12.397783 - (280,) train_epoch_time : 0.47050905227661133  train_throughput : 208931.15557361746 
DLL 2020-11-12 21:37:12.865091 - (281,) train_epoch_time : 0.4671592712402344  train_throughput : 210429.30334876655 
DLL 2020-11-12 21:37:13.331550 - (282,) train_epoch_time : 0.46629977226257324  train_throughput : 210817.17351696466 
DLL 2020-11-12 21:37:13.803117 - (283,) train_epoch_time : 0.471423864364624  train_throughput : 208525.71842643613 
DLL 2020-11-12 21:37:14.268294 - (284,) train_epoch_time : 0.46502208709716797  train_throughput : 211396.41046653994 
DLL 2020-11-12 21:37:15.122473 - (285,) train_epoch_time : 0.8540263175964355  train_throughput : 115106.52303627592 
DLL 2020-11-12 21:37:15.590414 - (286,) train_epoch_time : 0.4676938056945801  train_throughput : 210188.80045675836 
DLL 2020-11-12 21:37:16.066213 - (287,) train_epoch_time : 0.47564220428466797  train_throughput : 206676.36116908974 
DLL 2020-11-12 21:37:16.529979 - (288,) train_epoch_time : 0.46360111236572266  train_throughput : 212044.35748301347 
DLL 2020-11-12 21:37:16.999421 - (289,) train_epoch_time : 0.4692695140838623  train_throughput : 209483.0306458652 
DLL 2020-11-12 21:37:17.466600 - (290,) train_epoch_time : 0.46703481674194336  train_throughput : 210485.37812613905 
DLL 2020-11-12 21:37:17.933490 - (291,) train_epoch_time : 0.4667396545410156  train_throughput : 210618.48729495803 
DLL 2020-11-12 21:37:18.402675 - (292,) train_epoch_time : 0.4689919948577881  train_throughput : 209606.98919777642 
DLL 2020-11-12 21:37:18.868847 - (293,) train_epoch_time : 0.46602368354797363  train_throughput : 210942.06897723116 
DLL 2020-11-12 21:37:19.335829 - (294,) train_epoch_time : 0.4668290615081787  train_throughput : 210578.1497030423 
DLL 2020-11-12 21:37:19.801455 - (295,) train_epoch_time : 0.46546149253845215  train_throughput : 211196.847807725 
DLL 2020-11-12 21:37:20.267464 - (296,) train_epoch_time : 0.46587300300598145  train_throughput : 211010.2954361102 
DLL 2020-11-12 21:37:20.734867 - (297,) train_epoch_time : 0.4672586917877197  train_throughput : 210384.52944319008 
DLL 2020-11-12 21:37:21.202436 - (298,) train_epoch_time : 0.4674198627471924  train_throughput : 210311.98679113146 
DLL 2020-11-12 21:37:21.668573 - (299,) train_epoch_time : 0.4659709930419922  train_throughput : 210965.92163010687 
DLL 2020-11-12 21:37:22.136319 - (300,) train_epoch_time : 0.4675874710083008  train_throughput : 210236.59977034514 
DLL 2020-11-12 21:37:22.214917 - (300,) inference_throughput : 152595.61092030967 
DLL 2020-11-12 21:37:22.289898 - (300,) valid_time : 0.15342116355895996 
DLL 2020-11-12 21:37:22.290021 - (300,) ndcg@100 : 0.42907790503405885  recall@20 : 0.4037500707531203  recall@50 : 0.5430026364813616 
DLL 2020-11-12 21:37:22.760225 - (301,) train_epoch_time : 0.4701387882232666  train_throughput : 209095.7020830111 
DLL 2020-11-12 21:37:23.228123 - (302,) train_epoch_time : 0.46772074699401855  train_throughput : 210176.6933192236 
DLL 2020-11-12 21:37:23.696295 - (303,) train_epoch_time : 0.46800684928894043  train_throughput : 210048.20794686401 
DLL 2020-11-12 21:37:24.162608 - (304,) train_epoch_time : 0.4661591053009033  train_throughput : 210880.78916005572 
DLL 2020-11-12 21:37:24.632284 - (305,) train_epoch_time : 0.4695253372192383  train_throughput : 209368.89281035398 
DLL 2020-11-12 21:37:25.573425 - (306,) train_epoch_time : 0.940995454788208  train_throughput : 104468.09227375653 
DLL 2020-11-12 21:37:26.042427 - (307,) train_epoch_time : 0.46876001358032227  train_throughput : 209710.72009570108 
DLL 2020-11-12 21:37:26.510454 - (308,) train_epoch_time : 0.4678773880004883  train_throughput : 210106.32811324793 
DLL 2020-11-12 21:37:26.982650 - (309,) train_epoch_time : 0.4720447063446045  train_throughput : 208251.46152202712 
DLL 2020-11-12 21:37:27.450625 - (310,) train_epoch_time : 0.4678173065185547  train_throughput : 210133.3119365926 
DLL 2020-11-12 21:37:27.918385 - (311,) train_epoch_time : 0.4676077365875244  train_throughput : 210227.4883589313 
DLL 2020-11-12 21:37:28.387572 - (312,) train_epoch_time : 0.4690361022949219  train_throughput : 209587.27807734537 
DLL 2020-11-12 21:37:28.856681 - (313,) train_epoch_time : 0.46895861625671387  train_throughput : 209621.90818600325 
DLL 2020-11-12 21:37:29.325021 - (314,) train_epoch_time : 0.46819210052490234  train_throughput : 209965.0974243026 
DLL 2020-11-12 21:37:29.791638 - (315,) train_epoch_time : 0.4664592742919922  train_throughput : 210745.08626548195 
DLL 2020-11-12 21:37:30.261543 - (316,) train_epoch_time : 0.4697444438934326  train_throughput : 209271.23519592173 
DLL 2020-11-12 21:37:30.740829 - (317,) train_epoch_time : 0.4791257381439209  train_throughput : 205173.6990394601 
DLL 2020-11-12 21:37:31.223618 - (318,) train_epoch_time : 0.4826393127441406  train_throughput : 203680.05134325527 
DLL 2020-11-12 21:37:31.695277 - (319,) train_epoch_time : 0.4714822769165039  train_throughput : 208499.88390424466 
DLL 2020-11-12 21:37:32.160067 - (320,) train_epoch_time : 0.46463537216186523  train_throughput : 211572.35520534968 
DLL 2020-11-12 21:37:32.627688 - (321,) train_epoch_time : 0.46747565269470215  train_throughput : 210286.8875273813 
DLL 2020-11-12 21:37:33.101029 - (322,) train_epoch_time : 0.4731874465942383  train_throughput : 207748.53751413318 
DLL 2020-11-12 21:37:33.564360 - (323,) train_epoch_time : 0.463153600692749  train_throughput : 212249.2405391312 
DLL 2020-11-12 21:37:34.032753 - (324,) train_epoch_time : 0.4682438373565674  train_throughput : 209941.89812506078 
DLL 2020-11-12 21:37:34.502507 - (325,) train_epoch_time : 0.46959638595581055  train_throughput : 209337.21583038438 
DLL 2020-11-12 21:37:35.355622 - (326,) train_epoch_time : 0.8529689311981201  train_throughput : 115249.21530485008 
DLL 2020-11-12 21:37:35.823367 - (327,) train_epoch_time : 0.46757936477661133  train_throughput : 210240.2445560558 
DLL 2020-11-12 21:37:36.284950 - (328,) train_epoch_time : 0.4614386558532715  train_throughput : 213038.06855587487 
DLL 2020-11-12 21:37:36.750050 - (329,) train_epoch_time : 0.46494603157043457  train_throughput : 211430.99053445293 
DLL 2020-11-12 21:37:37.218472 - (330,) train_epoch_time : 0.46827006340026855  train_throughput : 209930.14006955974 
DLL 2020-11-12 21:37:37.690343 - (331,) train_epoch_time : 0.47171854972839355  train_throughput : 208395.45117867753 
DLL 2020-11-12 21:37:38.156159 - (332,) train_epoch_time : 0.4656708240509033  train_throughput : 211101.90916589228 
DLL 2020-11-12 21:37:38.623991 - (333,) train_epoch_time : 0.467679500579834  train_throughput : 210195.2295923205 
DLL 2020-11-12 21:37:39.092124 - (334,) train_epoch_time : 0.46798181533813477  train_throughput : 210059.44414522088 
DLL 2020-11-12 21:37:39.560034 - (335,) train_epoch_time : 0.467756986618042  train_throughput : 210160.409811842 
DLL 2020-11-12 21:37:40.026733 - (336,) train_epoch_time : 0.466550350189209  train_throughput : 210703.94644465044 
DLL 2020-11-12 21:37:40.492367 - (337,) train_epoch_time : 0.46546101570129395  train_throughput : 211197.06416634866 
DLL 2020-11-12 21:37:40.962197 - (338,) train_epoch_time : 0.4696824550628662  train_throughput : 209298.85487598676 
DLL 2020-11-12 21:37:41.429948 - (339,) train_epoch_time : 0.4676094055175781  train_throughput : 210226.7380426004 
DLL 2020-11-12 21:37:41.897304 - (340,) train_epoch_time : 0.46720194816589355  train_throughput : 210410.08152023872 
DLL 2020-11-12 21:37:42.375439 - (341,) train_epoch_time : 0.4779834747314453  train_throughput : 205664.01391853983 
DLL 2020-11-12 21:37:42.855829 - (342,) train_epoch_time : 0.48024749755859375  train_throughput : 204694.45546253197 
DLL 2020-11-12 21:37:43.323400 - (343,) train_epoch_time : 0.46736764907836914  train_throughput : 210335.4825560812 
DLL 2020-11-12 21:37:43.789770 - (344,) train_epoch_time : 0.46622729301452637  train_throughput : 210849.9469526704 
DLL 2020-11-12 21:37:44.260709 - (345,) train_epoch_time : 0.47078585624694824  train_throughput : 208808.31209261128 
DLL 2020-11-12 21:37:44.729431 - (346,) train_epoch_time : 0.4685547351837158  train_throughput : 209802.59640627887 
DLL 2020-11-12 21:37:45.590790 - (347,) train_epoch_time : 0.861208438873291  train_throughput : 114146.58236351002 
DLL 2020-11-12 21:37:46.061114 - (348,) train_epoch_time : 0.4701728820800781  train_throughput : 209080.53983270185 
DLL 2020-11-12 21:37:46.529151 - (349,) train_epoch_time : 0.46788501739501953  train_throughput : 210102.90209187282 
DLL 2020-11-12 21:37:46.998039 - (350,) train_epoch_time : 0.4687323570251465  train_throughput : 209723.09363043652 
DLL 2020-11-12 21:37:47.076816 - (350,) inference_throughput : 152352.8343419227 
DLL 2020-11-12 21:37:47.153736 - (350,) valid_time : 0.15553927421569824 
DLL 2020-11-12 21:37:47.153853 - (350,) ndcg@100 : 0.43029137575489385  recall@20 : 0.40471762875024875  recall@50 : 0.5448074592912086 
DLL 2020-11-12 21:37:47.620288 - (351,) train_epoch_time : 0.4663684368133545  train_throughput : 210786.13439558793 
DLL 2020-11-12 21:37:48.088680 - (352,) train_epoch_time : 0.4682488441467285  train_throughput : 209939.65330365207 
DLL 2020-11-12 21:37:48.555273 - (353,) train_epoch_time : 0.4664335250854492  train_throughput : 210756.72033220812 
DLL 2020-11-12 21:37:49.023620 - (354,) train_epoch_time : 0.4681863784790039  train_throughput : 209967.66356031116 
DLL 2020-11-12 21:37:49.493165 - (355,) train_epoch_time : 0.4693598747253418  train_throughput : 209442.7012056051 
DLL 2020-11-12 21:37:49.960354 - (356,) train_epoch_time : 0.4670231342315674  train_throughput : 210490.64338482477 
DLL 2020-11-12 21:37:50.425961 - (357,) train_epoch_time : 0.4654574394226074  train_throughput : 211198.6868701563 
DLL 2020-11-12 21:37:50.893078 - (358,) train_epoch_time : 0.46696925163269043  train_throughput : 210514.93145703766 
DLL 2020-11-12 21:37:51.361764 - (359,) train_epoch_time : 0.4685380458831787  train_throughput : 209810.0695637218 
DLL 2020-11-12 21:37:51.830586 - (360,) train_epoch_time : 0.4686763286590576  train_throughput : 209748.16518099006 
DLL 2020-11-12 21:37:52.299882 - (361,) train_epoch_time : 0.4691433906555176  train_throughput : 209539.34758122303 
DLL 2020-11-12 21:37:52.762735 - (362,) train_epoch_time : 0.4627101421356201  train_throughput : 212452.65890711153 
DLL 2020-11-12 21:37:53.226533 - (363,) train_epoch_time : 0.46364283561706543  train_throughput : 212025.2755963899 
DLL 2020-11-12 21:37:53.693388 - (364,) train_epoch_time : 0.4667022228240967  train_throughput : 210635.37988986922 
DLL 2020-11-12 21:37:54.162447 - (365,) train_epoch_time : 0.4689161777496338  train_throughput : 209640.87968081792 
DLL 2020-11-12 21:37:54.627941 - (366,) train_epoch_time : 0.4653451442718506  train_throughput : 211249.6524570409 
DLL 2020-11-12 21:37:55.485469 - (367,) train_epoch_time : 0.8573703765869141  train_throughput : 114657.56537021506 
DLL 2020-11-12 21:37:55.951621 - (368,) train_epoch_time : 0.4659733772277832  train_throughput : 210964.84220802545 
DLL 2020-11-12 21:37:56.413581 - (369,) train_epoch_time : 0.46181654930114746  train_throughput : 212863.74459460226 
DLL 2020-11-12 21:37:56.886252 - (370,) train_epoch_time : 0.47249650955200195  train_throughput : 208052.3305732079 
DLL 2020-11-12 21:37:57.354191 - (371,) train_epoch_time : 0.46778178215026855  train_throughput : 210149.26991838508 
DLL 2020-11-12 21:37:57.819789 - (372,) train_epoch_time : 0.4654250144958496  train_throughput : 211213.40052271003 
DLL 2020-11-12 21:37:58.284635 - (373,) train_epoch_time : 0.4646918773651123  train_throughput : 211546.62861206356 
DLL 2020-11-12 21:37:58.751738 - (374,) train_epoch_time : 0.466947078704834  train_throughput : 210524.92773413367 
DLL 2020-11-12 21:37:59.223176 - (375,) train_epoch_time : 0.4712812900543213  train_throughput : 208588.80264198306 
DLL 2020-11-12 21:37:59.692511 - (376,) train_epoch_time : 0.46911168098449707  train_throughput : 209553.511423325 
DLL 2020-11-12 21:38:00.160543 - (377,) train_epoch_time : 0.46788573265075684  train_throughput : 210102.58090809727 
DLL 2020-11-12 21:38:00.625087 - (378,) train_epoch_time : 0.46439146995544434  train_throughput : 211683.47474046345 
DLL 2020-11-12 21:38:01.087228 - (379,) train_epoch_time : 0.46200060844421387  train_throughput : 212778.94055386318 
DLL 2020-11-12 21:38:01.554692 - (380,) train_epoch_time : 0.46728515625  train_throughput : 210372.6144200627 
DLL 2020-11-12 21:38:02.021441 - (381,) train_epoch_time : 0.4666004180908203  train_throughput : 210681.33715402256 
DLL 2020-11-12 21:38:02.488046 - (382,) train_epoch_time : 0.466447114944458  train_throughput : 210750.579970262 
DLL 2020-11-12 21:38:02.956527 - (383,) train_epoch_time : 0.4683187007904053  train_throughput : 209908.3377069661 
DLL 2020-11-12 21:38:03.427051 - (384,) train_epoch_time : 0.47035813331604004  train_throughput : 208998.19315752792 
DLL 2020-11-12 21:38:03.893036 - (385,) train_epoch_time : 0.4658317565917969  train_throughput : 211028.97904434346 
DLL 2020-11-12 21:38:04.360569 - (386,) train_epoch_time : 0.467393159866333  train_throughput : 210324.00223425045 
DLL 2020-11-12 21:38:04.829693 - (387,) train_epoch_time : 0.46897101402282715  train_throughput : 209616.36659961048 
DLL 2020-11-12 21:38:05.685964 - (388,) train_epoch_time : 0.8561162948608398  train_throughput : 114825.52147425151 
DLL 2020-11-12 21:38:06.155383 - (389,) train_epoch_time : 0.4692351818084717  train_throughput : 209498.35777685756 
DLL 2020-11-12 21:38:06.620485 - (390,) train_epoch_time : 0.46494579315185547  train_throughput : 211431.09895370755 
DLL 2020-11-12 21:38:07.087391 - (391,) train_epoch_time : 0.4667530059814453  train_throughput : 210612.46256635324 
DLL 2020-11-12 21:38:07.554917 - (392,) train_epoch_time : 0.46738243103027344  train_throughput : 210328.8302542819 
DLL 2020-11-12 21:38:08.029188 - (393,) train_epoch_time : 0.4741179943084717  train_throughput : 207340.7910690714 
DLL 2020-11-12 21:38:08.494942 - (394,) train_epoch_time : 0.4656026363372803  train_throughput : 211132.82513458334 
DLL 2020-11-12 21:38:08.963808 - (395,) train_epoch_time : 0.4687197208404541  train_throughput : 209728.74754177744 
DLL 2020-11-12 21:38:09.434516 - (396,) train_epoch_time : 0.4705486297607422  train_throughput : 208913.5825344645 
DLL 2020-11-12 21:38:09.902857 - (397,) train_epoch_time : 0.46817946434020996  train_throughput : 209970.76439167748 
DLL 2020-11-12 21:38:10.369855 - (398,) train_epoch_time : 0.46683335304260254  train_throughput : 210576.21388724752 
DLL 2020-11-12 21:38:10.837470 - (399,) train_epoch_time : 0.4674644470214844  train_throughput : 210291.92835168063 
DLL 2020-11-12 21:38:11.307017 - (400,) train_epoch_time : 0.46937036514282227  train_throughput : 209438.02016577587 
DLL 2020-11-12 21:38:11.385360 - (400,) inference_throughput : 153062.82273515192 
DLL 2020-11-12 21:38:11.458828 - (400,) valid_time : 0.15166068077087402 
DLL 2020-11-12 21:38:11.458941 - (400,) ndcg@100 : 0.42910413327954355  recall@20 : 0.40471937243913636  recall@50 : 0.5451935793360538 
DLL 2020-11-12 21:38:11.459409 - () total_train_time : 195.81907439231873  total_valid_time : 1.3204126358032227  average_train_epoch time : 0.4895476859807968  average_validation_time : 0.16505157947540283  total_elapsed_time : 197.25992059707642  mean_training_throughput : 205200.28050950405  mean_inference_throughput : 142766.10783436586  max_training_throughput : 213751.63646599994  max_inference_throughput : 153495.8207076252  final_ndcg@100 : 0.42910413327954355  final_recall@20 : 0.40471937243913636  final_recall@50 : 0.5451935793360538

WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From main.py:26: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From main.py:26: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

--------------------------------------------------------------------------
[[1268,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: ef95f396c947

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
[VAE| INFO]: Already processed, skipping.
[VAE| INFO]: Cropping each epoch from: 116677 to 98304 samples
[VAE| INFO]: XLA disabled

Test the model.

The model is exported to the default model_dir and can be loaded and tested using the following command. We use the weights of the trained model that has been saved in the checkpoints folder and test data to test the performance of the model on the unseen data.

In the preprocessing step the ml-20m dataset was divided to train, test and validation datasets. In this step, we use the test dataset and trained model to test the performance of the model and recall here shows the accuracy of the model prediction. It shows how much is the probability that the model predicts the rate of a movie for a user correctly.

In [3]:

%%bash
python main.py --test --amp --checkpoint_dir ./checkpoints

DLL 2020-11-12 21:38:43.866671 - PARAMETER train : False  test : True  inference_benchmark : False  amp : True  epochs : 400  batch_size_train : 24576  batch_size_validation : 10000  validation_step : 50  warm_up_epochs : 5  total_anneal_steps : 15000  anneal_cap : 0.1  lam : 1.0  lr : 0.004  beta1 : 0.9  beta2 : 0.9  top_results : 100  xla : False  trace : False  activation : tanh  log_path : ./vae_cf.log  seed : 0  data_dir : /data  checkpoint_dir : ./checkpoints  world_size : 1  local_batch_size : 24576 
DLL 2020-11-12 21:38:51.534564 - (0,) inference_throughput : 15378.013600892842 
ndcg@100:	0.4300323040406368
recall@20:	0.4015537189302785
recall@50:	0.541860369693171

WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From main.py:26: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From main.py:26: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

--------------------------------------------------------------------------
[[565,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: ef95f396c947

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
[VAE| INFO]: Already processed, skipping.
[VAE| INFO]: Cropping each epoch from: 116677 to 98304 samples
[VAE| INFO]: XLA disabled

Main.py

This model was train using some default hyperparameters and small dataset for 400 epochs. You can try larger dataset, more training epochs and different hyperparameters to enhance the model performance. Main.py --help command shows you differnt options that you have for working with this specific model. The main.py script provides an entry point to all the provided functionalities. This includes running training, testing and inference. The behavior of the script is controlled by command-line arguments listed below in the Parameters section. The prepare_dataset.py script can be used to preprocess the MovieLens 20m dataset.

Parameters

The most important command-line parameters include:

--data_dir which specifies the directory inside the docker container where the data will be stored, overriding the default location /data
--checkpoint_dir which controls if and where the checkpoints will be stored
--amp for enabling mixed precision training
There are also multiple parameters controlling the various hyperparameters of the training process, such as the learning rate, batch size etc.

To see the full list of available options and their descriptions, use the -h or --help command-line option.

In [5]:

%%bash
python main.py --help

usage: main.py [-h] [--train] [--test] [--inference_benchmark] [--amp]
               [--epochs EPOCHS] [--batch_size_train BATCH_SIZE_TRAIN]
               [--batch_size_validation BATCH_SIZE_VALIDATION]
               [--validation_step VALIDATION_STEP]
               [--warm_up_epochs WARM_UP_EPOCHS]
               [--total_anneal_steps TOTAL_ANNEAL_STEPS]
               [--anneal_cap ANNEAL_CAP] [--lam LAM] [--lr LR] [--beta1 BETA1]
               [--beta2 BETA2] [--top_results TOP_RESULTS] [--xla] [--trace]
               [--activation ACTIVATION] [--log_path LOG_PATH] [--seed SEED]
               [--data_dir DATA_DIR] [--checkpoint_dir CHECKPOINT_DIR]

Train a Variational Autoencoder for Collaborative Filtering in TensorFlow

optional arguments:
  -h, --help            show this help message and exit
  --train               Run training of VAE
  --test                Run validation of VAE
  --inference_benchmark
                        Measure inference latency and throughput on a variety
                        of batch sizes
  --amp                 Enable Automatic Mixed Precision
  --epochs EPOCHS       Number of epochs to train
  --batch_size_train BATCH_SIZE_TRAIN
                        Global batch size for training
  --batch_size_validation BATCH_SIZE_VALIDATION
                        Used both for validation and testing
  --validation_step VALIDATION_STEP
                        Train epochs for one validation
  --warm_up_epochs WARM_UP_EPOCHS
                        Number of epochs to omit during benchmark
  --total_anneal_steps TOTAL_ANNEAL_STEPS
                        Number of annealing steps
  --anneal_cap ANNEAL_CAP
                        Annealing cap
  --lam LAM             Regularization parameter
  --lr LR               Learning rate
  --beta1 BETA1         Adam beta1
  --beta2 BETA2         Adam beta2
  --top_results TOP_RESULTS
                        Number of results to be recommended
  --xla                 Enable XLA
  --trace               Save profiling traces
  --activation ACTIVATION
                        Activation function
  --log_path LOG_PATH   Path to the detailed training log to be created
  --seed SEED           Random seed for TensorFlow and numpy
  --data_dir DATA_DIR   Directory for storing the training data
  --checkpoint_dir CHECKPOINT_DIR
                        Path for saving a checkpoint after the training

WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From main.py:26: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From main.py:26: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

--------------------------------------------------------------------------
[[819,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: ef95f396c947

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------

In [ ]: