Reproduction of numbers

[nomisto]

I've cloned the repository and downloaded the models and data. How can I reproduce the CQDA numbers for e.g. 2i?

I tried:

PYTHONPATH=. python cli/cq
da-cli.py --checkpoint_path "./models/fb15k-237-betae/" --
do_test --data_path data/FB15k-237-betae --negative_sample_size 1 --hidden_dim 1000 --gamma 12.0 --batch_size 2000 
--test_batch_size 2000 --learning_rate 0.1 --cpu_num 0 --save_path models/fb15k-237-betae --max_steps 100000 --warm
_up_steps 9999999 --save_checkpoint_steps 50000 --valid_steps 2000 --log_steps 500 --test_log_steps 1000 --nentity 
14505 --nrelation 474 --geo cqd --print_on_screen --reg_weight 0.1 --optimizer adagrad --cqd-type discrete --cqd-t-
norm prod --cqd-k 5 --cqd-sigmoid-normalize-scores --use-qa-iterator --tasks 2i --seed 0 --beta_mode "(1600,2)" --b
ox_mode "(none,0.02)" --prefix None --evaluate_union DNF --score-transform deep --score-transform-hidden 1000 --sco
re-transform-input spo

However these do not get the correct results. (Most of the parameters come from the config.json). Am I missing something?

[nomisto]

Unfortunately the paper does not mention what batch size e.g. was used, would you please be so kind to release the used commands/parameters @pminervini?

[nomisto]

This is my current command, I figured some things out, like to freeze embeddings etc.

CUDA_VISIBLE_DEVICES=4 PYTHONPATH=. python cli/cqda-cli.py --checkpoint_path "./models/fb15k-237-betae/" --do_test --data_path data/FB15k-237-betae --negative_sample_size 1 --hidden_dim 1000 --gamma 12.0 --batch_size 100 --test_batch_size 100 --learning_rate 0.001 --cpu_num 0 --save_path models/fb15k-237-betae-adapt --max_steps 101000 --warm_up_steps 9999999 --save_checkpoint_steps 50000 --valid_steps 5 --log_steps 500 --test_log_steps 1000 --nentity 14505 --nrelation 474 --geo cqd --print_on_screen --reg_weight 0.1 --optimizer adagrad --cqd-type discrete --cqd-t-norm prod --cqd-k 8000 --cqd-test-k 8000 --cqd-sigmoid-normalize-scores --tasks 2i.3i.2in.3in --seed 0 --beta_mode "(1600,2)" --box_mode "(none,0.02)" --prefix None --evaluate_union DNF --score-transform deep --score-transform-hidden 1000 --freeze-embeddings --new_optimizer --do_train --do_valid --cuda

and the test MRR increases now, however I still don't reach 0.345 MRR on FB15k-237 and 2i. After 0.311 the MRR decreases again (see attached output.txt). Any hints? I had to decrease the LR to 0.001, batch size to 100, steps=1000 (I guess steps = epochs?), and validate every 5 steps. Do I miss anything major other than LR/batch size configuration? What is e.g. --use-qa-iterator, should I set it to true or false?

output.txt

It would be really nice to be able to reproduce the results and cite the approach in a publication.