i noticed, that currently the benchmark works like this:
- train once on pairsDevTrain.txt (1100 matching, 1100 non-matching pairs)
- test against all 10 splits in pairs.txt (300 matching, 300 non-matching per split, 6000 all)
shouldn't pairsDevTrain.txt be tested against pairsDevTest.txt, like mentioned here (3.a), while each split from pairs.txt should be trained against the 9 other splits ?
(admittedly, training will take ages this way, but i'm having doubts, that it does not get enough traindata for the pairs.txt splits this way)
@Dmitriy Anisimov, as the maintainer of the datasets repo, maybe can you shed some light on this ?