arXiv Analytics

Sign in

arXiv:1909.09868 [cs.LG]AbstractReferencesReviewsResources

On the Importance of Delexicalization for Fact Verification

Sandeep Suntwal, Mithun Paul, Rebecca Sharp, Mihai Surdeanu

Published 2019-09-21Version 1

In this work we aim to understand and estimate the importance that a neural network assigns to various aspects of the data while learning and making predictions. Here we focus on the recognizing textual entailment (RTE) task and its application to fact verification. In this context, the contributions of this work are as follows. We investigate the attention weights a state of the art RTE method assigns to input tokens in the RTE component of fact verification systems, and confirm that most of the weight is assigned to POS tags of nouns (e.g., NN, NNP etc.) or their phrases. To verify that these lexicalized models transfer poorly, we implement a domain transfer experiment where a RTE component is trained on the FEVER data, and tested on the Fake News Challenge (FNC) dataset. As expected, even though this method achieves high accuracy when evaluated in the same domain, the performance in the target domain is poor, marginally above chance.To mitigate this dependence on lexicalized information, we experiment with several strategies for masking out names by replacing them with their semantic category, coupled with a unique identifier to mark that the same or new entities are referenced between claim and evidence. The results show that, while the performance on the FEVER dataset remains at par with that of the model trained on lexicalized data, it improves significantly when tested in the FNC dataset. Thus our experiments demonstrate that our strategy is successful in mitigating the dependency on lexical information.

Comments: submitted as an extended abstract to blackboxnlp workshop under acl 2019
Categories: cs.LG, cs.AI
Related articles: Most relevant | Search more
arXiv:1708.00631 [cs.LG] (Published 2017-08-02)
On the Importance of Consistency in Training Deep Neural Networks
arXiv:2305.14375 [cs.LG] (Published 2023-05-20)
Learning to Rank the Importance of Nodes in Road Networks Based on Multi-Graph Fusion
arXiv:2102.13651 [cs.LG] (Published 2021-02-26)
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Baohe Zhang et al.