Detecting and translating language ambiguity with multilingual LLMs
•
0 likes•91 views
Language is one of the most important landmarks in humans in history. However, most languages could be ambiguous, which means the same conveyed text or speech, results in different actions by different readers or listeners. In this project we propose a method to detect the ambiguity of a sentence using translation by multilingual LLMs. In this context, we hypothesize that a good machine translator should preserve the ambiguity of sentences in all target languages. Therefore, we investigate whether ambiguity is encoded in the hidden representation of a translation model or, instead, if only a single meaning is encoded. The potential applications of the proposed approach span i) detecting ambiguous sentences, ii) fine-tuning existing multilingual LLMs to preserve ambiguous information, and iii) developing AI systems that can generate ambiguity-free languages when needed.
Report
Share
More Related Content
Similar to Detecting and translating language ambiguity with multilingual LLMs
Similar to Detecting and translating language ambiguity with multilingual LLMs (20)
Recently uploaded
Recently uploaded (20)
Detecting and translating language ambiguity with multilingual LLMs
- 1. DETECTING AND TRANSLATING LANGUAGE AMBIGUITY WITH MULTILINGUAL LLMS Behrang Mehrparvar (behrang.mehrparvar@student.uva.nl) Sandro Pezzelle (s.pezzelle@uva.nl) (ILLC, UvA) Summer 2024
- 3. WHAT IS AMBIGUITY? 3 Mariano Ceccato, Nadzeya Kiyavitskaya, Nicola Zeni, Luisa Mich, and Daniel M Berry. 2004. Ambiguity identification and measurement in natural language texts. Publisher: University of Trento.
- 4. EXAMPLES • “Give me the bat!” ~ Lexical • “The professor said on Monday he would give an exam” ~ Syntactic • “Jane saw the man with a telescope” ~ Semantic • “I like you too!” ~ Pragmatic • “The prof said she would give us all A’s.” ~ Vagueness • “Proposal” to “voorstel” and “aanzoek” ~ Translational 4 Apurwa Yadav, Aarshil Patel, and Manan Shah. 2021b. A comprehensive review on resolving ambiguities in natural language processing. AI Open, 2:85–92.
- 5. RESEARCH QUESTIONS Question 1: Can a state-of-the-art Transformer-based MT model properly encode whether a sentence in the source language is (non-)ambiguous? Question 2: Are both semantic validity and ambiguity preserved by the translation when the sentence is translated into a target language, and then translated back? Question 3: Can we predict the ambiguity of a sentence by translating it into another language looking at the learned hidden representations? 5
- 6. APPROACH 6
- 7. STEP1&2: TRANSLATION • Model: • Facebook M2M100 418M • Many-to-many multilingual encoder-decoder (seq-to-seq) model • Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., ... & Joulin, A. (2021). Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107), 1- 48. • Source language: • English • Target languages (n=14): • {German, Greek, Persian, Spanish, French, Hindi, Italian, Korean, Dutch, Russian, Turkish, Croatian, Romanian, Chinese} 7
- 9. STEP3: MAPPING FUNCTION Idea: “Can we map source translation representation to target translation representation?” “What are the properties of this mapping function?” 9
- 10. STEP3: MAPPING FUNCTION (CONT.) Idea: Autoencoder as a mapping function! • Normalized reconstruction error • Model complexity 10
- 11. STEP3: MAPPING FUNCTION (CONT.) Idea: “An Auto-encoder should behave differently in mapping ambiguous and unambiguous sentences” “How reconstruction error changes over model size and target language is predictive of sentence ambiguity” 11
- 12. SAMPLE MAPPINGS Andrei picked up the chair or the bag and the telescope (ambiguous) Andrei picked up the chair, or both the bag and the telescope (unambiguous) 12
- 14. IS REC. ERROR INFORMATIVE? 14
- 15. STEP 4: CLASSIFICATION • Input: 1. Single language – All models 2. All languages - Best model 3. Only along languages 4. Whole mapping function info. • Input variable: 1. Reconstruction error 2. Reconstruction error differences • Output: 1. Ambiguous vs unambiguous 2. Ambiguity type (4 classes) • Model 1. Logistic regression 2. Neural network • Cross-validation • 10-fold 15
- 16. RESULTS Input Input variable Output Model Accuracy F-measure Engineering appr. Alpha Amb. Vs unamb. Logistic 52.53 % 0.525 Persian Differences Amb. Vs unamb. Logistic 57.81 % 0.578 Best AE Values Amb. Vs unamb. Logistic 66.67 % 0.667 Along languages Differences Amb. Vs unamb. Logistic 85.87 % 0.859 Whole Differences Amb. Type Logistic 92.83 % 0.928 Whole Differences Amb. Vs unamb. Logistic 88.19 % 0.882 Best AE Values Amb. Vs unamb. NN 73.20 % 0.732 Whole Values Amb. Vs unamb. NN 81.99 % 0.820 Whole Values Amb. Type NN 78.26 % - 16
- 17. FINDINGS 1. Single language translation is not informative enough in predicting ambiguity. • 57.8059 % 85.865 % • Adding more features … 2. Single best auto-encoder is not informative enough in predicting ambiguity. • 66.6667 % 88.1857 % • Adding more features about the gradual change over the size of the AE model … 3. Adding reconstruction error differences between languages improves accuracy. • 85.865 % 88.1857 % • Adding more features about the properties of the mapping function mesh … 17
- 18. FINDINGS (CONT.) 4. Reconstruction error differences is more informative than their values. • 81.9876 % 94.9367 % • The shape of the mapping function is informative not the position … 5. A simple linear model can perform relatively close to a complex NN model. • 88.1857 % 94.9367 % • More complex features learned … 6. Predicting more detailed classes improves the accuracy in linear models. • 88.1857 % 92.827 % (LR) • More detailed regions in the misclassified regions (see next slide) … • 94.9367 % 93.038 % (NN) • Nonlinear boundaries anyway … 18
- 21. MISCLASSIFICATION EXAMPLES • Italian (translation problem): • Source sentence: Andrei and Danny put down the yellow bag and chair (amb.) • Machine translation: Andrei e Danny mettono il sacchetto giallo e la sedia (unamb.) • Machine translation back: Andrei and Danny put the yellow bag and the chair. (unamb.) • Human translation: Andrei e Danny mettono il sacchetto e la sedia color giallo (amb.) • Persian (language problem): • Source sentence: Andrei left the person with a green bag (amb.) • Machine translation: مرد آندری را کرد ترک سبز چمدان با (unamb.) • Machine translation back: Andrew left the man with a green bag. (amb.) 21
- 22. RESEARCH QUESTIONS • Question 3: Predicting • Yes, we can predict language ambiguity using translation with LLMs with high accuracy. • Question 2: Preservation • Depends on the language. The problem could be either because of the translation or the target language itself. • Question 1: Encoding • Yes, as ambiguity could be preserved in translation and be predicted as well, the information shouldn’t have been lost in encoding. 22
- 23. CONTRIBUTIONS • Predicting ambiguity without direct use of semantics • Data complexity: Could be trained with small training data • Generalizability: Could be easily generalizable to new unseen samples • Robustness: No need to update classifier model due to change in input distribution 23
- 24. FUTURE DIRECTIONS • Interpretability: Detecting source of ambiguity (which word?) • Extendibility: Extending to source languages other than English • Analysis: Source of misclassification in all languages via data annotation 24
- 25. POTENTIAL APPLICATIONS 1. Automatic translation of critical documents e.g. legal, political, commercial, etc. • Ask the user for an unambiguous sentence 2. Fine-tuning existing multilingual LLMs • Prevent ambiguity 3. Developing AI systems for generating ambiguity-free languages (www.Synaptosearch.com) 1. Classify the sentence 2. Get the gradient w.r.t. input 3. Use as a loss function term 25
- 26. THANK YOU! 26