1. Gabatarwa
Software na Fassarar Injin (MT), musamman Fassarar Injin ta Jijiyoyi (NMT), ya zama cikakken sashi na rayuwar yau da kullun da aikace-aikace masu mahimmanci, tun daga kiwon lafiya har zuwa takaddun shari'a. Duk da ikirarin kusanci da aikin ɗan adam a cikin ma'auni kamar BLEU, ƙarfi da amincin waɗannan tsarin sun kasance babban abin damuwa. Fassarar da ba daidai ba na iya haifar da mummunan sakamako, gami da kuskuren ganewar asali na likita da rashin fahimtar siyasa. Wannan takarda ta magance babban kalubalen tabbatar da software na MT ta hanyar gabatar da Gwajin Tsarin Tsarin (SIT), wata sabuwar hanyar gwaji ta metamorphic.
2. Kalubalen Gwajin NMT
Gwajin tsarin NMT na zamani yana da wahala a zahiri saboda dalilai biyu na farko. Na farko, dabarunsu an ɓoye su a cikin hadaddun cibiyoyin sadarwa na jijiyoyi masu duhu tare da miliyoyin sigogi, wanda ke sa dabarun gwaji na al'ada da suka dogara da lambar ba su da tasiri. Na biyu, ba kamar ayyukan AI masu sauƙi ba (misali, rarraba hoto tare da fitarwa mai lakabi ɗaya), MT yana samar da hadaddun jimlolin harshe na halitta, wanda ke sa tabbatar da sakamako ya zama mai wahala musamman.
2.1. Iyakokin Gwajin Al'ada & AI
Binciken gwajin AI na yanzu sau da yawa yana mai da hankali kan nemo shigarwar "haram" ko adawa (misali, kurakuran rubutu, kurakuran tsari) waɗanda ke haifar da kuskuren rarrabuwa. Duk da haka, ga MT, matsalar ba kawai game da alamun da ba daidai ba ce amma game da raguwar ƙima a cikin ingancin fassara, rashin daidaituwa na tsari, da kurakuran ma'ana waɗanda ke da wahala a ayyana su da gano su ta atomatik.
3. Gwajin Tsarin Tsarin (SIT)
SIT wata hanyar gwaji ce ta metamorphic da ta dogara da mahimmin fahimtar cewa jimlolin tushe "masu kama" yakamata su samar da fassarori tare da tsarin jimloli masu kama. Yana canza matsalar tabbatarwa daga buƙatar fassarar tunani "daidai" zuwa duba daidaiton tsari a cikin shigarwar da ke da alaƙa.
3.1. Hanyar Aiki ta Asali
Tsarin SIT ya ƙunshi manyan matakai uku:
- Samar da Shigarwa: Ƙirƙiri saitin jimlolin tushe masu kama ta hanyar musanya kalma a cikin jimla ta asali tare da kalma mai ma'ana mai kama da tsari daidai (misali, ta amfani da WordNet ko abubuwan haɗe-haɗe na mahallin).
- Wakilcin Tsari: Wakilci tsarin duka jimlolin tushe da na fassara ta amfani da bishiyoyin nazarin tsari, ko dai bishiyoyin ƙungiya ko bishiyoyin dogaro.
- Duba Tsayayyen Tsari & Rahoton Kurakurai: Ƙididdige bambancin tsari tsakanin bishiyoyin nazarin fassarori don jimlolin tushe masu kama. Idan bambancin ya wuce ƙayyadaddun bakin kofa $δ$, ana ba da rahoton yuwuwar kuskure.
3.2. Aiwatar da Fasaha
Bambancin tsari $d(T_a, T_b)$ tsakanin bishiyoyin nazari biyu $T_a$ da $T_b$ ana iya auna shi ta amfani da nisan gyaran bishiya ko maki kamanceceniya da aka daidaita. Ana nuna alamar kuskure lokacin da $d(T_a, T_b) > δ$. Bakin kofa $δ$ ana iya daidaita shi bisa ga nau'in fassara da kuma hankalin da ake so.
4. Kimantawar Gwaji
Marubutan sun kimanta SIT akan manyan tsarin fassarar kasuwanci guda biyu: Google Translate da Bing Microsoft Translator.
Sakamakon Gwaji a Tsallake
- Shigarwar Gwaji: Jimlolin tushe 200
- Kurakuran Google Translate da aka Gano: Matsaloli 64
- Kurakuran Bing Translator da aka Gano: Matsaloli 70
- Madaidaicin Rahoton Kurakurai Top-1: ~70% (an tabbatar da hannu)
4.1. Saitawa & Gano Kurakurai
Ta amfani da jimlolin tushe 200 daban-daban, SIT ta samar da bambance-bambancen jimloli masu kama kuma ta gabatar da su ga APIs na fassara. Sakamakon fassarorin an yi nazarinsu, kuma an kwatanta tsarinsu.
4.2. Sakamako & Rarraba Kurakurai
SIT ta yi nasarar gano kurakuran fassara da yawa, waɗanda aka rarraba su zuwa rarrabuwa ciki har da:
- Rashin Fassara: Barin abubuwan da ke cikin tushe.
- Wuce Gona da Irin Fassara: Ƙara abubuwan da ba su da tushe.
- Gyaran da ba daidai ba: Haɗa masu gyara ba daidai ba (misali, sifa, maganganu).
- Kuskuren Kalma/Jumla: Zaɓin ƙamus ba daidai ba duk da mahallin daidai.
- Ma'ana marar fayyace: Fassarorin da suka karkatar da tsarin ma'ana na jimla ta asali.
Bayanin Chati (Tunani): Chati na sanduna zai nuna rarraba jimillar kurakurai 134 da aka gano a cikin tsarin biyu, an raba su ta wannan rarrabuwar kurakurai, yana nuna "Gyaran da ba daidai ba" da "Kuskuren Kalma/Jumla" a matsayin mafi yawan rukuni.
5. Muhimman Bayanai & Bincike
6. Cikakkun Bayanai na Fasaha & Tsarin Aiki
Tsarin Lissafi: Bari $S$ ya zama jimlar tushe ta asali. Ƙirƙiri saitin jimlolin bambance-bambance $V = \{S_1, S_2, ..., S_n\}$ inda kowane $S_i$ aka ƙirƙira shi ta hanyar musanya kalma ɗaya a cikin $S$ tare da kalma mai ma'ana mai kama. Ga kowane jimla $X \in \{S\} \cup V$, sami fassararta $T(X)$ ta hanyar tsarin MT da ake gwadawa. Yi nazarin kowane fassara zuwa wakilcin bishiya $\mathcal{T}(T(X))$. Duban tsayayyen tsari don nau'i-nau'i $(S_i, S_j)$ shine: $d(\mathcal{T}(T(S_i)), \mathcal{T}(T(S_j))) \leq \delta$, inda $d$ ma'aunin nisan bishiya ne (misali, Nisan Gyaran Bishiya da aka daidaita ta girman bishiya) kuma $\delta$ bakin kofa ne na haƙuri. Keta yana nuna yuwuwar kuskure.
Misalin Tsarin Bincike (Ba Lamba ba):
Yanayi: Gwada fassarar jimlar Turanci "The quick brown fox jumps over the lazy dog" zuwa Faransanci.
Mataki na 1 (Kaɗa): Samar da bambance-bambance: "The fast brown fox jumps...", "The quick brown fox leaps over..."
Mataki na 2 (Fassara): Sami fassarorin Faransanci ga duk jimlolin ta hanyar API.
Mataki na 3 (Nazari): Samar da bishiyoyin nazarin dogaro ga kowane fassarar Faransanci.
Mataki na 4 (Kwatanta): Lissafa kamancen bishiya. Idan bishiyar don bambance-bambancen "fast" ta bambanta sosai da bishiyar don bambance-bambancen "quick" (misali, canza alaƙar abu-abu ko haɗa maganganun aiki), SIT tana nuna alamar matsala. Duban hannu na iya bayyana cewa an yi kuskuren fassara "fast" ta hanyar da ta canza tsarin nahawu na jimla.
7. Aikace-aikace na Gaba & Jagorori
Tsarin SIT ya wuce MT na gaba ɗaya. Aikace-aikace nan da nan sun haɗa da:
- MT mai Keɓantaccen Yanki: Tabbatar da tsarin fassarar shari'a, likita, ko fasaha inda daidaiton tsari ya fi mahimmanci.
- Sauran Ayyukan NLG: Daidaita ƙa'idar tsayayyen tsari don gwada taƙaitaccen rubutu, sake fasalin rubutu, ko tsarin samar da bayanai zuwa rubutu.
- Gyaran Ƙira & Gyara Kurakurai: Amfani da shari'o'in gazawar da SIT ta gano a matsayin bayanai da aka yi niyya don horon adawa ko gyara ƙira.
- Haɗawa tare da Ma'auni na Ma'ana: Haɗa binciken tsari tare da ma'auni na kamancen ma'ana (misali, BERTScore, BLEURT) don ƙarin cikakkiyar saitin tabbatarwa.
- Sa ido na Ainihi: Tura gwaje-gwajen SIT masu sauƙi don sa ido kan aikin rayuwa na ayyukan MT kuma su jawo faɗakarwa don raguwar inganci.
Bincike na gaba yakamata ya bincika daidaita bakin kofa, haɗawa tare da masu kimantawa na ƙirar babban harshe (LLM), da tsawaita tsayayyen tsari zuwa tsarin matakin magana don gwada fassarar sakin layi ko takarda.
8. Nassoshi
- He, P., Meister, C., & Su, Z. (2020). Structure-Invariant Testing for Machine Translation. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE).
- Vaswani, A., et al. (2017). Attention is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
- Papineni, K., et al. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL).
- Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and Harnessing Adversarial Examples. arXiv preprint arXiv:1412.6572.
- Ribeiro, M. T., et al. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
- Zhu, J.-Y., et al. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV). (An ambata don kwatankwacin ra'ayi na daidaiton zagaye/tsayayyen tsari).
- Google AI Blog. (2016). A Neural Network for Machine Translation, at Production Scale. https://ai.googleblog.com/
- Microsoft Research. (2018). Achieving Human Parity on Automatic Chinese to English News Translation. https://www.microsoft.com/en-us/research/
Sharhin Manazarta: Rarraba Maki Hudu
Mahimmin Fahimta: Hikimar takardar tana cikin sake fasalin matsalar "marar warwarewa" ta gwajin MT. Maimakon bin fatalwar cikakkiyar fassarar tunani—matsalar da har ma masu kimantawa na ɗan adam ke fama da ita saboda ra'ayi—SIT tana amfani da daidaiton dangi a matsayin wakilin daidaito. Wannan yayi daidai da ainihin ra'ayi a cikin koyon da ba a kulawa da shi ba ko a cikin dabarun daidaitawa na daidaitawa da ake amfani da su a cikin koyon da ba a kulawa da shi ba don hangen nesa, inda aka tilasta tsinkayen ƙirar don bambance-bambancen shigarwa ɗaya su yarda. Fahimtar cewa tsarin tsari yakamata ya fi tsayayya ga musanyar ma'anar kalma fiye da ma'anar ma'ana yana da sauƙi kuma yana da ƙarfi.
Tsarin Ma'ana: Hanyar tana da kyau a layi kuma ana iya sarrafa ta ta atomatik: kaɗa, fassara, bincika, kwatanta. Yana da wayo yana amfani da kayan aikin NLP da aka kafa (masu nazari, WordNet) a matsayin tubalan ginin sabon tsarin tabbatarwa. Kwararar tana kwatanta ƙa'idodin gwaji na metamorphic da aka kafa a cikin aikin injiniyan software na baya amma yana amfani da su ga keɓantaccen sararin fitarwa na samar da harshe na halitta.
Ƙarfi & Kurakurai: Babban ƙarfi shine aikace-aikace na zahiri. SIT baya buƙatar samun dama ga ciki na ƙirar (baƙar fata), babu tarin rubutu mai kama, kuma babu tunanin da mutum ya rubuta, yana sa a iya amfani da shi nan da nan don gwada APIs na kasuwanci. Daidaitonsa na 70% yana da ban sha'awa ga hanyar atomatik. Duk da haka, hanyar tana da gurbatattun makaho. A zahiri yana da iyaka ga gano kurakuran da ke bayyana a matsayin rarrabuwar tsari. Fassarar na iya zama kuskuren ma'ana sosai duk da haka yana kama da tsari daidai (misali, fassara "banki" a matsayin cibiyar kuɗi da bankin kogi a cikin tsarin jimloli iri ɗaya). Bugu da ƙari, ya dogara sosai akan daidaiton mai nazari na asali, yana iya rasa kurakurai ko samar da ƙididdiga mara kyau idan mai nazari ya gaza. Idan aka kwatanta da hanyoyin harin adawa waɗanda ke neman mafi ƙarancin ɓarna don karya ƙirar, ɓarnawar SIT ta halitta ce kuma ba ta canza ma'ana, wanda ƙarfi ne don gwada ƙarfi a cikin yanayin duniya na gaske amma bazai bincika mafi munin halayen ƙirar ba.
Bayanai masu Aiki: Ga masu aiki a masana'antu, wannan takarda tsari ne. Aiki Nan da Nan: Haɗa SIT cikin tsarin CI/CD don kowane samfurin da ya dogara da MT na ɓangare na uku. Gwaji ne mai ƙarancin farashi, mai dawo da riba mai yawa. Ci gaba na Dabarun: Tsawaita ra'ayin "tsayayyen tsari" fiye da tsari. Aikin nan gaba yakamata ya bincika tsayayyen ma'ana ta amfani da abubuwan haɗe-haɗe na jimla (misali, daga ƙirar kamar BERT ko Sentence-BERT) don kama kurakuran da SIT ta rasa. Haɗa binciken tsayayyen tsari da na ma'ana zai iya ƙirƙirar saitin gwaji mai ƙarfi. Bugu da ƙari, rarrabuwar kurakuran da aka bayar yana da matuƙar mahimmanci don ba da fifiko ga ƙoƙarin inganta ƙirar—mayar da hankali kan gyara kurakuran "gyaran da ba daidai ba" da farko, kamar yadda suka fi yawa. Wannan aikin yakamata a ambata shi tare da takardun gwaji na asali don tsarin AI, kafa sabon ƙaramin fanni na gwaji don ƙirar samar da harshe.