Developing the Cultural Meaning Preservation Assessment (CMPA): A Multidimensional Methodology for Evaluating AI-Assisted Translation of Indigenous Cultural Knowledge Units
DOI:
https://doi.org/10.30872/jbssb.v10i3.28931Abstract
ABSTRAK
Kemajuan Akal Imitasi (AI) telah meningkatkan kualitas terjemahan melalui berbagai sistem terjemahan berbantuan AI yang menghasilkan terjemahan lebih cepat dan lebih akurat secara linguistik dibandingkan sebelumnya (Koehn, 2020; Toral & Way, 2018). Namun, AI masih menghadapi tantangan dalam mempertahankan makna budaya, khususnya pada Indigenous Cultural Knowledge Units (ICKUs) yang merepresentasikan identitas historis, sosial, dan budaya masyarakat adat. Penelitian ini bertujuan mengembangkan Cultural Meaning Preservation Assessment (CMPA), yaitu metode multidimensi untuk mengevaluasi kemampuan sistem AI dalam melestarikan makna budaya pada terjemahan ICKUs dari Kutai Adat Lawas. Penelitian menggunakan desain deskriptif kualitatif dengan korpus sebanyak 25 ICKUs yang diperoleh dari Badan Registrasi Wilayah Adat (BRWA) dan Indonesia Kaya. Sebagai acuan evaluasi, dikembangkan Human Gold Standard Translation (HGST) menggunakan teknik peminjaman dan deskripsi (Newmark, 1988). Selanjutnya, terjemahan yang dihasilkan oleh Google Translate, DeepL, ChatGPT, Gemini, dan Claude dievaluasi menggunakan enam dimensi CMPA, yaitu pelestarian leksikal, semantik, budaya, historis, pragmatik, dan identitas pribumi. Hasil penelitian menunjukkan bahwa meskipun sebagian besar sistem Akal Imitasi (AI) memiliki kinerja yang baik dalam kesetaraan leksikal dan semantik, pelestarian makna historis, pragmatik, dan budaya masih menjadi kelemahan utama. CMPA memberikan kontribusi metodologis sebagai kerangka kerja holistik untuk mengevaluasi pelestarian makna budaya dalam terjemahan berbantuan Akal Imitasi (AI).
Keywords: Artificial Intelligence-Assisted Translation, Cultural Meaning Preservation Assessment (CMPA), Indigenous Cultural Knowledge Units (ICKUs), Translation Quality Assessment, Cultural Meaning Preservation, Kutai Adat Lawas.
ABSTRACT
Recent advances in Artificial Intelligence (AI) have significantly improved translation quality through the development of AI-assisted translation systems capable of producing faster and more linguistically accurate translations than previous approaches (Koehn, 2020; Toral & Way, 2018). Nevertheless, AI continues to face challenges in preserving cultural meaning, particularly when translating Indigenous Cultural Knowledge Units (ICKUs), which embody the historical, social, and cultural identities of Indigenous communities. This study aims to develop the Cultural Meaning Preservation Assessment (CMPA), a multidimensional assessment framework designed to evaluate the ability of AI-assisted translation systems to preserve the cultural meaning of ICKUs derived from Kutai Adat Lawas. The study employed a qualitative descriptive research design using a corpus of 25 ICKUs collected from the Badan Registrasi Wilayah Adat (BRWA) and Indonesia Kaya. A Human Gold Standard Translation (HGST) was developed as the evaluation benchmark using borrowing and descriptive translation techniques (Newmark, 1988). Translations generated by Google Translate, DeepL, ChatGPT, Gemini, and Claude were subsequently evaluated using the six dimensions of the CMPA framework: lexical, semantic, cultural, historical, pragmatic, and Indigenous identity preservation. The findings indicate that although most AI systems perform well in achieving lexical equivalence and semantic accuracy, preserving historical, pragmatic, and culturally embedded meanings remains a major challenge. The proposed CMPA makes a methodological contribution by providing a holistic framework for evaluating cultural meaning preservation in AI-assisted translation.
Keywords: Artificial Intelligence-Assisted Translation; Cultural Meaning Preservation Assessment (CMPA); Indigenous Cultural Knowledge Units (ICKUs); Translation Quality Assessment; Cultural Meaning Preservation; Kutai Adat Lawas.
Downloads
References
REFERENCES
Badan Registrasi Wilayah Adat: https://brwa.or.id/wa/view/WmZnRXljazJtbmc?
Baker, M. (2018). In other words: A coursebook on translation (3rd ed.). Routledge.
Bassnett, S. (2021). Translation studies (5th ed.). Routledge.
Battiste, M. (2013). Decolonizing education: Nourishing the learning spirit. Purich Publishing.
Castilho, S. (2023). Machine translation and human translation in the age of artificial intelligence. Routledge.
Creswell, J. W., & Creswell, J. D. (2023). Research design: Qualitative, quantitative, and mixed methods approaches (6th ed.). SAGE Publications.
Guerreiro, N. M., Freitag, M., Foster, G., Cherry, C., & Martins, A. F. T. (2023). Quality evaluation of large language models in machine translation. arXiv. https://arxiv.org/abs/2308.01825
Hall, S. (1997). Representation: Cultural representations and signifying practices. SAGE Publications.
House, J. (2015). Translation quality assessment: Past and present. Routledge.
Kenny, D. (2022). Machine translation for everyone: Empowering users in the age of artificial intelligence. Language Science Press.
Koehn, P. (2020). Neural machine translation. Cambridge University Press.
Liu, Y., Zhang, X., Wang, J., & Li, H. (2024). Artificial intelligence-assisted translation and cultural meaning preservation: Challenges and opportunities. Machine Translation. Advance online publication.
Lommel, A. R., Burchardt, A., & Uszkoreit, H. (2014). Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12, 455–463.
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation (4th ed.). Jossey-Bass.
Mertens, D. M. (2020). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (5th ed.). SAGE Publications.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). SAGE Publications.
Molina, L., & Hurtado Albir, A. (2002). Translation techniques revisited: A dynamic and functionalist approach. Meta, 47(4), 498–512. https://doi.org/10.7202/008033ar
Munday, J. (2016). Introducing Translation Studies (4th ed.). Routledge.
Nakata, M. (2007). Disciplining the savages, savaging the disciplines. Aboriginal Studies Press.
Newmark, P. (1988). A textbook of translation. Prentice Hall.
Nord, C. (2018). Translating as a purposeful activity: Functionalist approaches explained (2nd ed.). Routledge.
OpenAI. (2023). GPT-4 technical report. https://arxiv.org/abs/2303.08774
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135
Patton, M. Q. (2015). Qualitative research & evaluation methods (4th ed.). SAGE Publications.
Popel, M., Tomkova, M., Tomek, J., Kaiser, Ł., Uszkoreit, J., Bojar, O., & Žabokrtský, Z. (2020). Transforming machine translation: A deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11, Article 4381. https://doi.org/10.1038/s41467-020-18073-9
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2685–2702). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.213
TAUS. (2023). Dynamic Quality Framework (DQF). https://www.taus.net
Toral, A., & Way, A. (2018). What level of quality can neural machine translation attain on literary text? In Translation Quality Assessment (pp. 263–287). Springer.
Tracy, S. J. (2020). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact (2nd ed.). Wiley-Blackwell.
UNESCO. (2021). Local and indigenous knowledge systems (LINKS). https://en.unesco.org/links
United Nations. (2007). United Nations Declaration on the Rights of Indigenous Peoples. United Nations. https://www.un.org/development/desa/indigenouspeoples/declaration-on-the-rights-of-indigenous-peoples.html
Venuti, L. (2018). The translator's invisibility: A history of translation (3rd ed.). Routledge.
Downloads
Published
Issue
Section
License
Every work in Ilmu Budaya: Jurnal Bahasa, Sastra, Seni, dan Budaya is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Under the following terms:
Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors can enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
