This is a report on an empirical study on the usability for translation trainees of neural machine translation systems when post-editing (mtpe). Sixty Chinese translation trainees completed a questionnaire on their perceptions of mtpe's usability. Fifty of them later performed both a post-editing task and a regular translation task, designed to examine mtpe's usability by comparing their performance in terms of text processing speed, effort, and translation quality. Contrasting data collected by the questionnaire, keylogging, eyetracking and retrospective reports we found that, compared with regular, unaided translation, mtpe's usefulness in performance was remarkable: (1) it increased translation trainees' text processing speed and also improved their translation quality; (2) mtpe's ease of use in performance was partly proved in that it significantly reduced informants' effort as measured by (a) fixation duration and fixation counts; (b) total task time; and (c) the number of insertion keystrokes and total keystrokes. However, (3) translation trainees generally perceived mtpe to be useful to increase productivity, but they were skeptical about its use to improve quality. They were neutral towards the ease of use of mtpe.
Bahdanau, D., Cho, K. & Bengio, Y. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. https://arxiv.org/abs/1409.0473.
Bentler, P. M. & Chou, C. P. 1987. Practical issues in structural modeling. Sociological Methods & Research Vol. 16. No. 1. 78–117.
Briggs, N. 2018. Neural Machine Translation Tools in the Language Learning Classroom: Students’ Use, Perceptions, and Analyses. The JALT CALL Journal Vol. 14. No. 1. 3–24.
Cadwell, P., Castilho, S. & O'Brien, S. 2016. Human Factors in Machine Translation and Post-editing among Institutional Translators. Translation Spaces Vol. 5. No. 2. 222–243.
Cadwell, P., O'Brien, S. & Teixeira, C. 2018. Resistance and Accommodation: Factors for the (non-)Adoption of Machine Translation among Professional Translators. Perspectives . Vol. 26. No. 3. 301–321.
Carl, M. 2012. Translog-II: A Program for Recording User Activity Data for Empirical Reading and Writing Research. Paper presented at the Eight International Conference on Language Resources and Evaluation (Istanbul, Turkey, May 23–25).
Carl, M. Dragsted, B., Elming, J., Hardt, D. & Jakobsen, A. L. 2011. The Process of Post-editing: A Pilot Study. In: Sharp, B., Zock, M., Carl, M. & Lykke Jakobsen, A. (eds) Human-Machine Interaction in Translation: Proceedings of the 8th International NLPCS Workshop. Frederiksberg: Samfundslitteratur. 131–142.
Carl, M., Dragsted, B., & Jakobsen, A. L. 2011. A Taxonomy of Human Translation Styles, Translation Journal , Vol. 16, No. 2.
Carl, M., Gutermuth, S. & Hansen-Schirra, S. 2015. Post-editing Machine Translation: Efficiency, Strategies and Revision Processes in Professional Translation Settings. In: Ferreira, A. & Schwieter, J. W. (eds). Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. Amsterdam: John Benjamins. 145–175.
Carl, M., Schaeffer, M. & Bangalore, S. 2016. The CRITT Translation Process Research Database. In: Carl, M., Bangalore, S. & Schaeffer, M. (eds) 2016. New Directions in Empirical Translation Process Research: Exploring the CRITT TPR-DB. New York: Springer. 13–54.
Castilho, S. & O'Brien, S. 2017. Acceptability of Machine-translated Content: A Multi-language Evaluation by Translators and End-users. Linguistica Antverpiensia, New Series: Themes in Translation Studies Vol. 16, 120–136.
Da Silva, I. Alves, F., Schmaltz, M., Pagano, A., Wong, D., Chao, L., Leal, A. L. V. Quaresma, P., Garcia, C. and da Silva, G. E. 2017. Translation, Post-editing and Directionality: A Study of Effort in the Chinese-Portuguese Language Pair. In: Jakobsen, A. L. & Mesa Lao, B. (eds) Translation in Transition: Between Cognition, Computing and Technology. Amsterdam: John Benjamins. 107–134.
Da Silva, I. L., Schmaltz, M., Alves, F., Pagano, A., Wong, D., Chao, L., Leal, A. L. V. Quaresma, P. & Garcia, C. 2015. Translating and Post-editing in the Chinese-Portuguese Language pair: Insights from an Exploratory Study of Key Logging and Eye Tracking. Translation Spaces Vol. 4. No. 1. 145–169.
Daems, J., Vandepitte, S., Hartsuiker, R. J. & Macken, L. 2017. Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators. Meta Vol. 62. No. 2. 245–270.
Davis, F. D. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly Vol. 13, 319–340.
De Sousa, S. C., Aziz, W. & Specia, L. 2011. Assessing the Post-editing Effort for Automatic and Semi-automatic Translations of DVD Subtitles. In Angelova. G., Bontcheva, K., Mitkov, R. & Nicolov, N. (eds) Proceedings of the Recent Advances in Natural Language Processing Conference. Hissar, Bulgaria. 97–103.
Depraetere, I., de Sutter, N. & Tezcan, A. 2014. Post-edited Quality, Post-editing Behavior and Human Evaluation: A Case Study. In: O'Brien, S., Balling, L. W., Carl, M., Simard, M. & Specia, L. (eds). Post-editing of Machine Translation: Processes and Applications. Newcastle: Cambridge Scholars Publishing. 78–108.
Fiederer, R. & O’Brien, S. 2009. Quality and Machine Translation: A Realistic Objective? The Journal of Specialised Translation Vol. 11, 52–74.
Garcia, I. 2010. Is Machine Translation Ready Yet? Target Vol. 22. No. 1. 7–21.
Garcia, I. 2011. Translating by Post-editing: Is it the Way Forward? Machine Translation Vol. 25. 217–237.
Germann, U. 2008. Yawat: Yet Another Word Alignment Tool. In Proceedings of the ACL-08: HLT Demo Session (Companion Volume). 20–23.
Green, S., Heer, J. & Manning, C. 2013. The Efficacy of Human Post-editing for Language Translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 439–448.
Guerberof, A. 2009. Productivity and Quality in MT Post-editing. In MT Summit XII-Workshop: Beyond Translation Memories: New Tools for Translators MT. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.575.5398&rep=rep1&type=pdf.
Guerberof, A. 2013. What do Professional Translators Think about Post-editing? The Journal of Specialised Translation Vol. 19. 75–95.
Hartson, H. R. 1998. Human–Computer Interaction: Interdisciplinary Roots and Trends. Journal of Systems and Software Vol. 43. No. 2. 103–118.
Hvelplund, K. T. H. 2011. Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study .Copenhagen: Copenhagen Business School.
Hvelplund, K. T. 2014. Eye Tracking and the Translation Process: Reflections on the Analysis and Interpretation of Eye-tracking Data. In: Muñoz, R. (ed.) Minding Translation. MonTI Special Issue 1. 201–223.
ISO 9241-11 (1998. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11: Guidance on Usability. Geneva: International Organization for Standardization.
Jia, Y., Carl, M. & Wang, X. 2019a. Post-editing Neural Machine Translation versus Phrase-Based Machine Translation for English–Chinese. Machine Translation Vol. 33. No. 1–2. 9–29.
Jia, Y., Carl, M. & Wang, X. 2019b. How Does the Post-editing of Neural Machine Translation Compare with From-scratch Translation? A Product and Process study. The Journal of Specialised Translation Vol. 31. 60–86.
Junczys-Dowmunt, M., Dwojak, T. & Hoang, H. 2016. Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions. https://arxiv.org/abs/1610.01108.
Just, A. & Carpenter, P. 1980. A Theory of Reading: From Eye Fixation to Comprehension. Psychological Review Vol. 4, 329–354.
Koglin, A. 2015. An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors. Translation & Interpreting Vol. 7. 126–141.
Koponen, M. 2016. Is Machine Translation Post-editing Worth the Effort? A Survey of Research into Post-editing and Effort. The Journal of Specialised Translation Vol. 25, 131–148.
Krings, H. P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Postediting Processes. Kent, OH: Kent State University Press.
Läubli, S. Fishel, M., Massey, G., Ehrensberger-Dow, M.& Volk, M. 2013. Assessing Post-editing Efficiency in a Realistic Translation Environment. In: O'Brien, S., Simard, M. & Specia, L. (eds). Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, Nice. 83–91.
Lee, J. & Liao, P. 2011. A Comparative Study of Human Translation and Machine Translation with Post-editing. Compilation and Translation Review Vol. 4. No. 2. 105–149.
Lu, Z. & Sun, J. 2018. An Eye-tracking Study of Cognitive Processing in Human Translation and Post-editing. Foreign Language Teaching and Research Vol. 50. No. 5. 760–769.
Lund, A. M. 2001. Measuring Usability with the USE Questionnaire. Usability Interface Vol. 8. No. 2. 3–6.
Mellinger, Ch., & Shreve, G. M. 2006. Match Evaluation and Over-editing in a Translation Memory Environment. In: Muñoz, R. (ed.) Reembedding Translation Process Research. Amsterdam: John Benjamins. 131–148.
Miller, R. B. 1971. Human Ease of Use Criteria and their Tradeoffs. IBM Report TR 00.2185. Poughkeepsie, NY: IBM Corporation.
Nielsen, J. 1993. Usability Engineering. London: Academic Press.
Nunnally, J. & Bernstein, I. 1994. Psychometric Theory 3E. New York: McGraw-Hill.
O'Brien, S. 2007. Eye-tracking and Translation Memory Matches. Perspectives Vol. 14, 185–205.
Pavlović, N. & Jensen, K. T. H. 2009. Eye Tracking Translation Directionality. In: Pym, A. & Perekrestenko, A. (eds) Translation Research Projects 2. Tarragona: Universitat Rovira i Virgili. 101–119.
Plitt, M. & Masselot, F. 2010. A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localization Context. The Prague Bulletin of Mathematical Linguistics Vol. 93. 7–16.
Popović, M. 2018. Language-related Issues for NMT and PBMT for English–German and English–Serbian. Machine Translation Vol. 32. No. 3. 237–253.
Revythi, A., & Tselios, N. 2019. Extension of Technology Acceptance Model by using System Usability Scale to Assess Behavioral Intention to Use E-learning. Education and Information Technologies Vol. 24. No. 4. 2341–2355.
Rossi, C. 2019. Uses and Perception of Machine Translation at the European Commission. The Journal of Specialised Translation Vol. 31, 177–200.
Screen, B. 2017. Machine Translation and Welsh: Analysing free Statistical Machine Translation for the Professional Translation of an Under-researched Language Pair. The Journal of Specialised Translation Vol. 28, 317–344.
Screen, B. 2019. What Effect does Post-editing Have on the Translation Product from an End-user’s Perspective? The Journal of Specialised Translation 31. 133–157.
Shackel, B. & Richardson, S. J. (eds) 1991. Human Factors for Informatics Usability. Cambridge: Cambridge University Press.
Suojanen, T., Koskinen, K. & Tuominen, T. 2015. User-centered Translation. Abingdon: Routledge.
TAUS. 2013. Adequacy/Fluency Guidelines. https://taus.net/academy/best-practices/evaluate-best-practices/adequacy-fluency-guidelines.
TAUS. 2016. TAUS Post-editing Guidelines. https://www.taus.net/think-tank/articles/postedit-articles/taus-post-editing-guidelines.
TAUS. 2019. TAUS Keynotes Asia 2019. https://www.taus.net/academy/reports/event-reports/taus-keynotes-asia-2019.
Thorpe, A., Nesbitt, K., & Eidels, A. 2019. Assessing Game Interface Workload and usability: A Cognitive Science Perspective. In: Proceedings of the Australasian Computer Science Week multiconference, ACSW 2019 .ACM. 44:1–44:8 .
Travis, D. 2017. E-commerce Usability: Tools and Techniques to Perfect the On-line Experience. London: CRC Press.
Vieira, L. N. 2016. Cognitive Effort in Post-Editing of Machine Translation: Evidence From Eye Movements, Subjective Ratings, and Think-Aloud Protocols. PhD Thesis. Newcastle University.
Yamada, M. 2015. Can College Students Be Post-editors? An Investigation into Employing Language Learners in Machine Translation plus Post-editing Settings. Machine Translation Vol. 29. No. 1. 49–67.
Yamada, M. 2019. The Impact of Google Neural Machine Translation on Post-editing by Student Translators. The Journal of Specialised Translation Vol. 31. 87–106.
Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. 2015. A Usability Evaluation of a Blended MOOC Environment: An Experimental Case Study. The International Review of Research in Open and Distributed Learning Vol. 16. No. 2. 69–93.