[go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Hybrid paradigm for Spanish Sign Language synthesis

  • Long paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

This work presents a hybrid approach to sign language synthesis. This approach allows the hand-tuning of the phonetic description of the signs, which focuses on the time aspect of the sign. Therefore, the approach retains the capacity for the performing of morpho-phonological operations, like notation-based approaches, and improves the synthetic signing performance, such as the hand-tuned animations approach. The proposed approach simplifies the input message description using a new high-level notation and storage of sign phonetic descriptions in a relational database. Such relational database allows for more flexible sign phonetic descriptions; it also allows for a description of sign timing and the synchronization between sign phonemes. The new notation, named HLSML, is a gloss-based notation focusing on message description in it. HLSML introduces several tags that allow for the modification of the signs in the message that defines dialect and mood variations, both of which are defined in the relational database, and message timing, including transition durations and pauses. A new avatar design is also proposed that simplifies the development of the synthesizer and avoids any interference with the independence of the sign language phonemes during animation. The obtained results showed an increase of the sign recognition rate compared to other approaches. This improvement was based on the active role that the sign language experts had in the description of signs, which was the result of the flexibility of the sign storage approach. The approach will simplify the description of synthesizable signed messages, thus facilitating the creation of multimedia-signed contents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. http://www.signwriting.com or http://www.cervantesvirtual.com.

  2. This notation is a general-purpose animation notation that allows for the definition of the animation of an H-Anim compliant avatar. This notation focuses on joint rotation definition; therefore, it presents a lower level of abstraction compared to SiGML.

  3. For example, the hand position is different if the signer uses an all-fingers-extend hand shape, the palm is facing the signer and the fingers pointing up. The Location PP is set in front of the right eye and the sign defines the Contact Point PP rather than setting the end of the little finger in front of the right eye or setting the end of the thumb in the same position.

  4. These realizations must not make the phoneme unrecognizable, which leads to the misunderstanding of a sign.

  5. This includes the duration of the transition between signs (which defined the duration of the Movement block between two consecutive signs) and message pauses (which modified the initial or final Hold blocks of the sign).

  6. The “language” attribute can be applied to a single sign, to a group of signs or to the sentence.

  7. The “tag” attribute defines this emotional variation. Like the “language” attribute, it can be applied to a single sign, a group of signs or a complete sentence.

  8. The position of the hand is defined using the Location, Plane and Movement PPs.

  9. This file format has been established by the JSR-184 standard.

  10. We have only considered client–server communication.

References

  1. Bangham, A., Cox, S., Elliot, R., Glauer, J., Marshal, I.: Virtual signing: capture, animation, storage and transmission—an overview of the ViSiCAST project. In: Proceedings of IEE Seminary on Speech and Language Processing for Disabled and Elderly People, pp. 1–7 (2000)

  2. Battison, R.: Phonology in American Sign Language: 3-D and digit-vision. In: Proceedings of the California Linguistic Association conference, Stanford (1973)

  3. Brentari, D.: Trilled movement: phonetic realization and formal representation. Lingua 98, 43–71 (1996)

    Article  Google Scholar 

  4. Corina, D.: Sign linguistics phonetics, phonology and morpho-syntax. Lingua 98, 73–102 (1996)

    Article  Google Scholar 

  5. Cox, S., Lincoln, M., Nakisa, M., Wells, M., Tutt, M., Abbott, S.: The development and evaluation of a speech-to-sign translation system to assist transactions. Int. J. Hum-comput. Interact. 16, 141–161 (2003). doi:10.1207/S15327590IJHC1602_02

    Article  Google Scholar 

  6. Elliot, R., Glauert, J., Jennings, V., Kennaway, R.: An overview of the SiGML notation and SiGML signing software system. In: Proceedings of LREC, Lisbon, pp. 98–104 (2004)

  7. Elliott, R., Glauert, J., Kennaway, R., Marshal, I., Sáfár, É.: Linguistic modelling and language-processing technologies for avatar-based sign language presentation. Univ. Acess. Inf. Soc. 6, 375–391 (2008)

    Article  Google Scholar 

  8. Emmorey, K., Herzig, M.: Categorical versus gradient properties of classifier constructions in ASL. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 221–246. Psychology Press (2003)

  9. European Media Masters of Art: Vsign. http://www.vsign.nl/ (2002). Accessed 21 Feb 2011

  10. Fotinea, S., Efthimiou, E., Caridakis, G., Karpouzis, K.: A knowledge-based sign synthesis architecture. Univ. Acess. Inf. Soc. 6, 405–418 (2008)

    Article  Google Scholar 

  11. Fundación CNSE: Diccionario Normativo de la Lengua de Signos Española [Spanish Sign Language Normative Dictionary]. Fundación CNSE, Madrid (2008)

    Google Scholar 

  12. Hanke, T.: Hamnosys—representing sign language data in language resources and language processing contexts. In: Proceedings of LREC, Lisbon, pp. 1–6 (2004)

  13. Herrero, B.Á.: A practical writing of sign languages. In: Proceedings of LREC, Lisbon, pp. 37–42 (2004)

  14. Herrero, B.Á.: Gramática Didáctica de la Lengua de Signos Española (LSE) [Spanish Sign Language Didactive Grammar], 1st edn. Ediciones SM, Madrid (2009)

    Google Scholar 

  15. Huang, Z., Elïens, A., Visser, C.: Step: a scripting language for embodied agents. In: Proceedings of Workshop on Lifelike Animated Agents, Tokyo, pp. 46–51

  16. Huenerfauth, M.: Spatial representation of classifier predicates for machine translation into American Sign Language. In: Workshop on the Representation and Processing of Signed Languages. In: Proceedings of LREC, Lisbon, pp 24–31 (2004)

  17. Huenerfauth, M.: Representing coordination and non-coordination in an American Sign Language. In: Proceedings of SIGACCESS, ACM, New York, pp. 44–51 (2005). doi:10.1145/1090785.1090796

  18. Huenerfauth, M.: A linguistically motivated model for speed and pausing in animations of American Sign Language. ACM Trans. Access. Comput. 2, 1–31 (2009). doi:10.1145/1530064.1530067

    Article  Google Scholar 

  19. Hulst, H., Mills, A.: Issues in sign linguistic: phonetics, phonology and morphosyntax. Lingua 98(1–3), 3–17 (1996)

    Article  Google Scholar 

  20. Hybrid Graphics: Hybrid rasteroid (2006). http://www.hybrid.fi. Accessed 20 Jan 2008

  21. Igi, S., Ujitani, M., Tamaru, M., Yamamoto, Y., Sugita, S.: Sign-language synthesis for mobile environments. In: Proceedings of Intern Conf in Central Europe on Computer Graphics, Visualization and Computer Vision (2003)

  22. Informatics and Telematics Institute: Vsigns. http://vsigns.iti.gr:8080/VSigns (2004). Accessed 21 Feb 2011

  23. ISO/IEC 14772–1:1997: Information technology—Computer graphics and ima—Part 1: Functional specification and UTF-8 encoding. International Organization for Standardization, Geneva (1997)

    Google Scholar 

  24. ISO/IEC 14772–2:2004: Information technology—Computer graphics and image processing—The Virtual Reality Modeling Language (VRML)–Part 2: External authoring interface (EAI). International Organization for Standardization, Geneva (2004)

    Google Scholar 

  25. ISO/IEC 19774:2005: Information technology—Computer graphics and image processing—Humanoid animation (H-Anim). International Organization for Standardization, Geneva (2005)

    Google Scholar 

  26. Karpouzis, K., Caridakis, G., Fotinea, S.E., Efthimiou, E.: Educational resources and implementation of a Greek sign language synthesis architecture. Comput. Educ. 49, 54–74 (2007). doi:10.1016/j.compedu.2005.06.004

    Article  Google Scholar 

  27. Kennaway, R., Glauert, J., Zwitserlood, I.: Providing signed content on the internet by synthesized animation. ACM Trans. Comput. Human Interact. 14, 1–29 (2007). doi:10.1145/1279700.1279705

    Article  Google Scholar 

  28. Liddell, S.: Sources of meaning in ASL classifier predicates. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 199–220. Psychology Press (2003)

  29. Liddell, S., Johnson, R.: American Sign Language: the phonological base. Sign Lang. Stud. 64, 195–278 (1989)

    Google Scholar 

  30. López-Colino, F., Colás, J.: Providing ubiquitous access to synthetic sign language contents over multiple platforms. In: Cruz-Cunha, M., Moreira, F. (eds.) Handbook of Research on Mobility and Computing: Evolving Technologies and Ubiquitous Impacts, IGI International (In Press)

  31. López-Colino, F., Garrido, J., Colás, J.: Description and synthesis of Spanish sign language classifiers. In: Proceedings of INTERACCION, AIPO, Barcelona (2009)

  32. Moccozet, L., Magnenat-Thalmann, N.: Dirichelet free form deformations and their application to hand simulation. In: Proceedings of IEEE Computer Animation, pp. 93–102 (1997)

  33. Muñoz Baell, I.:¿Cómo se articula la lengua de signos española? [How is the articulation of LSE?] CNSE, Madrid (1999)

  34. Prillwitz, S., Leven, R., Zienert, H., Hanke, T., Herming J.: HamNoSys. Version 2.0; Hamburg Notation System for Sign Languages. An introductory guide. Verlag (1989)

  35. Rocha, A., Pereira, G.: Supporting deaf sign languages in written form on the web. In: Proceedings of LREC, Lisbon, pp 26–28 (2004)

  36. San Segundo, R., Barra, R., D’Haro, L.F., Montero, J.M., Cordoba, R., Ferreiros, J.: A Spanish speech to sign language translation system for assisting deaf-mute people. In: Proceedings of INTERSPEECH, Pittsburgh, USA, pp. 1399–1402 (2006)

  37. San Segundo, R., Barra, R., Córdoba, R., D’Haro, L.F., Fernandez, F., Ferreiros, J., Lucas, J., Macías-Guarasa, J., Montero, J.M., Pardo, J.: Speech to sign language translation system for Spanish. Speech Commun. 50, 1009–1020 (2008). doi: 10.1016/j.specom.2008.02.001

    Google Scholar 

  38. Sandler, W.: Phonological Representation of the Sign: Linearity and Nonlinearity in American sign Language. Foris Publications, Holland (1989)

    Book  Google Scholar 

  39. Sandler, W., Lillo-Martin, D.: Sign Language and Linguistic Universals. Cambridge University Press (2006)

  40. Schembri, A.: Rethinking ‘classifiers’ in signed languages. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 3–34. Psichology Press (2003)

  41. Schembri, A., Jones, C., Burnham, D.: Comparing action gestures and classifier verbs of motion: evidence from Australian sign language, Taiwan sign language, and non signers’ gestures without speech. J. Deaf Stud. Deaf Edu. 10, 272–290 (2005)

    Article  Google Scholar 

  42. Segouat, J.: A study of sign language co articulation. In: Proceedings of SIGACCESS, ACM, New York, pp. 31–38 (2009). doi: 10.1145/1531930.1531935

  43. Segouat, J., Braffort, A.: Toward the study of sign language co articulation: methodology proposal. In: Proceedings of the International Conference on Advances in Computer-Human Interaction, IEEE Computer Society, Los Alamitos, CA, USA, pp. 369–374 (2009). doi:10.1109/ACHI.2009.25

  44. Solina, F., Krapez, S., Jaklic, A., Komac, V.: Multimedia dictionary and synthesis of sign language. In: Rahman, I. (ed.) Design and Management of Multimedia Information Systems, pp. 268–281. Idea Group Publishing (2001)

  45. Stokoe, W.: Sign language structure: an outline of the visual communication systems of the American deaf. Stud. Linguist. Occas. Papers 8, 1–78 (1960)

    Google Scholar 

  46. Stokoe, W.: Sign Language Structure: the First Linguistic Analysis of American Sign Language. Linstok Press Incorporated (1978)

  47. Supalla, T.: Revisiting visual analogy in ASL classifier predicates In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 249–257. Psychology Press (2003)

  48. Sutton, V.: Signwriting. http://www.signwriting.org/ (1974). Accessed 21 Feb 2011

  49. VCom3D.: Signsmith studio. http://www.vcom3d.com/signsmith.php (2009). Accessed 21 Feb 2011

  50. Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N.I., Palmer, M.: A machine translation system from English to American Sign Language. In: Proceedings of AMTA, Springer, London, UK, pp. 54–67 (2000)

  51. Zijl, L.: South African sign language machine translation project. In: Proceedings of SIGACCESS, ACM, New York, NY, USA, pp. 233–234 (2006). doi:10.1145/1168987.1169031

  52. Zwiterslood, I., Verlinden, M., Ros, J., Schoot, S.: Synthetic signing for the deaf: eSign. In: Proceedings of CVHI, Granada, Spain (2004)

Download references

Acknowledgments

The authors would like to acknowledge the teachers of the SIGNAR academy for their participation in the evaluations and to the FPU program of the Universidad Autónoma de Madrid for its financial support. The authors are grateful to the journal’s editor and the reviewers for their comments and suggestions, which have improved this paper and proposed interesting future research directions.

Author information

Authors and Affiliations

  1. Human Computer Technology Laboratory, Universidad Autónoma de Madrid, Francisco Tomás y Valiente 11, 28049, Cantoblanco, Spain

    Fernando López-Colino & José Colás

Authors
  1. Fernando López-Colino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José Colás
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando López-Colino.

Rights and permissions

Reprints and permissions

About this article

Cite this article

López-Colino, F., Colás, J. Hybrid paradigm for Spanish Sign Language synthesis. Univ Access Inf Soc 11, 151–168 (2012). https://doi.org/10.1007/s10209-011-0245-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-011-0245-9

Keywords