Core Technology

Implemented

These capabilities have been implemented on our robots, from lip movements and preset expressions to conversation.

Lip sync

From text to lip movements

  1. Voice inputText
  2. TTS speech synthesis
  3. Motion predictionAudio-to-motion model
  4. Physical executionMulti-axis servo commands
  5. Robot lip movements

Paired data from real robots establishes the mapping between mouth shapes and movements.

Typical phonemes and mouth shapes
PhonemesMouth shape category
/p/ /b/ /m/Bilabial
/f/ /v/Labiodental
/θ/ /ð/Dental
/u/ /o/ /w/Rounded
/a/ /æ/Open vowels

Typical sounds

In order: “bàba, bàibài” (bilabial), “fēi” (labiodental), “wūguī” (rounded). EVA Gen 2, mouth only.

Robot voice is AI-synthesized.

EVA Gen 2 lip movement uses 13 channels, driven in real time by Chinese and English speech.

Bionic facial structures & expressions

Soft silicone lips and coordinated actuators for the upper lip, lower lip and mouth corners produce preset expressions.

The emotion in the robot's reply triggers a matching preset expression, and it blinks naturally when idle.

Multilingual conversation

Conversation can follow the user as they switch between 14 languages, for speech recognition and replies.

Long-term memory, with character personas and voices tailored to each role.

Languages include Mandarin Chinese, Cantonese, English, Japanese, Korean, Russian, Vietnamese, Spanish, French, German, Arabic, Portuguese, Italian and Polish.

Emotional state

The robot keeps a simulated emotional state: six moods — happy, sad, upset, surprised, shy and sleepy — build up and fade on their own, persist across conversations, and the tone of some Chinese voices adjusts accordingly.

Vision and proactive conversation

The camera periodically observes the surroundings and the visual model's descriptions feed into conversation and memory; after the user has been quiet for a while, the robot may speak up on its own based on what it has seen and heard.

Research & development

The following capabilities remain in development.

In development

Eye tracking

Demonstrated on hardware and still in development.

In development

Model-driven expressions

Around 16,000 expression images and corresponding control commands have been collected from real robots. Human expressions can drive a digital robot avatar; bringing this to hardware is the next step.

Developed, not yet available

The following capabilities have been developed technically and are not currently available to users.

Not yet available

Face and voiceprint recognition

Used to tell people apart. This involves biometric personal information; it will only be enabled with the user's separate consent and after a compliance assessment.

Not yet available

Facial expression recognition

Recognizes the user's facial expressions so the robot's responses fit the moment. It likewise requires user consent and a compliance assessment before being enabled.

Connect through conversation

For business cooperation, contact Kathy Hu.

Business Cooperation