Pixel 11 Pro Fold Translates Sign Language into Text
Google's new foldable phone uses the front camera and an advanced AI model to transcribe American Sign Language, while the Gemini app simultaneously breaks records with a billion users.
Google's new foldable phone uses the front camera and an advanced AI model to transcribe American Sign Language, while the Gemini app simultaneously breaks records with a billion users.
Google officially unveiled the Pixel 11 series on Tuesday, with the Pixel 11 Pro Fold drawing the most attention thanks to an innovative feature that could significantly ease communication for deaf and hard-of-hearing individuals. The phone can translate American Sign Language (ASL) into written text in real time, using the front camera to recognize gestures and the external display to show the translation.
The feature is activated by opening the Translate app and tapping the sign language icon in the Gboard toolbar. As the user performs gestures in front of the phone, the translation is simultaneously displayed on the cover screen, allowing the conversation partner to read the message immediately and respond. The feature is available across the entire Pixel 11 series, not just the Pro Fold model.
At the core of this functionality is Google's new sign-language-to-text model, named SL2T. The model is capable of "seeing" and understanding physical movements, including hand, torso, and head gestures, as well as facial expressions. Based on this information, it interprets the meaning conveyed through sign language and translates it into English.
According to data published by Android Authority, the model was trained on over 100,000 hours of sign language videos covering more than 50 languages. Approximately a quarter of this data pertains specifically to American Sign Language, which explains why ASL is the first supported language. For now, the feature is available only in select countries and regions and exclusively for translating ASL into English.
In addition to sign language transcription, the Pixel 11 Pro Fold also supports speech-to-text transcription, a useful accessibility option for individuals with hearing impairments. The app also offers a text field for typing responses or questions that the other person can read. Google has confirmed that support for additional devices and languages will be expanded soon.
Alongside the hardware launch, Google also boasted impressive figures for its AI platform. The Gemini app has reached one billion monthly users, making it the fastest-growing product in the company's history. "More than a billion people now use the Gemini app every month to spark new ideas and get things done. It's our fastest-growing product ever and the fourteenth to reach the billion-user milestone," wrote Sundar Pichai, Google's CEO, in a post on X on Monday.
Josh Woodward, Google's executive in charge of Gemini, also shared some interesting insights in a post on X. A full 63% of users now interact with the Gemini app directly by voice, and busy parents are 43% more likely to use voice commands for everyday tasks. Woodward also announced the rollout of more than 60 new regional dialects in the near future.
Alongside the hardware and popularity news, Google also shared updates in the healthcare field. Research teams from Google Research and Google DeepMind have developed AMIE, an AI system that can now conduct real-time video consultations. In a randomized study using simulated consultations, the system met clinical benchmarks, demonstrating the potential of artificial intelligence to expand access to telemedicine, Google for Health announced on X on Monday.
The Pixel 11 Pro Fold is available for pre-order at a starting price of $1,899 and goes on sale on August 20. Every purchaser also receives six months of free Google AI Pro subscription with the device.
