Microsoft patents AI that can change people's words and faces in live videos

A person watching a live speech could soon have the option to hear something entirely different from what the speaker actually said. Microsoft is exploring an AI technology that could edit a person's words, remove unwanted expressions and even change their mouth movements and face while a video is still playing. The idea comes from a newly filed Microsoft patent that describes a system capable of making changes to live audio and video in near real time. Instead of simply filtering sound or adding subtitles, the proposed technology would modify the footage itself, making it appear as though the speaker had originally said the altered words.

The system would use AI agents to detect specific words or phrases and decide whether they need to be removed or replaced. Once a change is made to the audio, the technology would also adjust the speaker's mouth movements and facial appearance to match. It would edit the surrounding audio and video as well so that viewers do not notice sudden cuts or interruptions. The changes could be based on predefined instructions or individual user profiles, allowing the same broadcast to appear differently to different viewers.

One of the more practical applications mentioned in the patent is live language translation. A viewer could potentially watch a speech in their preferred language, with the speaker's mouth movements edited to match the translated audio. The system could also make speakers sound more fluent by removing unnecessary pauses, filler words such as 'um' and 'ah', and stutters. It could correct mispronounced or incorrectly used words without requiring the speaker to repeat anything. But Microsoft's proposal does not stop at making speech easier to understand.