Generative AI, the tech behind popular programs such as Midjourney, Stable Diffusion, and DALL-E 2 that allows anyone to turn words into inspiring pictures, has been gaining a lot of mainstream attention recently. And few groups seem more excited about it than writers. Using written language to build images is what they do. But often, to introduce fans to characters and worlds, they must resort to a little help from borrowed photos picked on Google. This new generation of AI tools presentS a much better way.
Best-selling Brazilian author PJ Caldas used that technology to release a book trailer for his first novel in English. The 82-second meditative animation uses AI for all aspects of the production—from design to motion and music—and can be found at http://www.pjcaldas.com/and his Instagram channel @thegirlfromwudang.
The film, inspired by the upcoming novel THE GIRL FROM WUDANG, uses visual references ranging from old Chinese painting and calligraphy to modern comic books and graffiti. Combined, they build a unique look that drags the audience into the universe of the story. Even the theme song was composed by AI and then performed by real people. Animator Nick Tengri from iamYork and music composers and sound designers from Quiet City lent their extensive experience with big studio movies and Netflix series to the project.
“Allowing the AI to create every frame by itself, with zero interference from humans, was nerve-racking, and that’s what made it so interesting,” said PJ Caldas. The only handmade parts were the title cards, because, ironically, Generative AI can build incredible images but can’t write. “We left any writing inside of the animation exactly as the AI created, but if you look close enough, it’s all gibberish.”
Artificial intelligence is one of the major inspirations of The Girl from Wudang, which even features AI characters in the plot. “Allowing AI to create the book trailer was like letting the characters tell their own story,”said the writer.
The technology is developing so fast the novelist didn’t want to wait for the book release, one year from now, to launch the trailer. Instead, he is already looking for other opportunities to experiment with AI. As the novelist was finishing this animation, for example, companies like Meta and Google released research papers with promising results with their tools that suggest users will soon be capable of generating realistic videos out of prompts too. “In the future, we may be able to upload the text and readers will choose between reading, listening or watching it,” PJ Caldas predicts.
In the meantime, he and his digital characters will focus on making the best books he can write, and the strongest trailers AI can produce. PJ Caldas is an award-winning, internationally best-selling writer with four fiction books published in Brazil. The Girl from Wudang is his first novel in English.