X (formerly Twitter) Premium subscribers can now ask the Grok AI assistant to describe images, not just make them. The Elon Musk-owned company xAI unveiled a new feature for visual content analysis, giving it the ability to describe photos, diagrams, and other snapshots using the Grok-2 AI model which powers the AI chatbot and its Flux AI image creation.
The feature brings Grok to parity with ChatGPT, Gemini, and other rivals. If you subscribe to X’s subscription plans, you can try it out now by clicking on a button in an image post within X and asking Grok questions about the image or just for a straight descriptive analysis.
In tandem with the new feature, Grok showed off a new benchmark called RealWorldQA that is supposed to show how well a model can describe a real-world image, including the space between objects. The company claims RealWorldQA shows Grok to be as good or better than its rivals at explaining images even though it’s still in development. You can see an example below of how it works, shared on X by Elon Musk.
Grok now understands images, even explaining the meaning of a joke.This is an early version. It will rapidly improve. https://t.co/gQ5BBISVRcOctober 28, 2024
See and Grok
As the screenshot illustrates, Grok is capable of breaking down a complex multi-stage image and explaining what happens in it. It can then extrapolate the humor of the joke, though, as is almost always the case, explaining the joke makes it much less funny. Still, it’s a sign that xAI is not done with putting out new features for Grok, especially multimodal tools. This could be a step toward Grok being able to explain audio and video content the same way it does with visuals.
One element not mentioned is how the visual analysis by Grok might portray the freewheeling image creation by the AI chatbot that seems to have little or no compunction about copyright issues. It’s something that users making images of Mario faced when Nintendo’s copyright infringement hunter Tracer went after them for infringement. Whether an AI image of Mario or any other intellectual property would be described as such or in more generic terms would be interesting to discover.
xAI’s owner being who he is, there’s also very obvious potential for the feature in other Musk-owned technology companies. Tesla’s semi-autonomous driving would certainly benefit from being able to identify people and objects around it and how they are spaced apart. The same goes for the long-promised humanoid robots Tesla’s had under development for the last few years.