Recent developments of vision large language models (LLMs) have seen remarkable progress, yet still encounter challenges towards multimodal generalists, such as coarse-grained instance-level understanding, lack of unified support for both images and videos, and insufficient coverage across various vision tasks. In this paper, we present VITRON, a universal pixel-level vision LLM designed for comprehensive understanding, generating, segmenting, and editing of both static images and dynamic videos. Building on top of an LLM backbone, Vitron incorporates encoders for images, videos, and pixel-level regional visuals within its frontend modules, while employing state-of-the-art visual specialists as its backend, via which Vitron supports a spectrum of vision end tasks, spanning visual comprehension to visual generation, from low level to high level. To ensure an effective and precise message passing from LLM to backend modules for function invocation, we propose a novel hybrid method by simultaneously integrating discrete textual instructions and continuous signal embeddings. Further, we design various pixel-level spatiotemporal vision-language alignment learning for Vitron to reach the best fine-grained visual capability. Finally, a cross-task synergy module is advised to learn to maximize the task-invariant fine-grained visual features, enhancing the synergy between different visual tasks. Demonstrated over 12 visual tasks and evaluated across 22 datasets, Vitron showcases its extensive capabilities in the four main vision task clusters. Overall, this work illuminates the great potential of developing a more unified multimodal generalist.
Figure 1: Vitron supports four 4 vision task clusters, spanning visual comprehension to generation, from low level to high level.
Figure 2: Overview of the Vitron framework.
Specifically, we design a structured invocation template, including 1) Module name, 2) Invocation command, and 3) Region (optional) specifying a fine-grained vision feature needed for certain tasks. The feature embeddings include both task-specific features and task-invariant fine-grained features. The purpose of this design is to achieve feature decoupling, during which we aim to have the task-invariant fine-grained features shared as widely as possible among all tasks to facilitate synergy between different tasks.
Table 1: Summary of backend modules in Vitron.
We train the Vitron to endow it with robust vision understanding and task execution capabilities, with three distinct phases.
Initially, the training equips the multimodal large language model (MLLM) with fundamental multimodal understanding features. This includes aligning the model's decoder with the encoders for a consistent backend response.
The focus is to enhance the model's ability to precisely understand and process both images and videos at a pixel level. The model is trained to accurately map the spatial details of visuals.
To ensure different specialist modules work synergistically in a generalist model, we separate signal feature embeddings into task-specific and task-invariant categories. By sharing task-invariant features widely, the modules can better support each other, enhancing overall system synergy. We introduce a cross-task synergy learning module using adversarial training to differentiate these features. This setup involves specialists making predictions with these features while a discriminator assesses task identity based on shared features alone. When the discriminator fails to recognize the task, the shared features are considered optimally generalized for cross-task application.
Figure 3: Illustration of the synergy module.
Could you help me segment this item?
Sure! Looks like you're aiming to cut out a grey pot.
Absolutely! That would be a dog. Let's outline it.
You are smart! Then highlight the item I pointed out here, will you?
Sure thing! Let's focus on the pot in front the dog.
Could you tell me what's happening in this photo?
The picture features a group of people riding in a small electric propulsion boat on a body of water. There are three people visible in the boat, with one person sitting in the front, another in the middle, and the third person in the back. The boat appears to be a motorboat with an electric motor, providing a more environmentally friendly mode of transportation. The people seem to be enjoying their time on the water, possibly engaging in recreational activities or simply taking a leisurely ride.
So what is the possible relationship between the two persons according to the picture？
In the image, there is a man holding a tennis racket and a chair umpire on a tennis court. The relationship between the two persons is that the chair umpire is responsible for overseeing the match and making decisions related to the game, such as calling balls and strikes, while the man holding the tennis racket is a tennis player participating in the match.
Hmm, it's cool but not quite what I had in mind. Let me paint a picture for you: how about starting with adding a white dog next to the car?
Got it, I've placed a white dog right next to the car, on its right side, to be precise.
Could you modify the chair I marked to give it four legs?
Sure thing! The chair you pointed out is sitting on a metal base. I've swapped that out for four legs instead.
Done. The flower and the wine bottle? Gone. The table's all clear now.
I went ahead and added two white cats lying on the floor. Check out the updated image.
VITRON:@inproceedings{fei2024vitron,
title={VITRON: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing},
author={Fei, Hao and Wu, Shengqiong and Zhang, Hanwang and Chua, Tat-Seng and Yan, Shuicheng},
year={2024},
journal={Proceedings of the Advances in neural information processing systems},
}