Title: Image and Video Tokenization with Binary Spherical Quantization
Authors: Yue Zhao, Yuanjun Xiong, Philipp Krähenbühl
Published: 11th June 2024 (Tuesday) @ 17:59:53
Link: http://arxiv.org/abs/2406.07548v1
Abstract
We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100 with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4 throughput compared to the best prior methods. Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN- and diffusion-based methods.
A tokenization-based compression algorithm has three basic steps: A visual tokenizer, i.e. VQVAE [8] or LFQ [17], translates raw visual inputs to a discrete set of tokens and back. A sequence model then predicts an auto-regressive probability distribution over these discrete tokens. Finally, arithmetic coding translates this distribution into a compressed representation.
- LFQ: Lookup-free Quantization from Language Model Beats Diffusion — Tokenizer is Key to Visual Generation