The devil is in the fine-grained details: evaluating open-vocabulary object detectors for fine-grained understanding

Bianchi L., Carrara F., Messina N., Gennaro C., Falchi F.

Computer Science - Machine Learning  Computer Vision and Pattern Recognition (cs.CV)  FOS: Computer and information sciences  open-vocabulary detection  Artificial Intelligence (cs.AI)  fine-grained understanding  Machine Learning (cs.LG)  benchmark  Computer Science - Artificial Intelligence  Computer Science - Computer Vision and Pattern Recognition 

Recent advancements in large vision-language models enabled visual object detection in open-vocabulary scenar-ios, where object classes are defined in free-text formats during inference. In this paper, we aim to probe the state-of-the-art methods for open-vocabulary object detection to determine to what extent they understand finegrained prop-erties of objects and their parts. To this end, we intro-duce an evaluation protocol based on dynamic vocabulary generation to test whether models detect, discern, and as-sign the correct fine-grained description to objects in the presence of hard-negative classes. We contribute with a benchmark suite of increasing difficulty and probing dif-ferent properties like color, pattern, and material. We fur-ther enhance our investigation by evaluating several state-of-the-art open-vocabulary object detectors using the proposed protocol and find that most existing solutions, which shine in standard open-vocabulary benchmarks, struggle to accurately capture and distinguish finer object details. We conclude the paper by highlighting the limitations of current methodologies and exploring promising research directions to overcome the discovered drawbacks. Data and code are available at https://lorebianchi98.github.io/FG-OVD/.

Source: PROCEEDINGS IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, pp. 22520-22529. Seattle (USA), 17-21/06/2024

Publisher: IEEE

Metrics