Researchers and their AI helpers have been hitting some speed bumps in the quest for automating the retrieval of nature images from vast databases. A group from MIT and collaborators recently put computer vision models to the test, and the results were a mixed bag, according to a report by MIT News. While the latest tech can fetch pictures of jellyfish washed ashore just fine, it's not quite up to snuff when the search gets more complex, like singling out a frog with axanthism.
It's no small feat to sift through images when there are about 11,000 North American tree species alone, let alone the rest of Earth's flora and fauna captured in millions of photos. Trying to assist in this daunting task are multimodal vision language models (VLMs), trained on text and images. However, as MIT News described, more nuanced queries that require deep ecological knowledge have these systems struggling.
MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), together with University College London and other hands on deck, designed a benchmark to gauge the performance of these VLMs. Using the "INQUIRE" dataset, which boasts 5 million wildlife images and 250 ecologist-created search prompts, the team found that the bigger and beefier the model, the better it did, especially with basic searches. But when you get into the weeds with scientific lingo or distinct biological phenomena, even the robust models like GPT-4o were left wanting, achieving a precision score of only 59.6 percent, as Edward Vendrow, an MIT PhD student and a researcher on the dataset, told MIT News.
Given the sheer number of images, manual curation is time-consuming and not a job for the faint-hearted. Sara Beery, the assistant professor involved in directing this study, emphasized the meticulousness required to accurately catalog the thousands of images that met their specific criteria. The researchers fed images of hermit crabs housing themselves in plastic or condors tagged with '26' into VLMs to assess how well the models could parse through and retrieve relevant images. The gap between human annotators' precision and VLMs' capabilities illustrated just how much growing these systems have to do, as suggested by MIT News.
The shortcomings of these vision models don't spell doom for the endeavor; rather, they are more like growing pains as AI continues to learn the ropes of the scientific world. By better understanding the nuances of scientific queries, VLMs stand to become powerful tools, aiding not just ecologists, but anyone dealing with large image databases. For now, though, AI can take a break for it, the mountain of biodiversity data remains a daunting climb, as MIT News highlighted in their findings.
