What is Meta’s Segment Anything AI Model and why should you care?

When thinking about AI, we now mostly think of chatbots such as ChatGPT, which made quite a splash last year with their auto-generated content. However, AI is not only about writing stories and compiling information from different sources.

Meta AI’s new Segment Anything Model (SAM) might be a revolutionary step forward in how computers see and process images. The new model promises to be a huge step forward in image segmentation, meaning that it will likely influence both commercial technologies like VR and help scientists in their research.

What is the Segment Anything Model?

First, let’s look at the new Segment Anything Model. One of the most critical elements when developing computer vision – the way computers can process and analyze visual data to categorize or extract information – is segmentation. Segmentation basically means the ability of a computer to take the image and divide it into functional elements, such as distinguishing between background and foreground, recognizing individual persons in the picture, or separating only the part of the picture where there is a jacket.

Meta’s Segment Anything Model is actually a set of new tasks, a dataset, and a model that all work together to enable a much more efficient segmentation method. The Segment Anything Model features the most extensive segmentation dataset to date (called the Segment Anything 1-Billion mask dataset).

Meta’s SAM is an image segmentation model that can respond to user prompts or clicks to select objects in their chosen image, making it extremely powerful and easy to use. Interestingly, Meta also announced that the SAM model and the dataset will be available to researchers under an open Apache 2.0 license.

You can already try the demo of this model on Meta’s website. It shows off three capabilities of the model – selecting an object with a mouse click, creating a semantic object within a chosen box in a picture, or segmenting all the objects in the image.

Why is SAM different from other segmentation methods?

Segment Anything Model certainly isn’t the first image segmentation solution, so why is it such a big deal? The difference between these older models and Meta’s approach is the way in which they are trained. So far, there have been two main approaches to this problem:

• Interactive segmentation allows the model to separate any object category in the image, but it has to be first trained and relies on human input to identify each object category correctly
• Automatic segmentation only allows selecting predefined object categories and can be trained wholly automatically, but requires many examples to start working efficiently. For example, if you want it to be able to recognize dogs in pictures, you first need to supply it with tens of thousands of dog pictures to train and “recognize”.

Conversely, Meta’s Segment Anything Model is essentially a synthesis of both of these approaches. On the one hand, it was trained on a huge dataset of over 1 billion masks from 11 million pictures. On the other hand, it can also recognize and segment object categories that it wasn’t trained on, thanks to the ability to generalize its training and apply it outside of its expertise.

Moreover, SAM is a promotable model that segments based on the user’s input. This means that it can be easily used in various scenarios, making it easy to implement and change based on the needs of a specific task.

Why is the Segment Anything Model important?

Generally, one of the biggest strengths of the newly-developed Segment Anything Model by Meta is its customizability. Because of its generalized nature – it can segment even the objects it wasn’t trained on – it is (relatively) extremely easy to customize that model and implement it in various use cases.

Image segmentation is crucial for all the AI and machine-learning-based tasks that have to do with images, as this is a way for these models to recognize and analyze visuals. Therefore, having a generalized model that does not require specialized training for every scenario, or at least extremely, reduces the time and resources needed. Meta claims it’s a big step toward democratizing AI, making it possible to use computer vision even with limited budgets and time.

As segmentation models are a crucial part of any AI, Meta’s efforts can significantly impact many industries. One of the obvious ones is virtual reality/augmented reality, which uses segmentation models to recognize what users are looking at and integrate these prompts into VR applications.

Content creation is another area where the Segment Anything Model can have a huge impact. Meta believes that SAM could greatly help photo or video editors, enabling them to quickly and efficiently extract pieces of images and videos, making the editing process faster and easier.

Meta also believes that such a model can greatly help researchers who rely on various forms of visual data. The company gives a few examples: nature researchers who capture footage of animals could use the model to identify the particular species they are looking for, and astronomers could employ the model in their research of the universe at large.

There are many more use cases for the model that Meta advertises. Because of the open nature of the company’s license, SAM will be available for all to try out and utilize in their projects. You can already get the code on GitHub, so if you want to try implementing the model, it’s available here.