Panoptic segmentation is a comprehensive image segmentation task that unifies the goals of semantic segmentation (label every pixel with a class, including background stuff like sky, road, etc.) and instance segmentation (distinguish individual object instances for countable things like people, cars) in one output. In panoptic segmentation, each pixel gets a semantic label and each pixel belonging to a “thing” class (object instances) also gets an instance id. So background classes like “sky” or “grass” have a label but no instance id (they’re amorphous regions), while foreground objects like “car” have a label “car” and an instance ID so that two cars are labeled separately.The term “panoptic” implies seeing everything – both stuff and things. The challenge is to produce a consistent scene parsing. Solutions often combine outputs of a semantic seg model and an instance seg model with some reconciliation step. Metrics for evaluation include panoptic quality (PQ), which penalizes both segmentation and recognition errors. This task was introduced to push the field towards a more unified scene understanding.