Image segmentation
Description
Image segmentation refers to the partitioning of an image into multiple sets of pixels (segments) that share some common characteristics such as color or intensity. The output is a segmentation map, i.e. an alternative representation of the source image that is easier and considerably faster to analyze.
photoQuad is equipped with a statistical region merging algorithm (Nock and Nielsen, 2004), selected upon its efficiency in the successful partitioning of highly complex benthic images. The source image is sequentially segmented into four different scales from coarse to fine detail, facilitating the automatic location of features and boundaries at different levels of detail.
Highlights
Automatic image segmentation to four different scales
Mulitple image segments from different scales can be interactively combined to provide the necessary level of detail required to accurately define and classify regions of interest
Segments can be automatically confined into the quadrat's active area
High-resolution images can be efficiently segmented (4000x3000 pixels or more)
Customizable appearance of all segment objects
Segment-based analysis can be performed simultaneously with any other analysis, e.g. random points, grid cell counts, or freehand species regions
Read more: | Overview | The basics | Walkthrough: from segments to species regions | User interface |
| Region descriptors and worksheet output | FAQs | Credits | tips: Segment extreme
Use the photo gallery below for a graphical illustration of photoQuad's automated multi-scale image segmentation process. Any number of segments from different scales can be combined or used independently to rapidly produce Species regions and a series of quantitative descriptors.
hover mouse over thumbnails to view the different segmentation scales
Before reading any further, we suggest you have a look at the Species regions overview.
Segments are regions on steroids, so being familiar with regions will really help.
that sounds rational, take me there..
ROIs and Species regions
photoQuad regions have a simple hierarchy: you draw an outline, and it's a generic Region Of Interest (ROI). You assign that ROI to a species, and the ROI is promoted to a Species region. Unassign the Species region, and it goes back to a ROI. So ROIs are at the bottom of the object hierarchy, Species regions are at the top. Both ROIs and Species regions operate on separate layers, but only Species regions carry statistical descriptors.
Where do segments fit?
They are even lower than ROIs, and there is a good reason for it. When you segment an image, you get four different segmentation scales, each one of them packed with segments of different detail. All these segments are available to be potentially used for analysis, but most of them will most probably won't. So you start with segments, you select some of them, promote them to ROIs, and then assign them to species. You can do all of that simultaneously; follow the walkthough below for details.
The source image
Everything starts from here. The thumbnail on the right shows a small portion of the original image, but photoQuad can easily segment RGB images with dimensions of 4000x3000 pixels or more.
The source image does not have to be clear of other analysis objects. You can work with random points, grid cell counts, species markers, and image segments simultaneously, they all operate on different layers. If the display gets confusing, just toggle some layers invisible.
Segment map
The segment map is the actual product of the image segmentation process. The thumbnail on the right shows the segmentation map at Scale 2 (low detail level); it goes up to Scale 4 (finest detail), but that would be too cluttered for this demonstration.
The segment map does not have to be filled with color, you can keep only the boundaries. Even better, you can optionally turn its visibility off; you can't see it, but the software can.
Active segment
The active segment is the yellow patch shown on the right, and temporarily highlights the segment that will be selected if you left-click the mouse button. It follows the movements of the mouse while you hover over the image.
There are many ways to customize its appearance, e.g. make it semi-transparent so that you don't lose the image information beneath, or even make it invisible.
Selected segment
A selected segment is something like the green patch shown on the right, and is a segment that can be converted to ROI. Left-click a segment, and it's selected; left-click it again to deselect it; left-click and drag the mouse to select multiple segments consecutively.
As the next two images illustrate, multiple segments can be in the "selected" status, even non-neighboring ones. Actually, you can combine segments from different segmentation scales to produce the desired result. They can be further customized to draw less attention to themselves:
Convert to ROI
This is the part where selected segments permanently detach themselves from the segment map and become ROIs, ready to be assigned to a species. The ROIs are the white regions on the right, and correspond to the selected segments in the previous image.
The nice part is that the segment selection process does not have to stop. You may have segments, ROIs, and Species regions alltogether at once, work with any object accordingly, or turn a layer's visibility off to temporarily remove clutter. It takes some experience to get familiar with the interface, but once there, you'll be happy with how fast and flexible the overall processing becomes.
Assign to species
Now we collect the goodies. ROIs defined through the image segmentation process can be assigned to species (i.e. become Species regions), so that photoQuad can calculate a series of descriptors for them. All calculations take place automatically in the background, just export and you're off for statistics.
Note that once segments are converted to ROIs, the software does not know, nor care, how ROIs were created. Everything that applies to freehand-drawn regions applies here also; a ROI is a ROI no matter how it got there.
For a more advanced walkthrough on how to get the most out of photoQuad, click here
The following image shows the image segmentation interface on photoQuad's floating GUI (Regions tab).
hover mouse over the GUI to see description
- Segmentation scale
Change segmentation scale (1: lowest detail, 4: highest).
The scale can be changed anytime during analysis wihout losing segments already selected with a different scale. This means that you can start creating a region using scale 2, temporarily switch to scale 4 for finer detail, and finish the region selection with scale 1. - Minimum segmentation size
A threshold to prevent the image from being over-segmented. No region smaller than the threshold will be created.
Disabling this option will most probably result into a segment map with numerous tiny regions ,which are too small to be of use, and may cause the interface to become sluggish and confusing. - Show segment map
Toggles visibility of segmentation map patches. - Show boundaries
Toggles visibility of segmentation map boundaries. - Boundaries edge color
Customizes the color of segmentation map boundaries. - Crop to quadrat
Forces the software to segment only the portion of the image that is within the quadrat's boundary.
It has no effect if a quadrat has not been defined.
Note: The appropriate option must be selected before segmenting the image. After that, this checkbox has no effect. If you change your mind, you must re-segment.
If you are analyzing a photoquadrat image, it is recommended to leave this option enabled: it saves a lot of system resources and makes the interface much easier to work with. - Segment image
segment image using current settings - Enable hover & highlight segments
Toggles the ability to interactively select image segments.
When "On", hovering the mouse over the image highlighs the segment that will be selected if left-clicked.
When "Off", segment selections dissapear, segments cannot be selected, and mouse hovering actions return to normal.
Tip: It is possible to maintain the ability to select segments, without highlighting the activated segment below the mouse pointer. Disable the "fill" checkbox and enable the "inverse color" checkbox to do this. This reduces the visual "noise" you have to deal with. - Add/remove.. selection toggle
Controls the action performed when left-clicking and dragging the mouse.
When "add+", moving the mouse around while holding the left button down continuously adds segments to the current selection.
When "rem-", it removes them.
Tip: This is like drawing with a pencil (add+) or with an eraser (rem-). Instead of left-clicking each particular segment, you can select/deselect multiple segments with a continuous mouse movement.
This tool only works when the Enable mouse hover option is "On". - Active segment drawing mode (xor)
Controls the mode used to render the active segment.
If enabled, photoQuad draws the active segment by performing an eXclusive OR (XOR) operation with the image color beneath it. The result is a transparency illusion, that makes easy to identify which segment will be selected if left-clicked, without losing the actual image information beneath.
If disabled, the active segment is drawn normally.
Tip: Disable the "fill" checkbox and enable the "inverse color" checkbox to maintain the ability to select segments, without highlighting the active segment below the mouse pointer (it is like an "I feel lucky" way of working though..) - Fill active segment
Controls whether the active segment is filled with color. If disabled, active segment is transparent and only its boundary is displayed. - Active segment fill color
Customizes the face (fill) color of the active segment. - Active segment edge color
Customizes the edge (boundary) color of the active segment. - Selected segments fill color
Controls whether selected segments are filled with color. If disabled, they are transparent and only their boundaries are displayed. - Allow holes
When enabled, selected segments can have internal holes, if they enclose one or more inner segments. Parent and inner segments are colored differently. - Parent selection fill color
Customizes the face (fill) color of parent selections. - Inner selection fill color
If selections are allowed to have inner holes, this button customizes their face (fill) color. - Convert selected segments to ROIs
Promotes all selected segments into Regions Of Interest (ROIs). After that, each ROI can be assigned to a species and become a "Species Region" - Delete selected segments
Deletes all selected segments.
Species regions or selected segments that have been converted to Regions Of Interest (ROIs) are not affected.
The final regions produced via the image segmentation process are actually Species regions. As such, they share the same output descriptors that are documented here.
Is there anything I need to know before working with image segmentation?
Yes. Please have a look at the Species regions page, so that you are familiar with Regions Of Interest (ROIs) and Species regions.
I know what ROIs and Species regions are, but I am still confused: what is a segment map, an active segment, and a selected segment?
These terms are documented in this walkthrough.
Can other analysis objects be present on the image when working with segments?
Yes. You can simultaneously work with any analysis object, e.g. random points, grid cell counts, species markers, measurement tools, whatever, they all operate on different layers. If the display gets confusing, just toggle some layers invisible.
Are the various image navigation tools available during segment-based processing (e.g. the magnifier, interactive zoom/pan etc)?
Yes. To our surpise also, all these work as normal...
Can I segment the image without having to import a species library?
Yes. A species library is only needed in order to assign Regions Of Interest (ROIs) to species.
Why do you call it "multi-scale" image segmentation?
Because the image is segmented into four different scales from coarse to fine detail, facilitating the automatic location of features and boundaries at different levels of detail.
Can segments from different scales be combined to produce a single ROI?
Yes, that's the point of having multiple scales.
Can I customize the level of detail produced by each segmentation scale?
In a strict sense, no. These settings are hard-coded and refined to produce optimum results in a wide range of scenarios.
But there is a workaround: you can increase the produced detail by lowering (or even unchecking) the "minimum segmentation threshold" checkbox. Please refer to this walkthrough for taking the most out of photoQuad's segment abilities.
Can ROIs produced from image segments be re-attached to the segment map?
No. The segment map provides the background for ROIs to be rapidly defined. Converting a particular group of segments into a ROI does not affect the segment map underneath. Just delete a ROI if you don't need it.
When I convert segments to Species regions, do I have to force photoQuad to calculate or update the region's descriptors?
No. This takes place automatically in the background. Even better, photoQuad automatically updates region descriptors each time the quadrat boundary, the calibration data or the region itself is modified. No matter when you request for a summary report and worksheet export, you will always get the updated information.
Can I save the segmentation map so that I can reuse it in another session?
No. The segment map cannot be saved; you can only save the ROIs and Species regions layers produced. Note however that since segmentation settings are hard-coded in the software, multiple segmentation attempts will produce identical results for a particular image. So, to continue work from a previous session, load the previous layers, segment the image again, and take it from there.
Can I modify the segment map?
No, but you can modify the ROIs and Species regions produced.
Can segments be confined within the quadrat's boundary?
Yes. Enable the "Crop to quadrat" checkbox in photoQuad's floating GUI. This not only looks better, but also saves system resources because image segmentation is demanding in computer memory.
I understand that segments can be optionally allowed to have internal holes, but that doesn't seem to make any difference in the output Species region descriptors. Why?
Please read the answer to the last question in the Species regions FAQs. The work with parent and child regions is still under progress, and currently the software simply ignores inner holes when calculating descriptors. Whether you use them or not, results will be identical as if holes where not there.
Can I make all segmentation objects temporarily dissapear, so that I can work on another analysis?
Yes. You must hide the segmentation map and disable the segment selection mouse hover function. To do this, go to photoQuad's floating GUI, disable the "Show segment map" and "Show boundaries" checkboxes, and toggle the Mouse hover button to "Off". All segmentation info is thereby hidden, but you can still come back to it and continue your work.
Segmentation method
photoQuad's image segmentation is based on the SRM algorithm developed by Nock and Nielsen (2004), selected upon its efficiency in the successful partitioning of highly complex benthic images.
Nock R., Nielsen, F., (2004). Statistical region merging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(11):1452-1458.
original paper available by IEEE at http://dx.doi.org/10.1109/TPAMI.2004.110
or at Prof. Nock's personal webpage: http://www1.univ-ag.fr/~rnock/
We thank the authors for publishing their research.
Segmentation code
The MATLAB implementation of Nock and Nielsen's algorithm was based on the work of Sylvain Boltz, who made the code publicly available at MATLAB's File Exchange community under the BSD license.
published as: Image segmentation using statistical region merging by Sylvain Boltz
original code available at: http://www.mathworks.com/matlabcentral/fileexchange/authors/73145
or at Sylvain's personal webpage: http://www.lix.polytechnique.fr/~boltz/index.php
We used Sylvain's MATLAB code as a starting basis, we modified it here and there, and implemented it into photoQuad's interface, building all the supporting functions and user interface tools from scratch.
We thank the author for sharing this brilliant piece of code.
Special thanks
We greatly appreciate the specific support of Nuria Teixido and colleagues (Institut Ciencies del Mar, ICM-CSIC), who provided valued insights into the Seascape software and the application of image segmentation to benthic images.
check Seascape out at: http://www.seascapesoft.com
Please refer to this walkthrough for taking the most out of photoQuad's segment abilities.
