Fix holes in exported instance masks (#3044)

* update datumaro dependency

* update coco format

* update format docs

- Update COCO documentation
- Add Datumaro format description
- Add TF guide on mask exporting
- Add keypoint export guide in COCO

* Add mask export docs

* update changelog

* add camvid

* add coco

* Update changelog

CHANGELOG.md

@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [1.4.0] - Unreleased
 
 ### Added
--
+- Documentation on mask annotation (<https://github.com/openvinotoolkit/cvat/pull/3044>)
 
 ### Changed
 -
@@ -20,7 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 -
 
 ### Fixed
--
+- Export of instance masks with holes (<https://github.com/openvinotoolkit/cvat/pull/3044>)
 
 ### Security
 -

cvat/apps/dataset_manager/formats/README.md

@@ -11,6 +11,7 @@
 - [How to add a format](#how-to-add)
 - [Format descriptions](#formats)
   - [CVAT](#cvat)
+  - [Datumaro](#datumaro)
   - [LabelMe](#labelme)
   - [MOT](#mot)
   - [MOTS](#mots)
@@ -175,7 +176,7 @@ features, so it can be used to make data backups.
 
 - [Format specification](/cvat/apps/documentation/xml_format.md)
 
-#### CVAT for images dumper
+#### CVAT for images export
 
 Downloaded file: a ZIP file of the following structure:
 
@@ -189,7 +190,7 @@ taskname.zip/
 
 - tracks are split by frames
 
-#### CVAT for videos dumper
+#### CVAT for videos export
 
 Downloaded file: a ZIP file of the following structure:
 
@@ -207,6 +208,17 @@ taskname.zip/
 
 Uploaded file: an XML file or a ZIP file of the structures above
 
+### Datumaro format <a id="datumaro" />
+
+[Datumaro](https://github.com/openvinotoolkit/datumaro/) is a tool, which can
+help with complex dataset and annotation transformations, format conversions,
+dataset statistics, merging, custom formats etc. It is used as a provider
+of dataset support in CVAT, so basically, everything possible in CVAT
+is possible in Datumaro too, but Datumaro can offer dataset operations.
+
+- supported annotations: any 2D shapes, labels
+- supported attributes: any
+
 ### [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/)<a id="voc" />
 
 - [Format specification](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/devkit_doc.pdf)
@@ -219,9 +231,10 @@ Uploaded file: an XML file or a ZIP file of the structures above
 
 - supported attributes:
 
-  - `occluded`
+  - `occluded` (both UI option and a separate attribute)
   - `truncated` and `difficult` (should be defined for labels as `checkbox` -es)
   - action attributes (import only, should be defined as `checkbox` -es)
+  - arbitrary attributes (in the `attributes` secion of XML files)
 
 #### Pascal VOC export
 
@@ -229,7 +242,7 @@ Downloaded file: a zip archive of the following structure:
 
 ```bash
 taskname.zip/
-├── JpegImages/
+├── JPEGImages/
 │   ├── <image_name1>.jpg
 │   ├── <image_name2>.jpg
 │   └── <image_nameN>.jpg
@@ -499,17 +512,47 @@ zip images.zip -j -@ < train.txt
 
 - [Format specification](http://cocodataset.org/#format-data)
 
-#### COCO dumper description
+#### COCO export
 
-Downloaded file: single unpacked `json`.
+Downloaded file: a zip archive with following structure:
+
+```bash
+archive.zip/
+├── images/
+│   ├── <image_name1.ext>
+│   ├── <image_name2.ext>
+│   └── ...
+└── annotations/
+    └── instances_default.json
+```
 
 - supported annotations: Polygons, Rectangles
+- supported attributes:
+  - `is_crowd` (checkbox or integer with values 0 and 1) -
+    specifies that the instance (an object group) should have an
+    RLE-encoded mask in the `segmentation` field. All the grouped shapes
+    are merged into a single mask, the largest one defines all
+    the object properties
+  - `score` (number) - the annotation `score` field
+  - arbitrary attributes - will be stored in the `attributes` annotation section
+
+
+*Note*: there is also a [support for COCO keypoints over Datumaro](https://github.com/openvinotoolkit/cvat/issues/2910#issuecomment-726077582)
 
-#### COCO loader description
+1. Install [Datumaro](https://github.com/openvinotoolkit/datumaro)
+  `pip install datumaro`
+1. Export the task in the `Datumaro` format, unzip
+1. Export the Datumaro project in `coco` / `coco_person_keypoints` formats
+  `datum export -f coco -p path/to/project [-- --save-images]`
 
-Uploaded file: single unpacked `*.json` .
+This way, one can export CVAT points as single keypoints or
+keypoint lists (without the `visibility` COCO flag).
 
-- supported annotations: Polygons, Rectangles (if `segmentation` field is empty)
+#### COCO import
+
+Uploaded file: a single unpacked `*.json` or a zip archive with the structure above (without images).
+
+- supported annotations: Polygons, Rectangles (if the `segmentation` field is empty)
 
 #### How to create a task from MS COCO dataset
 
@@ -559,25 +602,48 @@ image_feature_description = {
 }
 ```
 
-#### TFRecord dumper description
+#### TFRecord export
 
 Downloaded file: a zip archive with following structure:
 
 ```bash
 taskname.zip/
-├── task2.tfrecord
+├── default.tfrecord
 └── label_map.pbtxt
+
+# label_map.pbtxt
+item {
+	id: 1
+	name: 'label_0'
+}
+item {
+	id: 2
+	name: 'label_1'
+}
+...
 ```
 
-- supported annotations: Rectangles
+- supported annotations: Rectangles, Polygons (as masks, manually over [Datumaro](https://github.com/openvinotoolkit/datumaro/blob/develop/docs/user_manual.md))
 
-#### TFRecord loader description
+How to export masks:
+1. Export annotations in `Datumaro` format
+1. Apply `polygons_to_masks` and `boxes_to_masks` transforms
+  ```bash
+  datum transform -t polygons_to_masks -p path/to/proj -o ptm
+  datum transform -t boxes_to_masks -p ptm -o btm
+  ```
+1. Export in the `TF Detection API` format
+  ```bash
+  datum export -f tf_detection_api -p btm [-- --save-images]
+  ```
+
+#### TFRecord import
 
 Uploaded file: a zip archive of following structure:
 
 ```bash
 taskname.zip/
-└── task2.tfrecord
+└── <any name>.tfrecord
 ```
 
 - supported annotations: Rectangles
@@ -706,7 +772,7 @@ python create_pascal_tf_record.py --data_dir <path to VOCdevkit> --set train --y
 
 ### [MOT sequence](https://arxiv.org/pdf/1906.04567.pdf)<a id="mot" />
 
-#### MOT Dumper
+#### MOT export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -735,7 +801,7 @@ person
 - supported annotations: Rectangle shapes and tracks
 - supported attributes: `visibility` (number), `ignored` (checkbox)
 
-#### MOT Loader
+#### MOT import
 
 Uploaded file: a zip archive of the structure above or:
 
@@ -749,7 +815,7 @@ taskname.zip/
 
 ### [MOTS PNG](https://www.vision.rwth-aachen.de/page/mots)<a id="mots" />
 
-#### MOTS PNG Dumper
+#### MOTS PNG export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -773,7 +839,7 @@ person
 
 - supported annotations: Rectangle and Polygon tracks
 
-#### MOTS PNG Loader
+#### MOTS PNG import
 
 Uploaded file: a zip archive of the structure above
 
@@ -781,7 +847,7 @@ Uploaded file: a zip archive of the structure above
 
 ### [LabelMe](http://labelme.csail.mit.edu/Release3.0)<a id="labelme" />
 
-#### LabelMe Dumper
+#### LabelMe export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -793,7 +859,7 @@ taskname.zip/
 
 - supported annotations: Rectangles, Polygons (with attributes)
 
-#### LabelMe Loader
+#### LabelMe import
 
 Uploaded file: a zip archive of the following structure:
 
@@ -811,7 +877,7 @@ taskname.zip/
 
 ### [ImageNet](http://www.image-net.org)<a id="imagenet" />
 
-#### ImageNet Dumper
+#### ImageNet export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -835,7 +901,7 @@ taskname.zip/
 
 - supported annotations: Labels
 
-#### ImageNet Loader
+#### ImageNet import
 
 Uploaded file: a zip archive of the structure above
 
@@ -843,7 +909,7 @@ Uploaded file: a zip archive of the structure above
 
 ### [CamVid](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/)<a id="camvid" />
 
-#### CamVid Dumper
+#### CamVid export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -873,7 +939,7 @@ has own color which corresponds to a label.
 
 - supported annotations: Rectangles, Polygons
 
-#### CamVid Loader
+#### CamVid import
 
 Uploaded file: a zip archive of the structure above
 
@@ -881,7 +947,7 @@ Uploaded file: a zip archive of the structure above
 
 ### [WIDER Face](http://shuoyang1213.me/WIDERFACE/)<a id="widerface" />
 
-#### WIDER Face Dumper
+#### WIDER Face export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -899,19 +965,21 @@ taskname.zip/
 ```
 
 - supported annotations: Rectangles (with attributes), Labels
-- supported attributes: `blur`, `expression`, `illumination`,
-  `occluded` (both the annotation property & an attribute), `pose`, `invalid`
+- supported attributes:
+  - `blur`, `expression`, `illumination`, `pose`, `invalid`
+  - `occluded` (both the annotation property & an attribute)
 
-#### WIDER Face Loader
+#### WIDER Face import
 
 Uploaded file: a zip archive of the structure above
 
 - supported annotations: Rectangles (with attributes), Labels
-- supported attributes: `blur`, `expression`, `illumination`, `occluded`, `pose`, `invalid`
+- supported attributes:
+  - `blur`, `expression`, `illumination`, `occluded`, `pose`, `invalid`
 
 ### [VGGFace2](https://github.com/ox-vgg/vgg_face2)<a id="vggface2" />
 
-#### VGGFace2 Dumper
+#### VGGFace2 export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -934,7 +1002,7 @@ label1 <class1>
 
 - supported annotations: Rectangles, Points (landmarks - groups of 5 points)
 
-#### VGGFace2 Loader
+#### VGGFace2 import
 
 Uploaded file: a zip archive of the structure above
 
@@ -942,7 +1010,7 @@ Uploaded file: a zip archive of the structure above
 
 ### [Market-1501](https://www.aitribune.com/dataset/2018051063)<a id="market1501" />
 
-#### Market-1501 Dumper
+#### Market-1501 export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -970,7 +1038,7 @@ s1 - sequence
 
 - supported annotations: Label `market-1501` with atrributes (`query`, `person_id`, `camera_id`)
 
-#### Market-1501 Loader
+#### Market-1501 import
 
 Uploaded file: a zip archive of the structure above
 
@@ -978,7 +1046,7 @@ Uploaded file: a zip archive of the structure above
 
 ### [ICDAR13/15](https://rrc.cvc.uab.es/?ch=2)<a id="icdar" />
 
-#### ICDAR13/15 Dumper
+#### ICDAR13/15 export
 
 Downloaded file: a zip archive of the following structure:
 
@@ -1027,7 +1095,7 @@ taskname.zip/
 - supported annotations: Rectangles and Polygons with label `icdar`
   and attributes `index`, `text`, `color`, `center`
 
-#### ICDAR13/15 Loader
+#### ICDAR13/15 import
 
 Uploaded file: a zip archive of the structure above
 

cvat/apps/dataset_manager/formats/coco.py

@@ -19,7 +19,8 @@ def _export(dst_file, task_data, save_images=False):
     dataset = Dataset.from_extractors(CvatTaskDataExtractor(
         task_data, include_images=save_images), env=dm_env)
     with TemporaryDirectory() as temp_dir:
-        dataset.export(temp_dir, 'coco_instances', save_images=save_images)
+        dataset.export(temp_dir, 'coco_instances', save_images=save_images,
+            merge_images=True)
 
         make_zip_archive(temp_dir, dst_file)
 

cvat/apps/documentation/user_guide.md

@@ -29,6 +29,7 @@
   - [OpenCV Tools](#opencv-tools)
   - [Annotation with rectangle by 4 points](#annotation-with-rectangle-by-4-points)
   - [Annotation with polygons](#annotation-with-polygons)
+    - [Creating masks](#creating-masks)
   - [Annotation with polylines](#annotation-with-polylines)
   - [Annotation with points](#annotation-with-points)
     - [Points in shape mode](#points-in-shape-mode)
@@ -1282,6 +1283,67 @@ To edit a polygon you have to click on it while holding `Shift`, it will open th
 
   ![](static/documentation/images/gif007_mapillary_vistas.gif)
 
+### Cutting holes in polygons
+
+Currently, CVAT does not support cutting transparent holes in polygons. However,
+it is poissble to generate holes in exported instance and class masks.
+To do this, one needs to define a background class in the task and draw holes
+with it as additional shapes above the shapes needed to have holes:
+
+The editor window:
+  ![The editor](static/documentation/images/mask_export_example1_editor.png)
+
+Remember to use z-axis ordering for shapes by \[\-\] and \[\+\, \=\] keys.
+
+Exported masks:
+  ![A class mask](static/documentation/images/mask_export_example1_cls_mask.png)  ![An instance mask](static/documentation/images/mask_export_example1_inst_mask.png)
+
+Notice that it is currently impossible to have a single instance number for
+internal shapes (they will be merged into the largest one and then covered by
+"holes").
+
+### Creating masks
+
+There are several formats in CVAT that can be used to export masks:
+- `Segmentation Mask` (PASCAL VOC masks)
+- `CamVid`
+- `MOTS`
+- `ICDAR`
+- `COCO` (RLE-encoded instance masks, [guide](https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/dataset_manager/formats/README.md#coco))
+- `TFRecord` ([over Datumaro](https://github.com/openvinotoolkit/datumaro/blob/develop/docs/user_manual.md), [guide](https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/dataset_manager/formats/README.md#tfrecord)):
+- `Datumaro`
+
+An example of exported masks (in the `Segmentation Mask` format):
+
+  ![A class mask](static/documentation/images/exported_cls_masks_example.png) ![An instance mask](static/documentation/images/exported_inst_masks_example.png)
+
+Important notices:
+- Both boxes and polygons are converted into masks
+- Grouped objects are considered as a single instance and exported as a single
+  mask (label and attributes are taken from the largest object in the group)
+
+#### Class colors
+
+All the labels have associated colors, which are used in the generated masks.
+These colors can be changed in the task label properties:
+
+  ![](static/documentation/images/label_color_picker.jpg)
+
+Label colors are also displayed in the annotation window on the right panel,
+where you can show or hide specific labels
+(only the presented labels are displayed):
+
+  ![](static/documentation/images/label_panel_anno_window.jpg)
+
+A background class can be:
+- A default class, which is implicitly-added, of black color (RGB 0, 0, 0)
+- `background` class with any color (has a priority, name is case-insensitive)
+- Any class of black color (RGB 0, 0, 0)
+
+To change backgound color in generated masks (default is black),
+change `background` class color to the desired one.
+
+
 ## Annotation with polylines
 
 It is used for road markup annotation etc.

cvat/requirements/base.txt

@@ -50,4 +50,4 @@ open3d==0.11.2
 # --no-binary=pycocotools: workaround for binary incompatibility on numpy 1.20
 # of pycocotools and tensorflow 2.4.1
 # when pycocotools is installed by wheel in python 3.8+
-datumaro==0.1.7 --no-binary=datumaro --no-binary=pycocotools
+datumaro==0.1.8 --no-binary=datumaro --no-binary=pycocotools