56 Commits (411a3df57ae2ce7021bdb5c07eaee142b2ff7dff)

Author SHA1 Message Date
zhiltsov-max 887c6f0432
Move annotation formats to dataset manager (#1256)
* Move formats to dataset manager

* Unify datataset export and anno export implementations

* Add track_id to TrackedShape, export tracked shapes

* Replace MOT format

* Replace LabelMe format

* Add new formats to dm

* Add dm tests

* Extend TrackedShape

* Enable dm test in CI

* Fix tests

* Add import

* Fix tests

* Fix mot track ids

* Fix mot format

* Update attribute logic in labelme tests

* Use common code in yolo

* Put datumaro in path in settings

* Expect labels file in MOT next to annotations file

* Add MOT format description

* Add import

* Add labelme format description

* Linter fix

* Linter fix2

* Compare attributes ordered

* Update docs

* Update tests
6 years ago
Andrey Zhavoronkov e7808cfb03
Data streaming using chunks (#1007)
Huge feature (200+ commits from different developers). It completely changes layout of data (please expect very long DB migration process if you have a lot of tasks). The primary idea is to send data as zip chunks (e.g. 36 images in one chunk) or encoded video chunks and decode them on the client side. It helps to solve the problem with latency  when you try to view a separate frame in the UI quickly (play mode).
Another important feature of the patch is to provide access to the original images. Thus for annotations the client uses compressed chunks but if you want to export a dataset Datumaro will use original chunks (but video will be decoded with original quality and encoded with maximum/optimal quality in any case).
6 years ago
zhiltsov-max 6a4ccdea1f
[Datumaro] Extract common extractor functionality (#1319)
* Extract common extractor functionality

* Simplify coco extractor

* Fix tfrecord
6 years ago
zhiltsov-max 03817f050b
[Datumaro] Fix image merging (#1301)
* Always merge images for own dataset

* Fix codacy
6 years ago
zhiltsov-max 4f20f46cf4
Fix attributes with spaces in names (#1305) 6 years ago
zhiltsov-max fe862b4abc
[Datumaro] Update LabelMe format (#1296)
* Little refactoring

* Add LabelMe format

* Add usernames

* Update tests

* Add extractor test
6 years ago
zhiltsov-max c91e8957df
[Datumaro] LabelMe format (#1293)
* Little refactoring

* Add LabelMe format
6 years ago
zhiltsov-max 0744c6ae16
[Datumaro] MOT format (#1289)
* Add mot format base

* Add mot format

* Extract common code
6 years ago
zhiltsov-max 8efaf58f11
Add recursive importers (#1290) 6 years ago
zhiltsov-max 9545132097
[Datumaro] Allow empty COCO dataset export (#1272)
* Allow empty dataset export in coco

* Add line to changelog

Co-authored-by: Nikita Manovich <40690625+nmanovic@users.noreply.github.com>
6 years ago
zhiltsov-max 936d3059cf
[Datumaro] Fix frame matching in video annotations import (#1274)
* Add extra frame matching way for videos

* Add line to changelog
6 years ago
Nikita Manovich ec2fa6ee51
Fixed security issues in Datumaro (#1244)
* Fixed security issues reported by bandit.
* Fixed voc_format extractor
* Sorted requirements, added a comment, removed nosec for exec.
6 years ago
zhiltsov-max 2ebca5bb5a
[Datumaro] Dataset format auto detection (#1242)
* Add dataset format detection

* Add auto format detection for import

* Split VOC extractor
6 years ago
Nikita Manovich 24130cda41
[Datumaro] Optimize mask operations (#1232)
* Optimize mask to rle

* Optimize mask operations

* Fix dm format cmdline

* Use RLE masks in datumaro format
6 years ago
Nikita Manovich be5577d013
[Datumaro] Label remapping transform (#1233)
* Add label remapping transform

* Apply transforms before project saving

* Refactor voc converter
6 years ago
zhiltsov-max b3f7f5b8bc
Add tags support for VOC (#1201)
* Extend voc format test with tags

* Add import and export of voc labels

* Fix voc and yolo format version numbers
6 years ago
zhiltsov-max 0d873a3de6
Fix remainder logic for subset splitting (#1222) 6 years ago
zhiltsov-max a058765d29
Add tags to cvat xml (#1200)
* Extend cvat format test

* Add tags to cvat for images

* Add tags to cvat format in dm

* Add import of tags from datumaro
6 years ago
zhiltsov-max da69a40b96
Add random split transform (#1213) 6 years ago
zhiltsov-max 3604f0c5ce
Auto-generate labelmap for voc from task (#1214) 6 years ago
zhiltsov-max 79c30696dd
Move project dir to .datumaro (#1207) 6 years ago
zhiltsov-max 54c277a0c9
Replace COCO implementation (#1195) 6 years ago
zhiltsov-max f2dfcc039b
[Datumaro] Remote project export fixes (#1193)
* Export project with trask name

* Do not expose server paths

* Fix tfrecord mask reading in tf>1.14

* Setuptools compatibility
6 years ago
zhiltsov-max cd8ef2aca4
Replace VOC format support in CVAT with Datumaro (#1167)
* Add image meta reading to voc

* Replace voc support in cvat

* Bump format version

* Materialize lazy transforms in voc export

* Store voc instance id as group id

* Add flat format import

* Add documentation

* Fix format name in doc
6 years ago
zhiltsov-max 8caa1695c5
Replace mask format support with Datumaro (#1163)
* Add box to mask transform

* Fix 'source' labelmap mode in voc converter

* Import groups

* Replace mask format support

* Update mask format documentation

* codacy

* Fix tests

* Fix dataset

* Fix segments grouping

* Merge instances in mask export
6 years ago
zhiltsov-max 80d3f97583
Replace tfrecord format support in CVAT with Datumaro (#1157) 6 years ago
zhiltsov-max b36f402f9c
[Datumaro] Add masks to tfrecord format (#1156)
* Employ transforms and item wrapper

* Add image class and tests

* Add image info support to formats

* Fix cli

* Fix merge and voc converte

* Update remote images extractor

* Codacy

* Remove item name, require path in Image

* Merge images of dataset items

* Update tests

* Add image dir converter

* Update Datumaro format

* Update COCO format with image info

* Update CVAT format with image info

* Update TFrecord format with image info

* Update VOC formar with image info

* Update YOLO format with image info

* Update dataset manager bindings with image info

* Add image name to id transform

* Fix coco export

* Add masks support for tfrecord

* Refactor coco

* Fix comparison

* Remove dead code

* Extract common code for instances
6 years ago
zhiltsov-max a376ee76fd
[Datumaro] Introduce image info (#1140)
* Employ transforms and item wrapper

* Add image class and tests

* Add image info support to formats

* Fix cli

* Fix merge and voc converte

* Update remote images extractor

* Codacy

* Remove item name, require path in Image

* Merge images of dataset items

* Update tests

* Add image dir converter

* Update Datumaro format

* Update COCO format with image info

* Update CVAT format with image info

* Update TFrecord format with image info

* Update VOC formar with image info

* Update YOLO format with image info

* Update dataset manager bindings with image info

* Add image name to id transform

* Fix coco export
6 years ago
zhiltsov-max 6c2d93ec3e
[Datumaro] Pretty output folder names (#1149)
* Generate output dir name from operation parameters

* Fix failing command
6 years ago
zhiltsov-max a2da181f01
Avoid tf deprecation warning (#1148) 6 years ago
zhiltsov-max 15dc61dd38
Add offline subset remapping and bbox conversion (#1147) 6 years ago
zhiltsov-max c523420c57
Bump gitpython version (#1146) 6 years ago
zhiltsov-max 985fdd0d70
[Datumaro] fixes (#1137)
* Fix import command

* Fix project name for spawned projects

* Fix voc and coco converter parameters

* Fix voc colormap color interpretation

* Change order of image search for cvat extractor

* fix CVAT image search paths
6 years ago
zhiltsov-max f800e6a5d3
Add plugin system documentation (#1131) 6 years ago
zhiltsov-max 29d65b0cda
Fix tensorflow installation (#1129)
* Make tf dependency optional

* Reduce opencv dependency

* Import tf eagerly as it is a plugin

* Do not install TF with Datumaro
6 years ago
zhiltsov-max 2848f1d14f
[Datumaro] Plugins and transforms (#1126)
* Fix model run command

* Rename annotation types, update class interfaces

* Fix random cvat format test fails

* Mask operations and dataset format fixes

* Update tests, extract format testing functions

* Add transform interface

* Implement plugin system

* Update tests with plugins

* Fix logging

* Add transfroms

* Update cvat integration
6 years ago
zhiltsov-max 04c7669cf5 Optimize mask conversions (#1097) 6 years ago
zhiltsov-max 93b3c091f5 [Datumaro] CLI updates + better documentation (#1057) 6 years ago
Nikita Manovich a5c3da1235
Cleanup Dockerfiles for CVAT (#1060)
* Replaced wget by curl

* Moved CI stuff into Dockerfile.ci

* Use docker-compose to run commnands inside docker (need environment variables)

* Added patool again (to support different archive formats)

* Roll back tensorflow version: 1.15 -> 1.13.1

Fixed https://github.com/opencv/cvat/issues/982
Fixed https://github.com/opencv/cvat/issues/1017

* datumaro install tensorflow 2.x now. It breaks automatic annotation
using TF.

* Follow redirects in curl (auto_segmentation)
6 years ago
zhiltsov-max e5b4c194bb [Datumaro] Dataset annotations filter (#1053)
* Fix deprecation message

* Update launcher interface

* Add dataset entity, anno filter, remove filter from project config, update transform

* Update project and source cli

* Fix help message

* Refactor tests
6 years ago
zhiltsov-max e0bcc4652b [Datumaro] Instance polygon-mask conversions in COCO format (#1008)
* Microoptimizations

* Mask conversion functions

* Add mask-polygon conversions

* Add mask-polygon conversions in coco

* Add mask-polygon conversions in coco

* Update requirements

* Option to disable crop

* Fix cli parameter passing

* Fix test

* Fixes in COCO
6 years ago
zhiltsov-max 8da20b38d5 [Datumaro] Add cvat format export (#1034)
* Add cvat format export

* Remove wrong items in test
6 years ago
zhiltsov-max 337f7a0018 [Datumaro] Fix project loading (#1013)
* Fix occasional infinite loop in project loading

* Fix project import source options saving

* Fix project import .git dir placement

* Make code aware of grayscale images
6 years ago
zhiltsov-max 7c9043974b [Datumaro] Fix TFrecord converter constructor (#993) 6 years ago
zhiltsov-max 36b1e9c1ce [Datumaro] CVAT format import (#974)
* Add label-specific attributes
* Add CVAT format import
* Register CVAT format
* Add little more logs
* Little refactoring for tests
* Cvat format checks
* Add missing check
* Refactor datumaro format
* Little refactoring
* Regularize dataset importer logic
* Fix project import issue
* Refactor coco extractor
* Refactor tests
* Codacy
6 years ago
zhiltsov-max e9f1db00db Add polygon point count checks (#975) 6 years ago
zhiltsov-max c84daaf2ef [Datumaro] VOC labelmap support (#957)
* Add import result checks and options to skip
* Add label-specific attributes
* Overwrite option for export
* Add labelmap file support in voc
* Add labelmap tests
* Little refactoring
6 years ago
zhiltsov-max 32254530cf [Datumaro] Fix voc colormap (#945)
* Add polygon merging option to coco converter

* Add test, refactor coco, add support for cli args

* Drop colormap application in datumaro format

* Add cli support in voc converter

* Add cli support in yolo converter

* Add converter cli options in project cli

* Add image data type conversion in image saving

* Add image data type conversion in image saving

* Update mask support in voc

* Replace null with quotes in coco export

* Improve cli

* Enable Datumaro intellisense in vs cde

* Adjust fields in voc detection export
6 years ago
zhiltsov-max 944d85370d [Datumaro] COCO 'merge instance polygons' option (#938)
* Add polygon merging option to coco converter
* Add test, refactor coco, add support for cli args
* Drop colormap application in datumaro format
* Add cli support in voc converter
* Add cli support in yolo converter
* Add converter cli options in project cli
* Add image data type conversion in image saving
6 years ago
zhiltsov-max 0f5cf2064d [Datumaro] Added tf detection api tfrecord import and export (#894)
* Added tf detection api tfrecord import and export
* Added export button in dashboard
* Add tf to requirements
* Extend test
* Add tf dependency
* Require images in tfrecord export
* Add video task case handling
* Maintain image order in CVAT export
* Fix Task image id-path conversions
* Update tfrecord tests
* Extend image utilities
* Update tfrecord format
* Fix image loading bug
* Add some logs
* Add 'copy' option to project import command
* Reduce default cache size
6 years ago