102 Commits (18d04ff0f4f85c425cf76e0ca6f028da040136a3)

Author SHA1 Message Date
zhiltsov-max 0062ecdec3
[Datumaro] Fixes (#1953)
* Add absolute image path in rest api extractor

* Add default split for random split

* Fix image path in datumaro format

* Preserve bboxes in coco format

* update changelog

Co-authored-by: Nikita Manovich <40690625+nmanovic@users.noreply.github.com>
6 years ago
Nikita Manovich e7585b8ce9
DL models as serverless functions (#1767)
* Initial experiments with nuclio

* Update nuclio prototype

* Improve nuclio prototype for dextr.

* Dummy lambda manager

* OpenFaaS prototype (dextr.bin and dextr.xml are empty).

* Moved openfaas prototype.

* Add comments

* Add serializers and HLD for lambda_manager

* Initial version of Mask RCNN (without debugging)

* Initial version for faster_rcnn_inception_v2_coco

* Fix faster_rcnn_inception_v2_coco

* Implemented mask_rcnn_inception_resnet_v2_atrous_coco

* Implemented yolo detector as a lambda function

* Removed dextr app.

* Added types for each function (detector and interactor)

* Initial version of lambda_manager.

* Implement a couple of methods for lambda:

GET /api/v1/lambda/functions
GET /api/v1/lambda/functions/public.dextr

* First working version of dextr serverless function

* First version of dextr which works in UI.

* Modify omz.public.faster_rcnn_inception_v2_coco

- image decoding
- restart policy always for the function

* Improve omz.public.mask_rcnn_inception_resnet_v2_atrous_coco

* Improve omz.public.yolo-v3-tf function

* Implemented the initial version of requests for lambda manager.

* First working version of POST /api/v1/lambda/requests

* Updated specification of function.yaml (added labels and used annotations section).

* Added health check for containers (nuclio dashboard feature)

* Read labels spec from function.yaml.

* Added settings for NUCLIO

* Fixed a couple of typos. Now it works in most cases.

* Remove Plugin REST API

* Remove tf_annotation app (it will be replaced by serverless function)

* Remove tf_annotation and cuda components

* Cleanup docs and Dockerfile from CUDA component.

* Just renamed directories inside serverless

* Remove redundant files and code

* Remove redundant files.

* Remove outdated files

* Remove outdated code

* Delete reid app and add draft of serverless function for reid.

* Model list in UI.

* Fixed the framework name (got it from lambda function).

* Add maxRequestBodySize for functions, remove redundant code from UI for auto_annotation.

* Update view of models page.

* Unblock mapping for "primary" models.

* Implement cleanup flag for lambda/requests and labeling mapping for functions.

* Implement protection from running multiple jobs for the same task.

* Fix invocation of functions in docker container.

* Fix Dockerfile.ci

* Remove unused files from lambda_manager

* Fix codacy warnings

* Fix codacy issues.

* Fix codacy warnings

* Implement progress and cancel (aka delete) operation.

* Send annotations in batch.

* Fix UI. Now it can retrieve information about inference requests in progress.

* Update CHANGELOG.md

* Update cvat-ui version.

* Update nuclio version.

* Implement serverless/tensorflow/faster_rcnn_inception_v2_coco

* Add information how to install nuclio platform and run serverless functions.

* Add installation instructions for serverless functions.

* Update OpenVINO files which are responsible for loading network

* relocated functions

* Update dextr function.

* Update faster_rcnn function from omz

* Fix OpenVINO Mask-RCNN

* Fix YOLO v3 serverless function.

* Dummy serverless functions for a couple of more OpenVINO models.

* Protected lambda manager views by correct permissions.

* Fix name of Faster RCNN from Tensorflow.

* Implement Mask RCNN via Tensorflow serverless function.

* Minor client changes (#1847)

* Minor client changes

* Removed extra code

* Add reid serverless function (no support in lambda manager).

* Fix contribution guide.

* Fix person-reidentification-retail-300 and implement text-detection-0004

* Add semantic-segmentation-adas-0001

* Moving model management to cvat-core (#1905)

* Squached changes

* Removed extra line

* Remove duplicated files for OpenVINO serverless functions.

* Updated CHANGELOG.md

* Remove outdated code.

* Running dextr via lambda manager (#1912)

* Deleted outdated migration.

* Add name for DEXTR function.

* Fix restart policy for serverless functions.

* Fix openvino serverless functions for images with alpha channel

* Add more tensorflow serverless functions into deploy.sh

* Use ID instead of name for DEXTR (#1926)

* Update DEXTR function

* Added source "auto" inside lambda manager for automatic annotation.

* Customize payload (depends on type of lambda function).

* First working version of REID (Server only).

* Fix codacy warnings

* Avoid exception during migration (workaround)

File "/usr/local/lib/python3.5/dist-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: table "engine_pluginoption" does not exist

* Add siammask serverless function (it doesn't work, need to serialize state)

* Run ReID from UI (#1949)

* Removed reid route in installation.md

* Fix a command to get lena image in CONTRIBUTION guide.

* Fix typo and crash in case a polygon is a line.

Co-authored-by: Boris Sekachev <40690378+bsekachev@users.noreply.github.com>
6 years ago
zhiltsov-max 3e12bbb43e
Fix convert command (#1943) 6 years ago
zhiltsov-max a3448a20ad
Add image sha256 to tfrecord (#1931) 6 years ago
zhiltsov-max fa4eea70d1
Add type conversion (#1932) 6 years ago
Ilya Kochankov 7ecdcf182b
[Datumaro] Reducing nesting of tests (#1875)
* Add `Dataset.from_iterable` constructor
* Simplify creation of `Dataset` objects in common simple cases
* Refactor tests
6 years ago
zhiltsov-max f807714870
[Datumaro] Image control in converters (#1799)
* Update converter interface, add image control

* Update bindings

* update changelog

* Fix build
6 years ago
Zhiltsov Max 082ff36741 Fix build 6 years ago
zhiltsov-max d7c9a0fd21
Refactorings (#1876) 6 years ago
zhiltsov-max c6b3c797ab
[Datumaro] CLI updates (#1824)
* updates

* diff fix
6 years ago
zhiltsov-max 575c93ff2c
[Datumaro] Convert command (#1837)
* Add convert command

* fixes

* update readme

* update changelog

Co-authored-by: Andrey Zhavoronkov <41117609+azhavoro@users.noreply.github.com>
6 years ago
zhiltsov-max 78f1893ab2
Allow saving empty xmls in voc (#1883) 6 years ago
zhiltsov-max fc2fb6156a
Don't export outside annotations (#1729)
* Add option to omit outside annotations

* update changelog

* Fix mot format and test

* Fix outside in mot

* fix repo problem

* t

* Update CHANGELOG.md

Co-authored-by: Nikita Manovich <40690625+nmanovic@users.noreply.github.com>
6 years ago
zhiltsov-max 80ed45b452
Allow attributes in COCO format (#1782)
* allow attributes in coco

* update changelog

* update cli

* update coco cli
6 years ago
zhiltsov-max 962f61fa27
Add attributes in VOC format (#1792)
* Add voc attributes

* Allow any values for voc pose

* update changelog

* Add attribute conversion

* linter

* fix tests
6 years ago
zhiltsov-max ae3b06b465
[Datumaro] Add generic accuracy checker model launcher (#1661)
* Refactor inference wrapper

* Add accuracy checker launcher wrapper

* t

* rename method

* Add importer for openvino launcher

* Move openvino plugin to iecore

* add generic AC launcher

* Implement cli for AC launcher

* move ac plugin dir

* prevent tf reimport

* Fix outputs conversion

* t

* add pytorch model example

* Require config path in launcher

* Clear extra whitespace
6 years ago
zhiltsov-max 12f78559d2
[Datumaro] Mean and std for dataset (#1734)
* Add meanstd

* Add stats cli

* Update changelog

Co-authored-by: Nikita Manovich <40690625+nmanovic@users.noreply.github.com>
6 years ago
zhiltsov-max be30aa6354
[Datumaro] Refactor explain and models CLI (#1714)
* Update explain cli

* Update model cli

* Update config for models

* Remove input size hint for models
6 years ago
zhiltsov-max 5912bf0447
[Datumaro] Support relative paths (#1715)
* Support relative image paths in Datumaro

* Update bindings

* Fix merge

* linter
6 years ago
zhiltsov-max 47470b8470
Allow any annotations file name in coco extractor (#1650) 6 years ago
zhiltsov-max 1385dc46b8
Prevent reimports for TF (#1713) 6 years ago
zhiltsov-max 4c74a31cb8
Add color map conversion example in docs (#1670)
* add color map conversion example in docs
6 years ago
zhiltsov-max 39d3c93cfd
[Datumaro] Update model interface and OpenVino launcher (#1626)
* Refactor inference wrapper
6 years ago
zhiltsov-max e0ef2cfa70
Update coco tests (#1669) 6 years ago
zhiltsov-max 0053411f6e
[Datumaro] Add debug output in VOC import (#1662)
* Add debug output in VOC import
6 years ago
zhiltsov-max 696a47b3c5
Add assets importing tests for cvat format (#1652) 6 years ago
zhiltsov-max ba309c882f
[Datumaro] Add DatasetItem attributes (#1639)
* Add DatasetItem attributes

* Update tests

* Update datumaro format
6 years ago
zhiltsov-max 1b413c65a2
Fix image size conversion (#1593)
* Fix image size conversion

* update changelog

Co-authored-by: Nikita Manovich <40690625+nmanovic@users.noreply.github.com>
6 years ago
zhiltsov-max 727fcd50b3
[Datumaro] Remove annotation dir for cvat dataset format (#1637)
* Remove annotation dir for cvat dataset format

* fixes
6 years ago
zhiltsov-max 2044d35566
Add directory check on image save (#1636)
* Add directory check on image save

* Add tests

* change exception type

* update test
6 years ago
zhiltsov-max 306e91c3d7
[Datumaro] Add tests for dataset examples (#1648)
* add dataset examples

* update docs

* update yolo tests

* join voc format test classes

* remplace voc extractor tests with import test

* update tfrecord format tests

* update mot tests

* update labelme tests

* update image dir tests
6 years ago
zhiltsov-max d1b2960b3d
[Datumaro] Add dataset examples (#1640)
* add dataset examples
6 years ago
zhiltsov-max da3fa34ce0
Add tf import checks and tests (#1567)
* Add tf import checks and tests

* implement disabled formats on server

* python 3.5 compatibility

* add checks to dm tests

* fix tests

* Support for disabled formats in UI

* add sorting for formats, mark grey disabled items

* update changelog

* advance package versions
6 years ago
zhiltsov-max ac922e33a5
Add default colors for masks (#1549)
* Use default colors for masks

* do not change labels in voc converter

* update tests

* add backgound color handling

* consider case with existing source colormap

* update changelog

* fix file name

* fix insert

* move color table colors to beginning
6 years ago
zhiltsov-max e2dedb2f83
Use z_order as a class property (#1589)
* Use z_order as a class property

* Fix z_order use in voc

* Update changelog
6 years ago
zhiltsov-max 1200ecf67c
[Datumaro] Simplify log level setting (#1583)
* Simplify loglevel setting

* update changelog
6 years ago
zhiltsov-max 42c5637ddd
[Datumaro] Fix mask to polygons warning (#1581)
* Fix message, add test
* update changelog
6 years ago
zhiltsov-max 4c603504b7
Add VOC grayscale masks test and documentation (#1576)
* Add a test for unpainted masks
* Update format documentation
6 years ago
zhiltsov-max fb380d9855
Include empty images in exported annotations (#1479) 6 years ago
zhiltsov-max 98a9718e63
[Datumaro] Change alignment in mask parsing (#1547) 6 years ago
zhiltsov-max 42990906b9
[Datumaro] Fix coco import conflict with labels (#1548) 6 years ago
zhiltsov-max 5ab549956f
Merge annotations and dataset_manager apps (#1352) 6 years ago
zhiltsov-max c78db45d9a
[Datumaro] Fix duplicating keypoints in COCO export (#1435) 6 years ago
zhiltsov-max 819b81d7f3
Fix label comparison in voc format (#1382) 6 years ago
zhiltsov-max bd659246e9
[Datumaro] Fix COCO keypoint export bug (#1388) 6 years ago
zhiltsov-max afeab69aa3
Save full image paths in coco (#1381) 6 years ago
zhiltsov-max 887c6f0432
Move annotation formats to dataset manager (#1256)
* Move formats to dataset manager

* Unify datataset export and anno export implementations

* Add track_id to TrackedShape, export tracked shapes

* Replace MOT format

* Replace LabelMe format

* Add new formats to dm

* Add dm tests

* Extend TrackedShape

* Enable dm test in CI

* Fix tests

* Add import

* Fix tests

* Fix mot track ids

* Fix mot format

* Update attribute logic in labelme tests

* Use common code in yolo

* Put datumaro in path in settings

* Expect labels file in MOT next to annotations file

* Add MOT format description

* Add import

* Add labelme format description

* Linter fix

* Linter fix2

* Compare attributes ordered

* Update docs

* Update tests
6 years ago
Andrey Zhavoronkov e7808cfb03
Data streaming using chunks (#1007)
Huge feature (200+ commits from different developers). It completely changes layout of data (please expect very long DB migration process if you have a lot of tasks). The primary idea is to send data as zip chunks (e.g. 36 images in one chunk) or encoded video chunks and decode them on the client side. It helps to solve the problem with latency  when you try to view a separate frame in the UI quickly (play mode).
Another important feature of the patch is to provide access to the original images. Thus for annotations the client uses compressed chunks but if you want to export a dataset Datumaro will use original chunks (but video will be decoded with original quality and encoded with maximum/optimal quality in any case).
6 years ago
zhiltsov-max 6a4ccdea1f
[Datumaro] Extract common extractor functionality (#1319)
* Extract common extractor functionality

* Simplify coco extractor

* Fix tfrecord
6 years ago
zhiltsov-max 03817f050b
[Datumaro] Fix image merging (#1301)
* Always merge images for own dataset

* Fix codacy
6 years ago