CVAT-3D Milestone2 (#2645)

* CVAT-3D Updated the Mime Types with Bin Support, added dependency of open3D

* CVAT-3D Added additional column as Dimension for engine_task table and created a relatedfiles table for PCD to Image mapping.

* Added Support for 3D file Upload in BIN and PCD.

* Added Dimension attribute defaulting to 2D for importer and exporter.

* Added props passing for dimension attribute, filtering of import, Migration Scripts and Dimension attribute for MpegChunk Writers

* Modified code as per review comments

* Updated Unit test cases for 3D task creation

* Refactored Dimension Enum in UI and backend code

* Resolving conflicts

* Updated Unit Test Case

* Refactored TaskDimension to DimensionType, Simplified usage of Dimension accross classes

* Removing manually created test files

* Removing old pcd mime-type mapping

* Added test files generated by synthetic data using open3d

* Merged with develop branch latest changes

* Added libraries required for open3d

* Added files

* Added synthethic pcd,bin and img test files

* Modified test file name

* Trigger travis ci

* Modified test case to ignore 3D preview images

* Trigger notification

* Deleting DS Store files

* Modified test cases as per review comments

* Checking pre-commit hook

* Fixed Lint issues - precommit hook verification

* Added changes for CVAT-3D Milestone2 changes - Frame Navigation, photo context hide and show

* Modified changes

* Added canvas3D for 3D Perspective

* Added missing files

* Added code to get image context for 3D view

* Codacy check for stylesheet

* Modified frame navigantion for 3D View

* Modified style for context-image

* Trigger notification

* Added Support for 3D file Upload in BIN and PCD.

* Added props passing for dimension attribute, filtering of import, Migration Scripts and Dimension attribute for MpegChunk Writers

* Modified code as per review comments

* Refactored Dimension Enum in UI and backend code

* Merged with develop branch latest changes

* Added files

* Added changes for CVAT-3D Milestone2 changes - Frame Navigation, photo context hide and show

* Modified changes

* Added canvas3D for 3D Perspective

* Added missing files

* Added code to get image context for 3D view

* Codacy check for stylesheet

* Modified frame navigantion for 3D View

* Modified style for context-image

* Changed cvat-data lint issues

* Modified to use opencv as per review comments

* Removed unwanted imports

* Fixed css and added usage of hooks

* Merged Develop branch code

* Removed unused data structures

* Removed unused data structures

* Refactored unused data structures

* Added three js dependency in cvat ui package-lock.json

* Merged develop branch code and refactored code

* Fixed snyk issue

* Modified Camera Icon in photo-context

* Update icons.tsx

* Remove unused svg file

* Modified changelog file

Co-authored-by: cdp <cdp123>
main
manasars 5 years ago committed by GitHub
parent 71e2ddbb59
commit 5c67846c20
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### Added
- CVAT-3D: support lidar data on the server side (<https://github.com/openvinotoolkit/cvat/pull/2534>) - CVAT-3D: support lidar data on the server side (<https://github.com/openvinotoolkit/cvat/pull/2534>)
- CVAT-3D: Load all frames corresponding to the job instance
(<https://github.com/openvinotoolkit/cvat/pull/2645>)
- Intelligent scissors with OpenCV javascript (<https://github.com/openvinotoolkit/cvat/pull/2689>) - Intelligent scissors with OpenCV javascript (<https://github.com/openvinotoolkit/cvat/pull/2689>)
### Changed ### Changed

@ -20,6 +20,7 @@ RUN apk add python3 g++ make
# Install dependencies # Install dependencies
COPY cvat-core/package*.json /tmp/cvat-core/ COPY cvat-core/package*.json /tmp/cvat-core/
COPY cvat-canvas/package*.json /tmp/cvat-canvas/ COPY cvat-canvas/package*.json /tmp/cvat-canvas/
COPY cvat-canvas3d/package*.json /tmp/cvat-canvas3d/
COPY cvat-ui/package*.json /tmp/cvat-ui/ COPY cvat-ui/package*.json /tmp/cvat-ui/
COPY cvat-data/package*.json /tmp/cvat-data/ COPY cvat-data/package*.json /tmp/cvat-data/
@ -35,6 +36,10 @@ RUN npm ci
WORKDIR /tmp/cvat-canvas/ WORKDIR /tmp/cvat-canvas/
RUN npm ci RUN npm ci
# Install cvat-canvas dependencies
WORKDIR /tmp/cvat-canvas3d/
RUN npm ci
# Install cvat-ui dependencies # Install cvat-ui dependencies
WORKDIR /tmp/cvat-ui/ WORKDIR /tmp/cvat-ui/
RUN npm ci RUN npm ci
@ -42,6 +47,7 @@ RUN npm ci
# Build source code # Build source code
COPY cvat-data/ /tmp/cvat-data/ COPY cvat-data/ /tmp/cvat-data/
COPY cvat-core/ /tmp/cvat-core/ COPY cvat-core/ /tmp/cvat-core/
COPY cvat-canvas3d/ /tmp/cvat-canvas3d/
COPY cvat-canvas/ /tmp/cvat-canvas/ COPY cvat-canvas/ /tmp/cvat-canvas/
COPY cvat-ui/ /tmp/cvat-ui/ COPY cvat-ui/ /tmp/cvat-ui/
RUN npm run build RUN npm run build

@ -0,0 +1 @@
webpack.config.js

@ -0,0 +1,46 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
module.exports = {
env: {
node: true,
},
parserOptions: {
parser: '@typescript-eslint/parser',
ecmaVersion: 6,
},
plugins: ['@typescript-eslint', 'import'],
extends: [
'plugin:@typescript-eslint/recommended',
'airbnb-typescript/base',
'plugin:import/errors',
'plugin:import/warnings',
'plugin:import/typescript',
],
rules: {
'@typescript-eslint/no-explicit-any': 0,
'@typescript-eslint/indent': ['warn', 4],
'no-plusplus': 0,
'no-restricted-syntax': [
0,
{
selector: 'ForOfStatement',
},
],
'max-len': ['error', { code: 120 }],
'no-continue': 0,
'func-names': 0,
'no-console': 0, // this rule deprecates console.log, console.warn etc. because 'it is not good in production code'
'lines-between-class-members': 0,
'import/prefer-default-export': 0, // works incorrect with interfaces
'newline-per-chained-call': 0, // makes code uglier
},
settings: {
'import/resolver': {
node: {
extensions: ['.ts', '.js', '.json'],
},
},
},
};

@ -0,0 +1 @@
dist

@ -0,0 +1,50 @@
# Module CVAT-CANVAS-3D
## Description
The CVAT module written in TypeScript language.
It presents a canvas to viewing, drawing and editing of 3D annotations.
## Versioning
If you make changes in this package, please do following:
- After not important changes (typos, backward compatible bug fixes, refactoring) do: `npm version patch`
- After changing API (backward compatible new features) do: `npm version minor`
- After changing API (changes that break backward compatibility) do: `npm version major`
## Commands
- Building of the module from sources in the `dist` directory:
```bash
npm run build
npm run build -- --mode=development # without a minification
```
### API Methods
```ts
interface Canvas3d {
html(): HTMLDivElement;
setup(frameData: any): void;
fitCanvas(): void;
mode(): Mode;
isAbleToChangeFrame(): boolean;
render(): void;
}
```
### WEB
```js
// Create an instance of a canvas
const canvas = new window.canvas.Canvas3d();
console.log('Version ', window.canvas.CanvasVersion);
console.log('Current mode is ', window.canvas.mode());
// Put canvas to a html container
htmlContainer.appendChild(canvas.html());
canvas.fitCanvas();
```

File diff suppressed because it is too large Load Diff

@ -0,0 +1,43 @@
{
"name": "cvat-canvas3d",
"version": "0.0.1",
"description": "Part of Computer Vision Annotation Tool which presents its canvas3D library",
"main": "src/canvas3d.ts",
"scripts": {
"build": "tsc && webpack --config ./webpack.config.js",
"server": "nodemon --watch config --exec 'webpack-dev-server --config ./webpack.config.js --mode=development --open'"
},
"author": "Intel",
"license": "MIT",
"devDependencies": {
"@babel/cli": "^7.5.5",
"@babel/core": "^7.5.5",
"@babel/plugin-proposal-class-properties": "^7.8.3",
"@babel/plugin-proposal-optional-chaining": "^7.11.0",
"@babel/preset-env": "^7.5.5",
"@babel/preset-typescript": "^7.3.3",
"@types/node": "^12.6.8",
"@typescript-eslint/eslint-plugin": "^1.13.0",
"@typescript-eslint/parser": "^1.13.0",
"babel-loader": "^8.0.6",
"css-loader": "^3.4.2",
"dts-bundle-webpack": "^1.0.2",
"eslint": "^6.1.0",
"eslint-config-airbnb-typescript": "^4.0.1",
"eslint-config-typescript-recommended": "^1.4.17",
"eslint-plugin-import": "^2.18.2",
"node-sass": "^4.14.1",
"nodemon": "^1.19.4",
"postcss-loader": "^3.0.0",
"postcss-preset-env": "^6.7.0",
"sass-loader": "^8.0.2",
"style-loader": "^1.0.0",
"typescript": "^3.5.3",
"webpack": "^4.44.2",
"webpack-cli": "^3.3.6",
"webpack-dev-server": "^3.11.0"
},
"dependencies": {
"three": "^0.125.0"
}
}

@ -0,0 +1,12 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
module.exports = {
parser: false,
plugins: {
'postcss-preset-env': {
browsers: '> 2.5%', // https://github.com/browserslist/browserslist
},
},
};

@ -0,0 +1,58 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import pjson from '../../package.json';
import { Canvas3dController, Canvas3dControllerImpl } from './canvas3dController';
import { Canvas3dModel, Canvas3dModelImpl, Mode } from './canvas3dModel';
import { Canvas3dView, Canvas3dViewImpl } from './canvas3dView';
import { Master } from './master';
const Canvas3dVersion = pjson.version;
interface Canvas3d {
html(): any;
setup(frameData: any): void;
isAbleToChangeFrame(): boolean;
fitCanvas(): void;
mode(): Mode;
render(): void;
}
class Canvas3dImpl implements Canvas3d {
private model: Canvas3dModel & Master;
private controller: Canvas3dController;
private view: Canvas3dView;
public constructor() {
this.model = new Canvas3dModelImpl();
this.controller = new Canvas3dControllerImpl(this.model);
this.view = new Canvas3dViewImpl(this.model, this.controller);
}
public html(): any {
return this.view.html();
}
public render(): void {
this.view.render();
}
public setup(frameData: any): void {
this.model.setup(frameData);
}
public mode(): Mode {
return this.model.mode;
}
public isAbleToChangeFrame(): boolean {
return this.model.isAbleToChangeFrame();
}
public fitCanvas(): void {
this.model.fitCanvas(this.view.html().clientWidth, this.view.html().clientHeight);
}
}
export { Canvas3dImpl as Canvas3d, Canvas3dVersion };

@ -0,0 +1,25 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import { Canvas3dModel, Mode } from './canvas3dModel';
export interface Canvas3dController {
mode: Mode;
}
export class Canvas3dControllerImpl implements Canvas3dController {
private model: Canvas3dModel;
public constructor(model: Canvas3dModel) {
this.model = model;
}
public set mode(value: Mode) {
this.model.mode = value;
}
public get mode(): Mode {
return this.model.mode;
}
}

@ -0,0 +1,153 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import { MasterImpl } from './master';
export interface Size {
width: number;
height: number;
}
export interface Image {
renderWidth: number;
renderHeight: number;
imageData: ImageData | CanvasImageSource;
}
export interface DrawData {
enabled: boolean;
initialState?: any;
redraw?: number;
}
export enum FrameZoom {
MIN = 0.1,
MAX = 10,
}
export enum UpdateReasons {
IMAGE_CHANGED = 'image_changed',
OBJECTS_UPDATED = 'objects_updated',
FITTED_CANVAS = 'fitted_canvas',
DRAW = 'draw',
SELECT = 'select',
CANCEL = 'cancel',
DATA_FAILED = 'data_failed',
}
export enum Mode {
IDLE = 'idle',
DRAG = 'drag',
RESIZE = 'resize',
DRAW = 'draw',
EDIT = 'edit',
INTERACT = 'interact',
}
export interface Canvas3dModel {
mode: Mode;
setup(frameData: any): void;
isAbleToChangeFrame(): boolean;
fitCanvas(width: number, height: number): void;
}
export class Canvas3dModelImpl extends MasterImpl implements Canvas3dModel {
private data: {
canvasSize: Size;
image: Image | null;
imageID: number | null;
imageOffset: number;
imageSize: Size;
drawData: DrawData;
mode: Mode;
exception: Error | null;
};
public constructor() {
super();
this.data = {
canvasSize: {
height: 0,
width: 0,
},
image: null,
imageID: null,
imageOffset: 0,
imageSize: {
height: 0,
width: 0,
},
drawData: {
enabled: false,
initialState: null,
},
mode: Mode.IDLE,
exception: null,
};
}
public setup(frameData: any): void {
if (this.data.imageID !== frameData.number) {
if ([Mode.EDIT, Mode.DRAG, Mode.RESIZE].includes(this.data.mode)) {
throw Error(`Canvas is busy. Action: ${this.data.mode}`);
}
}
this.data.imageID = frameData.number;
frameData
.data((): void => {
this.data.image = null;
this.notify(UpdateReasons.IMAGE_CHANGED);
})
.then((data: Image): void => {
if (frameData.number !== this.data.imageID) {
// already another image
return;
}
this.data.imageSize = {
height: frameData.height as number,
width: frameData.width as number,
};
this.data.image = data;
this.notify(UpdateReasons.IMAGE_CHANGED);
})
.catch((exception: any): void => {
this.data.exception = exception;
this.notify(UpdateReasons.DATA_FAILED);
throw exception;
});
}
public set mode(value: Mode) {
this.data.mode = value;
}
public get mode(): Mode {
return this.data.mode;
}
public isAbleToChangeFrame(): boolean {
const isUnable = [Mode.DRAG, Mode.EDIT, Mode.RESIZE, Mode.INTERACT].includes(this.data.mode)
|| (this.data.mode === Mode.DRAW && typeof this.data.drawData.redraw === 'number');
return !isUnable;
}
public fitCanvas(width: number, height: number): void {
this.data.canvasSize.height = height;
this.data.canvasSize.width = width;
this.data.imageOffset = Math.floor(
Math.max(this.data.canvasSize.height / FrameZoom.MIN, this.data.canvasSize.width / FrameZoom.MIN),
);
this.notify(UpdateReasons.FITTED_CANVAS);
this.notify(UpdateReasons.OBJECTS_UPDATED);
}
}

@ -0,0 +1,86 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import * as THREE from 'three';
import { PCDLoader } from 'three/examples/jsm/loaders/PCDLoader';
import { Canvas3dController } from './canvas3dController';
import { Listener, Master } from './master';
import { Canvas3dModel, UpdateReasons, Mode } from './canvas3dModel';
export interface Canvas3dView {
html(): HTMLDivElement;
render(): void;
}
export class Canvas3dViewImpl implements Canvas3dView, Listener {
private controller: Canvas3dController;
private renderer: any;
private scene: any;
private camera: any;
private set mode(value: Mode) {
this.controller.mode = value;
}
private get mode(): Mode {
return this.controller.mode;
}
public constructor(model: Canvas3dModel & Master, controller: Canvas3dController) {
this.controller = controller;
this.mode = Mode.IDLE;
this.scene = new THREE.Scene();
this.scene.background = new THREE.Color(0x000000);
// setting up the camera and adding it in the scene
this.camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 1, 500);
this.camera.position.set(-15, 0, 4);
this.camera.up.set(0, 0, 1);
this.camera.lookAt(0, 0, 0);
this.scene.add(this.camera);
this.renderer = new THREE.WebGLRenderer({ antialias: true });
this.renderer.setPixelRatio(window.devicePixelRatio);
this.renderer.setSize(window.innerWidth, window.innerHeight);
model.subscribe(this);
}
public notify(model: Canvas3dModel & Master, reason: UpdateReasons): void {
if (reason === UpdateReasons.IMAGE_CHANGED) {
const loader = new PCDLoader();
this.clearScene();
const objectURL = URL.createObjectURL(model.data.image.imageData);
loader.load(objectURL, this.addScene.bind(this));
URL.revokeObjectURL(objectURL);
const event: CustomEvent = new CustomEvent('canvas.setup');
this.renderer.domElement.dispatchEvent(event);
}
}
private clearScene(): void {
for (let i = this.scene.children.length - 1; i >= 0; i--) {
this.scene.remove(this.scene.children[i]);
}
}
private addScene(points: any): void {
// eslint-disable-next-line no-param-reassign
points.material.size = 0.03;
// eslint-disable-next-line no-param-reassign
points.material.color = new THREE.Color(0x0000ff);
this.scene.add(points);
}
public render(): void {
this.renderer.render(this.scene, this.camera);
}
public html(): any {
return this.renderer.domElement;
}
}

@ -0,0 +1,9 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
const BASE_GRID_WIDTH = 2;
export default {
BASE_GRID_WIDTH,
};

@ -0,0 +1,44 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
export interface Master {
subscribe(listener: Listener): void;
unsubscribe(listener: Listener): void;
unsubscribeAll(): void;
notify(reason: string): void;
}
export interface Listener {
notify(master: Master, reason: string): void;
}
export class MasterImpl implements Master {
private listeners: Listener[];
public constructor() {
this.listeners = [];
}
public subscribe(listener: Listener): void {
this.listeners.push(listener);
}
public unsubscribe(listener: Listener): void {
for (let i = 0; i < this.listeners.length; i++) {
if (this.listeners[i] === listener) {
this.listeners.splice(i, 1);
}
}
}
public unsubscribeAll(): void {
this.listeners = [];
}
public notify(reason: string): void {
for (const listener of this.listeners) {
listener.notify(this, reason);
}
}
}

@ -0,0 +1,19 @@
{
"compilerOptions": {
"baseUrl": ".",
"emitDeclarationOnly": true,
"module": "es6",
"target": "es6",
"noImplicitAny": true,
"preserveConstEnums": true,
"declaration": true,
"resolveJsonModule": true,
"esModuleInterop": true,
"moduleResolution": "node",
"declarationDir": "dist/declaration",
"paths": {
"cvat-canvas.node": ["dist/cvat-canvas3d.node"]
}
},
"include": ["src/typescript/*.ts"]
}

@ -0,0 +1,138 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
// eslint-disable-next-line @typescript-eslint/no-var-requires
const path = require('path');
// eslint-disable-next-line @typescript-eslint/no-var-requires
const DtsBundleWebpack = require('dts-bundle-webpack');
const nodeConfig = {
target: 'node',
mode: 'production',
devtool: 'source-map',
entry: './src/typescript/canvas3d.ts',
output: {
path: path.resolve(__dirname, 'dist'),
filename: 'cvat-canvas3d.node.js',
library: 'canvas3d',
libraryTarget: 'commonjs',
},
resolve: {
extensions: ['.ts', '.js', '.json'],
},
module: {
rules: [
{
test: /\.ts$/,
exclude: /node_modules/,
use: {
loader: 'babel-loader',
options: {
plugins: [
'@babel/plugin-proposal-class-properties',
'@babel/plugin-proposal-optional-chaining',
],
presets: [['@babel/preset-env'], ['@babel/typescript']],
sourceType: 'unambiguous',
},
},
},
{
test: /\.(css|scss)$/,
exclude: /node_modules/,
use: [
'style-loader',
{
loader: 'css-loader',
options: {
importLoaders: 2,
},
},
'postcss-loader',
'sass-loader',
],
},
],
},
plugins: [
new DtsBundleWebpack({
name: 'cvat-canvas3d.node',
main: 'dist/declaration/src/typescript/canvas3d.d.ts',
out: '../cvat-canvas3d.node.d.ts',
}),
],
};
const webConfig = {
target: 'web',
mode: 'production',
devtool: 'source-map',
entry: {
'cvat-canvas3d': './src/typescript/canvas3d.ts',
},
output: {
path: path.resolve(__dirname, 'dist'),
filename: '[name].[contenthash].js',
library: 'canvas3d',
libraryTarget: 'window',
},
devServer: {
contentBase: path.join(__dirname, 'dist'),
compress: false,
inline: true,
port: 3000,
},
resolve: {
extensions: ['.ts', '.js', '.json'],
},
module: {
rules: [
{
test: /\.ts$/,
exclude: /node_modules/,
use: {
loader: 'babel-loader',
options: {
plugins: ['@babel/plugin-proposal-class-properties'],
presets: [
[
'@babel/preset-env',
{
targets: '> 2.5%', // https://github.com/browserslist/browserslist
},
],
['@babel/typescript'],
],
sourceType: 'unambiguous',
},
},
},
{
test: /\.scss$/,
exclude: /node_modules/,
use: [
'style-loader',
{
loader: 'css-loader',
options: {
importLoaders: 2,
},
},
'postcss-loader',
'sass-loader',
],
},
],
},
plugins: [
new DtsBundleWebpack({
name: 'cvat-canvas3d',
main: 'dist/declaration/src/typescript/canvas3d.d.ts',
out: '../cvat-canvas3d.d.ts',
}),
],
};
module.exports = [webConfig, nodeConfig];

@ -1,4 +1,4 @@
// Copyright (C) 2019-2020 Intel Corporation // Copyright (C) 2021 Intel Corporation
// //
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
@ -341,6 +341,7 @@
constructor(size, chunkSize, stopFrame, taskID) { constructor(size, chunkSize, stopFrame, taskID) {
this._size = size; this._size = size;
this._buffer = {}; this._buffer = {};
this._contextImage = {};
this._requestedChunks = {}; this._requestedChunks = {};
this._chunkSize = chunkSize; this._chunkSize = chunkSize;
this._stopFrame = stopFrame; this._stopFrame = stopFrame;
@ -348,6 +349,18 @@
this._taskID = taskID; this._taskID = taskID;
} }
isContextImageAvailable(frame) {
return frame in this._contextImage;
}
getContextImage(frame) {
return this._contextImage[frame] || null;
}
addContextImage(frame, data) {
this._contextImage[frame] = data;
}
getFreeBufferSize() { getFreeBufferSize() {
let requestedFrameCount = 0; let requestedFrameCount = 0;
for (const chunk of Object.values(this._requestedChunks)) { for (const chunk of Object.values(this._requestedChunks)) {
@ -535,6 +548,37 @@
} }
} }
async function getImageContext(taskID, frame) {
return new Promise((resolve, reject) => {
serverProxy.frames
.getImageContext(taskID, frame)
.then((result) => {
if (isNode) {
// eslint-disable-next-line no-undef
resolve(global.Buffer.from(result, 'binary').toString('base64'));
} else if (isBrowser) {
const reader = new FileReader();
reader.onload = () => {
resolve(reader.result);
};
reader.readAsDataURL(result);
}
})
.catch((error) => {
reject(error);
});
});
}
async function getContextImage(taskID, frame) {
if (frameDataCache[taskID].frameBuffer.isContextImageAvailable(frame)) {
return frameDataCache[taskID].frameBuffer.getContextImage(frame);
}
const response = getImageContext(taskID, frame);
frameDataCache[taskID].frameBuffer.addContextImage(frame, response);
return frameDataCache[taskID].frameBuffer.getContextImage(frame);
}
async function getPreview(taskID) { async function getPreview(taskID) {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
// Just go to server and get preview (no any cache) // Just go to server and get preview (no any cache)
@ -558,7 +602,18 @@
}); });
} }
async function getFrame(taskID, chunkSize, chunkType, mode, frame, startFrame, stopFrame, isPlaying, step) { async function getFrame(
taskID,
chunkSize,
chunkType,
mode,
frame,
startFrame,
stopFrame,
isPlaying,
step,
dimension,
) {
if (!(taskID in frameDataCache)) { if (!(taskID in frameDataCache)) {
const blockType = chunkType === 'video' ? cvatData.BlockType.MP4VIDEO : cvatData.BlockType.ARCHIVE; const blockType = chunkType === 'video' ? cvatData.BlockType.MP4VIDEO : cvatData.BlockType.ARCHIVE;
@ -584,6 +639,7 @@
Math.max(decodedBlocksCacheSize, 9), Math.max(decodedBlocksCacheSize, 9),
decodedBlocksCacheSize, decodedBlocksCacheSize,
1, 1,
dimension,
), ),
frameBuffer: new FrameBuffer( frameBuffer: new FrameBuffer(
Math.min(180, decodedBlocksCacheSize * chunkSize), Math.min(180, decodedBlocksCacheSize * chunkSize),
@ -630,5 +686,6 @@
getRanges, getRanges,
getPreview, getPreview,
clear, clear,
getContextImage,
}; };
})(); })();

@ -718,6 +718,29 @@
return response.data; return response.data;
} }
async function getImageContext(tid, frame) {
const { backendAPI } = config;
let response = null;
try {
response = await Axios.get(
`${backendAPI}/tasks/${tid}/data?quality=original&type=context_image&number=${frame}`,
{
proxy: config.proxy,
responseType: 'blob',
},
);
} catch (errorData) {
const code = errorData.response ? errorData.response.status : errorData.code;
throw new ServerError(
`Could not get Image Context of the frame for the task ${tid} from the server`,
code,
);
}
return response.data;
}
async function getData(tid, chunk) { async function getData(tid, chunk) {
const { backendAPI } = config; const { backendAPI } = config;
@ -1053,6 +1076,7 @@
getData, getData,
getMeta, getMeta,
getPreview, getPreview,
getImageContext,
}), }),
writable: false, writable: false,
}, },

@ -8,7 +8,7 @@
const loggerStorage = require('./logger-storage'); const loggerStorage = require('./logger-storage');
const serverProxy = require('./server-proxy'); const serverProxy = require('./server-proxy');
const { const {
getFrame, getRanges, getPreview, clear: clearFrames, getFrame, getRanges, getPreview, clear: clearFrames, getContextImage,
} = require('./frames'); } = require('./frames');
const { ArgumentError } = require('./exceptions'); const { ArgumentError } = require('./exceptions');
const { TaskStatus } = require('./enums'); const { TaskStatus } = require('./enums');
@ -183,6 +183,15 @@
const result = await PluginRegistry.apiWrapper.call(this, prototype.frames.preview); const result = await PluginRegistry.apiWrapper.call(this, prototype.frames.preview);
return result; return result;
}, },
async contextImage(taskId, frameId) {
const result = await PluginRegistry.apiWrapper.call(
this,
prototype.frames.contextImage,
taskId,
frameId,
);
return result;
},
}, },
writable: true, writable: true,
}), }),
@ -850,6 +859,7 @@
get: Object.getPrototypeOf(this).frames.get.bind(this), get: Object.getPrototypeOf(this).frames.get.bind(this),
ranges: Object.getPrototypeOf(this).frames.ranges.bind(this), ranges: Object.getPrototypeOf(this).frames.ranges.bind(this),
preview: Object.getPrototypeOf(this).frames.preview.bind(this), preview: Object.getPrototypeOf(this).frames.preview.bind(this),
contextImage: Object.getPrototypeOf(this).frames.contextImage.bind(this),
}; };
this.logger = { this.logger = {
@ -1501,6 +1511,7 @@
get: Object.getPrototypeOf(this).frames.get.bind(this), get: Object.getPrototypeOf(this).frames.get.bind(this),
ranges: Object.getPrototypeOf(this).frames.ranges.bind(this), ranges: Object.getPrototypeOf(this).frames.ranges.bind(this),
preview: Object.getPrototypeOf(this).frames.preview.bind(this), preview: Object.getPrototypeOf(this).frames.preview.bind(this),
contextImage: Object.getPrototypeOf(this).frames.contextImage.bind(this),
}; };
this.logger = { this.logger = {
@ -1683,6 +1694,7 @@
this.stopFrame, this.stopFrame,
isPlaying, isPlaying,
step, step,
this.task.dimension,
); );
return frameData; return frameData;
}; };
@ -2142,4 +2154,9 @@
const result = await loggerStorage.log(logType, { ...payload, task_id: this.id }, wait); const result = await loggerStorage.log(logType, { ...payload, task_id: this.id }, wait);
return result; return result;
}; };
Job.prototype.frames.contextImage.implementation = async function (taskId, frameId) {
const result = await getContextImage(taskId, frameId);
return result;
};
})(); })();

@ -1,4 +1,4 @@
// Copyright (C) 2020 Intel Corporation // Copyright (C) 2021 Intel Corporation
// //
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
@ -13,8 +13,20 @@ const BlockType = Object.freeze({
ARCHIVE: 'archive', ARCHIVE: 'archive',
}); });
const DimensionType = Object.freeze({
DIM_3D: '3d',
DIM_2D: '2d',
});
class FrameProvider { class FrameProvider {
constructor(blockType, blockSize, cachedBlockCount, decodedBlocksCacheSize = 5, maxWorkerThreadCount = 2) { constructor(
blockType,
blockSize,
cachedBlockCount,
decodedBlocksCacheSize = 5,
maxWorkerThreadCount = 2,
dimension = DimensionType.DIM_2D,
) {
this._frames = {}; this._frames = {};
this._cachedBlockCount = Math.max(1, cachedBlockCount); // number of stored blocks this._cachedBlockCount = Math.max(1, cachedBlockCount); // number of stored blocks
this._decodedBlocksCacheSize = decodedBlocksCacheSize; this._decodedBlocksCacheSize = decodedBlocksCacheSize;
@ -33,6 +45,7 @@ class FrameProvider {
this._mutex = new Mutex(); this._mutex = new Mutex();
this._promisedFrames = {}; this._promisedFrames = {};
this._maxWorkerThreadCount = maxWorkerThreadCount; this._maxWorkerThreadCount = maxWorkerThreadCount;
this._dimension = dimension;
} }
async _worker() { async _worker() {
@ -291,7 +304,7 @@ class FrameProvider {
}; };
worker.onmessage = async (event) => { worker.onmessage = async (event) => {
if (event.data.isRaw) { if (this._dimension === DimensionType.DIM_2D && event.data.isRaw) {
// safary doesn't support createImageBitmap // safary doesn't support createImageBitmap
// there is a way to polyfill it with using document.createElement // there is a way to polyfill it with using document.createElement
// but document.createElement doesn't work in worker // but document.createElement doesn't work in worker
@ -328,8 +341,14 @@ class FrameProvider {
} }
index++; index++;
}; };
const dimension = this._dimension;
worker.postMessage({ block, start, end }); worker.postMessage({
block,
start,
end,
dimension,
dimension2D: DimensionType.DIM_2D,
});
this._decodeThreadCount++; this._decodeThreadCount++;
} }
} finally { } finally {
@ -357,4 +376,5 @@ class FrameProvider {
module.exports = { module.exports = {
FrameProvider, FrameProvider,
BlockType, BlockType,
DimensionType,
}; };

@ -7,7 +7,7 @@ const JSZip = require('jszip');
onmessage = (e) => { onmessage = (e) => {
const zip = new JSZip(); const zip = new JSZip();
if (e.data) { if (e.data) {
const { start, end, block } = e.data; const { start, end, block, dimension, dimension2D } = e.data;
zip.loadAsync(block).then((_zip) => { zip.loadAsync(block).then((_zip) => {
let index = start; let index = start;
@ -18,7 +18,7 @@ onmessage = (e) => {
.async('blob') .async('blob')
.then((fileData) => { .then((fileData) => {
// eslint-disable-next-line no-restricted-globals // eslint-disable-next-line no-restricted-globals
if (self.createImageBitmap) { if (dimension === dimension2D && self.createImageBitmap) {
createImageBitmap(fileData).then((img) => { createImageBitmap(fileData).then((img) => {
postMessage({ postMessage({
fileName: relativePath, fileName: relativePath,

@ -3988,6 +3988,19 @@
"array-find-index": "^1.0.1" "array-find-index": "^1.0.1"
} }
}, },
"cvat-canvas3d": {
"version": "file:../cvat-canvas3d",
"requires": {
"three": "^0.125.0"
},
"dependencies": {
"three": {
"version": "0.125.2",
"resolved": "https://registry.npmjs.org/three/-/three-0.125.2.tgz",
"integrity": "sha512-7rIRO23jVKWcAPFdW/HREU2NZMGWPBZ4XwEMt0Ak0jwLUKVJhcKM55eCBWyGZq/KiQbeo1IeuAoo/9l2dzhTXA=="
}
}
},
"cvat-canvas": { "cvat-canvas": {
"version": "file:../cvat-canvas", "version": "file:../cvat-canvas",
"requires": { "requires": {

@ -60,6 +60,7 @@
"@types/redux-logger": "^3.0.8", "@types/redux-logger": "^3.0.8",
"antd": "^4.10.2", "antd": "^4.10.2",
"copy-to-clipboard": "^3.3.1", "copy-to-clipboard": "^3.3.1",
"cvat-canvas3d": "file:../cvat-canvas3d",
"cvat-canvas": "file:../cvat-canvas", "cvat-canvas": "file:../cvat-canvas",
"cvat-core": "file:../cvat-core", "cvat-core": "file:../cvat-core",
"dotenv-webpack": "^1.8.0", "dotenv-webpack": "^1.8.0",

@ -1,4 +1,4 @@
// Copyright (C) 2020 Intel Corporation // Copyright (C) 2021 Intel Corporation
// //
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
@ -18,6 +18,7 @@ import {
ContextMenuType, ContextMenuType,
Workspace, Workspace,
Model, Model,
DimensionType,
OpenCVTool, OpenCVTool,
} from 'reducers/interfaces'; } from 'reducers/interfaces';
@ -190,6 +191,8 @@ export enum AnnotationActionTypes {
SWITCH_REQUEST_REVIEW_DIALOG = 'SWITCH_REQUEST_REVIEW_DIALOG', SWITCH_REQUEST_REVIEW_DIALOG = 'SWITCH_REQUEST_REVIEW_DIALOG',
SWITCH_SUBMIT_REVIEW_DIALOG = 'SWITCH_SUBMIT_REVIEW_DIALOG', SWITCH_SUBMIT_REVIEW_DIALOG = 'SWITCH_SUBMIT_REVIEW_DIALOG',
SET_FORCE_EXIT_ANNOTATION_PAGE_FLAG = 'SET_FORCE_EXIT_ANNOTATION_PAGE_FLAG', SET_FORCE_EXIT_ANNOTATION_PAGE_FLAG = 'SET_FORCE_EXIT_ANNOTATION_PAGE_FLAG',
HIDE_SHOW_CONTEXT_IMAGE = 'HIDE_SHOW_CONTEXT_IMAGE',
GET_CONTEXT_IMAGE = 'GET_CONTEXT_IMAGE',
} }
export function saveLogsAsync(): ThunkAction { export function saveLogsAsync(): ThunkAction {
@ -958,6 +961,10 @@ export function getJobAsync(tid: number, jid: number, initialFrame: number, init
maxZ, maxZ,
}, },
}); });
if (job.task.dimension === DimensionType.DIM_3D) {
const workspace = Workspace.STANDARD3D;
dispatch(changeWorkspace(workspace));
}
dispatch(changeFrameAsync(frameNumber, false)); dispatch(changeFrameAsync(frameNumber, false));
} catch (error) { } catch (error) {
dispatch({ dispatch({
@ -1522,3 +1529,46 @@ export function setForceExitAnnotationFlag(forceExit: boolean): AnyAction {
}, },
}; };
} }
export function hideShowContextImage(hidden: boolean): AnyAction {
return {
type: AnnotationActionTypes.HIDE_SHOW_CONTEXT_IMAGE,
payload: {
hidden,
},
};
}
export function getContextImage(): ThunkAction {
return async (dispatch: ActionCreator<Dispatch>): Promise<void> => {
const state: CombinedState = getStore().getState();
const { instance: job } = state.annotation.job;
const { frame, contextImage } = state.annotation.player;
try {
const context = await job.frames.contextImage(job.task.id, frame.number);
const loaded = true;
const contextImageHide = contextImage.hidden;
dispatch({
type: AnnotationActionTypes.GET_CONTEXT_IMAGE,
payload: {
context,
loaded,
contextImageHide,
},
});
} catch (error) {
const context = '';
const loaded = true;
const contextImageHide = contextImage.hidden;
dispatch({
type: AnnotationActionTypes.GET_CONTEXT_IMAGE,
payload: {
context,
loaded,
contextImageHide,
},
});
}
};
}

@ -18,6 +18,7 @@ import TagAnnotationWorkspace from 'components/annotation-page/tag-annotation-wo
import ReviewAnnotationsWorkspace from 'components/annotation-page/review-workspace/review-workspace'; import ReviewAnnotationsWorkspace from 'components/annotation-page/review-workspace/review-workspace';
import SubmitAnnotationsModal from 'components/annotation-page/request-review-modal'; import SubmitAnnotationsModal from 'components/annotation-page/request-review-modal';
import SubmitReviewModal from 'components/annotation-page/review/submit-review-modal'; import SubmitReviewModal from 'components/annotation-page/review/submit-review-modal';
import StandardWorkspace3DComponent from 'components/annotation-page/standard3D-workspace/standard3D-workspace';
interface Props { interface Props {
job: any | null | undefined; job: any | null | undefined;
@ -79,23 +80,28 @@ export default function AnnotationPageComponent(props: Props): JSX.Element {
<Layout.Header className='cvat-annotation-header'> <Layout.Header className='cvat-annotation-header'>
<AnnotationTopBarContainer /> <AnnotationTopBarContainer />
</Layout.Header> </Layout.Header>
{workspace === Workspace.STANDARD3D && (
<Layout.Content className='cvat-annotation-layout-content'>
<StandardWorkspace3DComponent />
</Layout.Content>
)}
{workspace === Workspace.STANDARD && ( {workspace === Workspace.STANDARD && (
<Layout.Content style={{ height: '100%' }}> <Layout.Content className='cvat-annotation-layout-content'>
<StandardWorkspaceComponent /> <StandardWorkspaceComponent />
</Layout.Content> </Layout.Content>
)} )}
{workspace === Workspace.ATTRIBUTE_ANNOTATION && ( {workspace === Workspace.ATTRIBUTE_ANNOTATION && (
<Layout.Content style={{ height: '100%' }}> <Layout.Content className='cvat-annotation-layout-content'>
<AttributeAnnotationWorkspace /> <AttributeAnnotationWorkspace />
</Layout.Content> </Layout.Content>
)} )}
{workspace === Workspace.TAG_ANNOTATION && ( {workspace === Workspace.TAG_ANNOTATION && (
<Layout.Content style={{ height: '100%' }}> <Layout.Content className='cvat-annotation-layout-content'>
<TagAnnotationWorkspace /> <TagAnnotationWorkspace />
</Layout.Content> </Layout.Content>
)} )}
{workspace === Workspace.REVIEW_WORKSPACE && ( {workspace === Workspace.REVIEW_WORKSPACE && (
<Layout.Content style={{ height: '100%' }}> <Layout.Content className='cvat-annotation-layout-content'>
<ReviewAnnotationsWorkspace /> <ReviewAnnotationsWorkspace />
</Layout.Content> </Layout.Content>
)} )}

@ -0,0 +1,109 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React, { ReactElement, useEffect, useRef } from 'react';
import { GlobalHotKeys } from 'react-hotkeys';
import Layout from 'antd/lib/layout/layout';
import { Workspace } from 'reducers/interfaces';
import { Canvas3d } from 'cvat-canvas3d-wrapper';
import ContextImage from '../standard3D-workspace/context-image/context-image';
interface Props {
canvasInstance: Canvas3d;
jobInstance: any;
frameData: any;
curZLayer: number;
contextImageHide: boolean;
loaded: boolean;
data: string;
annotations: any[];
onSetupCanvas: () => void;
getContextImage(): void;
workspace: Workspace;
animateID: any;
automaticBordering: boolean;
showObjectsTextAlways: boolean;
}
const CanvasWrapperComponent = (props: Props): ReactElement => {
const animateId = useRef(0);
const cvatCanvasContainerRef = useRef();
const {
frameData, contextImageHide, getContextImage, loaded, data, annotations, curZLayer,
} = props;
const fitCanvas = (): void => {
const { canvasInstance } = props;
canvasInstance.fitCanvas();
};
const onCanvasSetup = (): void => {
const { onSetupCanvas } = props;
onSetupCanvas();
};
const animateCanvas = (): void => {
const { canvasInstance } = props;
canvasInstance.render();
animateId.current = requestAnimationFrame(animateCanvas);
};
const updateCanvas = (): void => {
const { canvasInstance } = props;
if (frameData !== null) {
canvasInstance.setup(frameData);
}
};
const initialSetup = (): void => {
const { canvasInstance } = props;
// Size
window.addEventListener('resize', fitCanvas);
fitCanvas();
// Events
canvasInstance.html().addEventListener('canvas.setup', onCanvasSetup);
};
useEffect(() => {
const { canvasInstance } = props;
cvatCanvasContainerRef.current.appendChild(canvasInstance.html());
initialSetup();
updateCanvas();
animateCanvas();
return () => {
canvasInstance.html().removeEventListener('canvas.setup', onCanvasSetup);
window.removeEventListener('resize', fitCanvas);
cancelAnimationFrame(animateId.current);
};
});
useEffect(() => {
updateCanvas();
}, [frameData, annotations, curZLayer]);
return (
<Layout.Content style={{ position: 'relative' }}>
<GlobalHotKeys />
<ContextImage
frame={frameData}
contextImageHide={contextImageHide}
getContextImage={getContextImage}
loaded={loaded}
data={data}
/>
<div ref={cvatCanvasContainerRef} className='cvat-canvas-container cvat-canvas-container-overflow' />
</Layout.Content>
);
};
export default React.memo(CanvasWrapperComponent);

@ -0,0 +1,34 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React, { useEffect } from 'react';
interface Props {
frame: number;
contextImageHide: boolean;
loaded: boolean;
data: string;
getContextImage(): void;
}
export default function ContextImage(props: Props): JSX.Element {
const {
contextImageHide, loaded, data, getContextImage,
} = props;
useEffect(() => {
if (!contextImageHide && !loaded) {
getContextImage();
}
}, [contextImageHide, loaded]);
if (!contextImageHide && data !== '') {
return (
<div className='cvat-contextImage'>
<img src={data} alt='Context not available' className='cvat-contextImage-show' />
</div>
);
}
return null;
}

@ -0,0 +1,56 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React from 'react';
import { GlobalHotKeys } from 'react-hotkeys';
import Layout from 'antd/lib/layout';
import { ActiveControl } from 'reducers/interfaces';
import { Canvas3d as Canvas } from 'cvat-canvas3d-wrapper';
import CursorControl from './cursor-control';
import MoveControl from './move-control';
import DrawCuboidControl from './draw-cuboid-control';
import PhotoContextControl from './photo-context';
interface Props {
canvasInstance: Canvas;
activeControl: ActiveControl;
normalizedKeyMap: Record<string, string>;
contextImageHide: boolean;
hideShowContextImage: (hidden: boolean) => void;
}
export default function ControlsSideBarComponent(props: Props): JSX.Element {
const {
canvasInstance, activeControl, normalizedKeyMap, contextImageHide, hideShowContextImage,
} = props;
return (
<Layout.Sider className='cvat-canvas-controls-sidebar' theme='light' width={44}>
<GlobalHotKeys />
<MoveControl canvasInstance={canvasInstance} activeControl={activeControl} />
<CursorControl
cursorShortkey={normalizedKeyMap.CANCEL}
canvasInstance={canvasInstance}
activeControl={activeControl}
/>
<DrawCuboidControl
canvasInstance={canvasInstance}
isDrawing={activeControl === ActiveControl.DRAW_CUBOID}
/>
<PhotoContextControl
canvasInstance={canvasInstance}
activeControl={activeControl}
contextImageHide={contextImageHide}
hideShowContextImage={hideShowContextImage}
/>
</Layout.Sider>
);
}

@ -0,0 +1,35 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React from 'react';
import Icon from '@ant-design/icons';
import Tooltip from 'antd/lib/tooltip';
import { CursorIcon } from 'icons';
import { ActiveControl } from 'reducers/interfaces';
import { Canvas3d as Canvas } from 'cvat-canvas3d-wrapper';
interface Props {
canvasInstance: Canvas;
cursorShortkey: string;
activeControl: ActiveControl;
}
function CursorControl(props: Props): JSX.Element {
const { activeControl, cursorShortkey } = props;
return (
<Tooltip title={`Cursor ${cursorShortkey}`} placement='right' mouseLeaveDelay={0}>
<Icon
component={CursorIcon}
className={[
'cvat-cursor-control',
activeControl === ActiveControl.CURSOR ? 'cvat-active-canvas-control ' : null,
]}
/>
</Tooltip>
);
}
export default React.memo(CursorControl);

@ -0,0 +1,55 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React from 'react';
import Popover from 'antd/lib/popover';
import Icon from '@ant-design/icons';
import { Canvas3d as Canvas } from 'cvat-canvas3d-wrapper';
import { ShapeType } from 'reducers/interfaces';
import { CubeIcon } from 'icons';
import DrawShapePopoverContainer from 'containers/annotation-page/standard-workspace/controls-side-bar/draw-shape-popover';
interface Props {
canvasInstance: Canvas;
isDrawing: boolean;
}
function DrawPolygonControl(props: Props): JSX.Element {
const { canvasInstance, isDrawing } = props;
const dynamcPopoverPros = isDrawing ?
{
overlayStyle: {
display: 'none',
},
} :
{};
const dynamicIconProps = isDrawing ?
{
className: 'cvat-draw-cuboid-control cvat-active-canvas-control',
onClick: (): void => {
canvasInstance.draw({ enabled: false });
},
} :
{
className: 'cvat-draw-cuboid-control',
};
return (
<Popover
{...dynamcPopoverPros}
overlayClassName='cvat-draw-shape-popover'
placement='right'
content={<DrawShapePopoverContainer shapeType={ShapeType.CUBOID} />}
>
<Icon {...dynamicIconProps} component={CubeIcon} />
</Popover>
);
}
export default React.memo(DrawPolygonControl);

@ -0,0 +1,34 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import React from 'react';
import Icon from '@ant-design/icons';
import Tooltip from 'antd/lib/tooltip';
import { MoveIcon } from 'icons';
import { ActiveControl } from 'reducers/interfaces';
import { Canvas3d as Canvas } from 'cvat-canvas3d-wrapper';
interface Props {
canvasInstance: Canvas;
activeControl: ActiveControl;
}
function MoveControl(props: Props): JSX.Element {
const { activeControl } = props;
return (
<Tooltip title='Move the image' placement='right' mouseLeaveDelay={0}>
<Icon
component={MoveIcon}
className={[
'cvat-move-control',
activeControl === ActiveControl.DRAG_CANVAS ? ' cvat-active-canvas-control' : null,
]}
/>
</Tooltip>
);
}
export default React.memo(MoveControl);

@ -0,0 +1,36 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import CameraIcon from '@ant-design/icons/CameraOutlined';
import Tooltip from 'antd/lib/tooltip';
import { Canvas3d as Canvas } from 'cvat-canvas3d-wrapper';
import React from 'react';
import { ActiveControl } from 'reducers/interfaces';
interface Props {
canvasInstance: Canvas;
activeControl: ActiveControl;
hideShowContextImage: (hidden: boolean) => void;
contextImageHide: boolean;
}
function PhotoContextControl(props: Props): JSX.Element {
const { activeControl, contextImageHide, hideShowContextImage } = props;
return (
<Tooltip title='Photo context show/hide' placement='right' mouseLeaveDelay={0}>
<CameraIcon
className={`cvat-move-control
cvat-control-side-bar-icon-size ${
activeControl === ActiveControl.PHOTO_CONTEXT ? 'cvat-active-canvas-control' : ''
}`}
onClick={(): void => {
hideShowContextImage(!contextImageHide);
}}
/>
</Tooltip>
);
}
export default React.memo(PhotoContextControl);

@ -0,0 +1,19 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import './styles.scss';
import React from 'react';
import Layout from 'antd/lib/layout';
import CanvasWrapperContainer from 'containers/annotation-page/canvas/canvas-wrapper3D';
import ControlsSideBarContainer from 'containers/annotation-page/standard3D-workspace/controls-side-bar/controls-side-bar';
export default function StandardWorkspace3DComponent(): JSX.Element {
return (
<Layout hasSider className='cvat-standard-workspace'>
<ControlsSideBarContainer />
<CanvasWrapperContainer />
</Layout>
);
}

@ -0,0 +1,173 @@
// Copyright (C) 2020 Intel Corporation
//
// SPDX-License-Identifier: MIT
@import 'base.scss';
.cvat-standard-workspace.ant-layout {
height: 100%;
}
.cvat-contextImage {
width: $grid-unit-size * 32;
position: absolute;
background: $border-color-3;
top: $grid-unit-size;
right: $grid-unit-size;
max-height: $grid-unit-size * 16;
z-index: 100;
border-radius: $grid-unit-size;
border: 1px solid $border-color-3;
display: flex;
flex-direction: column;
justify-content: space-between;
padding: $grid-unit-size/2;
}
.cvat-contextImage-show {
max-width: 100%;
max-height: 100%;
}
.cvat-contextImage-loading {
text-align: center;
}
.cvat-objects-sidebar-filter-input {
width: calc(100% - 35px);
}
.cvat-objects-sidebar-sider {
top: 0;
right: 0;
left: auto;
background-color: $background-color-2;
border-left: 1px solid $border-color-1;
border-bottom: 1px solid $border-color-1;
border-radius: $grid-unit-size/2 0 0 $grid-unit-size/2;
z-index: 2;
}
.cvat-objects-sidebar {
height: 100%;
}
.cvat-rotate-canvas-controls-right > svg {
transform: scaleX(-1);
}
.cvat-canvas-controls-sidebar {
background-color: $background-color-2;
border-right: 1px solid $border-color-1;
> div {
> i {
border-radius: 3.3px;
transform: scale(0.65);
padding: $grid-unit-size/4;
&:hover {
background: $header-color;
transform: scale(0.75);
}
&:active {
transform: scale(0.65);
}
> svg {
transform: scale(0.8);
}
}
}
}
.cvat-active-canvas-control {
background: $header-color;
transform: scale(0.75);
}
.cvat-rotate-canvas-controls-left,
.cvat-rotate-canvas-controls-right {
transform: scale(0.65);
border-radius: $grid-unit-size/2;
&:hover {
transform: scale(0.75);
}
&:active {
transform: scale(0.65);
}
}
.cvat-rotate-canvas-controls > .ant-popover-content > .ant-popover-inner > div > .ant-popover-inner-content {
padding: 0;
}
.cvat-draw-shape-popover,
.cvat-tools-control-popover {
> .ant-popover-content > .ant-popover-inner > div > .ant-popover-inner-content {
padding: 0;
}
}
.cvat-tools-track-button,
.cvat-tools-interact-button {
width: 100%;
margin-top: $grid-unit-size;
}
.cvat-draw-shape-popover-points-selector {
width: 100%;
}
.cvat-tools-control-popover-content {
width: fit-content;
padding: $grid-unit-size;
border-radius: $grid-unit-size/2;
background: $background-color-2;
}
.cvat-draw-shape-popover-content {
padding: $grid-unit-size;
border-radius: $grid-unit-size/2;
background: $background-color-2;
width: 270px;
> div {
margin-top: $grid-unit-size/2;
}
> div:nth-child(3) > div > div {
width: 100%;
}
> div:last-child {
span {
width: 100%;
}
button {
width: 100%;
&:nth-child(1) {
border-radius: $grid-unit-size/2 0 0 $grid-unit-size/2;
}
&:nth-child(2) {
border-radius: 0 $grid-unit-size/2 $grid-unit-size/2 0;
}
}
}
}
.cvat-canvas-container-overflow {
overflow: hidden;
width: 100%;
height: 100%;
}
.cvat-control-side-bar-icon-size {
font-size: $grid-unit-size * 5;
}

@ -9,6 +9,10 @@
overflow: hidden; overflow: hidden;
} }
.cvat-annotation-layout-content {
height: 100%;
}
.ant-layout-header.cvat-annotation-header { .ant-layout-header.cvat-annotation-header {
background-color: $background-color-2; background-color: $background-color-2;
border-bottom: 1px solid $border-color-1; border-bottom: 1px solid $border-color-1;

@ -8,17 +8,20 @@ import Icon from '@ant-design/icons';
import Select from 'antd/lib/select'; import Select from 'antd/lib/select';
import Button from 'antd/lib/button'; import Button from 'antd/lib/button';
import { Workspace } from 'reducers/interfaces'; import { DimensionType, Workspace } from 'reducers/interfaces';
import { InfoIcon, FullscreenIcon } from 'icons'; import { InfoIcon, FullscreenIcon } from 'icons';
interface Props { interface Props {
workspace: Workspace; workspace: Workspace;
showStatistics(): void; showStatistics(): void;
changeWorkspace(workspace: Workspace): void; changeWorkspace(workspace: Workspace): void;
jobInstance: any;
} }
function RightGroup(props: Props): JSX.Element { function RightGroup(props: Props): JSX.Element {
const { showStatistics, changeWorkspace, workspace } = props; const {
showStatistics, changeWorkspace, workspace, jobInstance,
} = props;
return ( return (
<Col className='cvat-annotation-header-right-group'> <Col className='cvat-annotation-header-right-group'>
@ -49,11 +52,26 @@ function RightGroup(props: Props): JSX.Element {
onChange={changeWorkspace} onChange={changeWorkspace}
value={workspace} value={workspace}
> >
{Object.values(Workspace).map((ws) => ( {Object.values(Workspace).map((ws) => {
<Select.Option key={ws} value={ws}> if (jobInstance.task.dimension === DimensionType.DIM_3D) {
{ws} if (ws === Workspace.STANDARD) {
</Select.Option> return null;
))} }
return (
<Select.Option disabled={ws !== Workspace.STANDARD3D} key={ws} value={ws}>
{ws}
</Select.Option>
);
}
if (ws !== Workspace.STANDARD3D) {
return (
<Select.Option key={ws} value={ws}>
{ws}
</Select.Option>
);
}
return null;
})}
</Select> </Select>
</div> </div>
</Col> </Col>

@ -53,6 +53,8 @@ interface Props {
onURLIconClick(): void; onURLIconClick(): void;
onUndoClick(): void; onUndoClick(): void;
onRedoClick(): void; onRedoClick(): void;
jobInstance: any;
hideShowContextImage(): any;
} }
export default function AnnotationTopBarComponent(props: Props): JSX.Element { export default function AnnotationTopBarComponent(props: Props): JSX.Element {
@ -96,6 +98,7 @@ export default function AnnotationTopBarComponent(props: Props): JSX.Element {
onURLIconClick, onURLIconClick,
onUndoClick, onUndoClick,
onRedoClick, onRedoClick,
jobInstance,
} = props; } = props;
return ( return (
@ -146,7 +149,12 @@ export default function AnnotationTopBarComponent(props: Props): JSX.Element {
/> />
</Row> </Row>
</Col> </Col>
<RightGroup workspace={workspace} changeWorkspace={changeWorkspace} showStatistics={showStatistics} /> <RightGroup
jobInstance={jobInstance}
workspace={workspace}
changeWorkspace={changeWorkspace}
showStatistics={showStatistics}
/>
</Row> </Row>
); );
} }

@ -1,4 +1,4 @@
// Copyright (C) 2020 Intel Corporation // Copyright (C) 2021 Intel Corporation
// //
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT

@ -0,0 +1,69 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import { connect } from 'react-redux';
import CanvasWrapperComponent from 'components/annotation-page/canvas/canvas-wrapper3D';
import { confirmCanvasReady, getContextImage } from 'actions/annotation-actions';
import { CombinedState } from 'reducers/interfaces';
import { Canvas3d } from 'cvat-canvas3d-wrapper';
interface StateToProps {
canvasInstance: Canvas3d;
jobInstance: any;
frameData: any;
curZLayer: number;
contextImageHide: boolean;
loaded: boolean;
data: string;
annotations: any[];
}
interface DispatchToProps {
onSetupCanvas(): void;
getContextImage(): void;
}
function mapStateToProps(state: CombinedState): StateToProps {
const {
annotation: {
canvas: { instance: canvasInstance },
job: { instance: jobInstance },
player: {
frame: { data: frameData },
contextImage: { hidden: contextImageHide, data, loaded },
},
annotations: {
states: annotations,
zLayer: { cur: curZLayer },
},
},
} = state;
return {
canvasInstance,
jobInstance,
frameData,
curZLayer,
contextImageHide,
loaded,
data,
annotations,
};
}
function mapDispatchToProps(dispatch: any): DispatchToProps {
return {
onSetupCanvas(): void {
dispatch(confirmCanvasReady());
},
getContextImage(): void {
dispatch(getContextImage());
},
};
}
export default connect(mapStateToProps, mapDispatchToProps)(CanvasWrapperComponent);

@ -0,0 +1,55 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import { ExtendedKeyMapOptions } from 'react-hotkeys';
import { connect } from 'react-redux';
import { Canvas } from 'cvat-canvas-wrapper';
import { hideShowContextImage } from 'actions/annotation-actions';
import ControlsSideBarComponent from 'components/annotation-page/standard3D-workspace/controls-side-bar/controls-side-bar';
import { ActiveControl, CombinedState } from 'reducers/interfaces';
interface StateToProps {
canvasInstance: Canvas;
activeControl: ActiveControl;
keyMap: Record<string, ExtendedKeyMapOptions>;
normalizedKeyMap: Record<string, string>;
contextImageHide: boolean;
loaded: boolean;
}
interface DispatchToProps {
hideShowContextImage(hidden: boolean): void;
}
function mapStateToProps(state: CombinedState): StateToProps {
const {
annotation: {
canvas: { instance: canvasInstance, activeControl },
player: {
contextImage: { hidden: contextImageHide, loaded },
},
},
shortcuts: { keyMap, normalizedKeyMap },
} = state;
return {
canvasInstance,
activeControl,
normalizedKeyMap,
keyMap,
contextImageHide,
loaded,
};
}
function dispatchToProps(dispatch: any): DispatchToProps {
return {
hideShowContextImage(hidden: boolean): void {
dispatch(hideShowContextImage(hidden));
},
};
}
export default connect(mapStateToProps, dispatchToProps)(ControlsSideBarComponent);

@ -625,6 +625,7 @@ class AnnotationTopBarContainer extends React.PureComponent<Props, State> {
focusFrameInputShortcut={normalizedKeyMap.FOCUS_INPUT_FRAME} focusFrameInputShortcut={normalizedKeyMap.FOCUS_INPUT_FRAME}
onUndoClick={this.undo} onUndoClick={this.undo}
onRedoClick={this.redo} onRedoClick={this.redo}
jobInstance={jobInstance}
/> />
</> </>
); );

@ -0,0 +1,7 @@
// Copyright (C) 2021 Intel Corporation
//
// SPDX-License-Identifier: MIT
import { Canvas3d, Canvas3dVersion } from 'cvat-canvas3d/src/typescript/canvas3d';
export { Canvas3d, Canvas3dVersion };

@ -1,4 +1,4 @@
// Copyright (C) 2020 Intel Corporation // Copyright (C) 2021 Intel Corporation
// //
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
@ -6,6 +6,7 @@ import React from 'react';
import { AnyAction } from 'redux'; import { AnyAction } from 'redux';
import { Canvas, CanvasMode } from 'cvat-canvas-wrapper'; import { Canvas, CanvasMode } from 'cvat-canvas-wrapper';
import { Canvas3d } from 'cvat-canvas3d-wrapper';
import { AnnotationActionTypes } from 'actions/annotation-actions'; import { AnnotationActionTypes } from 'actions/annotation-actions';
import { AuthActionTypes } from 'actions/auth-actions'; import { AuthActionTypes } from 'actions/auth-actions';
import { BoundariesActionTypes } from 'actions/boundaries-actions'; import { BoundariesActionTypes } from 'actions/boundaries-actions';
@ -17,6 +18,7 @@ import {
ContextMenuType, ContextMenuType,
Workspace, Workspace,
TaskStatus, TaskStatus,
DimensionType,
} from './interfaces'; } from './interfaces';
const defaultState: AnnotationState = { const defaultState: AnnotationState = {
@ -55,6 +57,11 @@ const defaultState: AnnotationState = {
}, },
playing: false, playing: false,
frameAngles: [], frameAngles: [],
contextImage: {
loaded: false,
data: '',
hidden: false,
},
}, },
drawing: { drawing: {
activeShapeType: ShapeType.RECTANGLE, activeShapeType: ShapeType.RECTANGLE,
@ -133,7 +140,11 @@ export default (state = defaultState, action: AnyAction): AnnotationState => {
} = action.payload; } = action.payload;
const isReview = job.status === TaskStatus.REVIEW; const isReview = job.status === TaskStatus.REVIEW;
let workspaceSelected = Workspace.STANDARD;
if (job.task.dimension === DimensionType.DIM_3D) {
workspaceSelected = Workspace.STANDARD3D;
}
return { return {
...state, ...state,
job: { job: {
@ -176,10 +187,10 @@ export default (state = defaultState, action: AnyAction): AnnotationState => {
}, },
canvas: { canvas: {
...state.canvas, ...state.canvas,
instance: new Canvas(), instance: job.task.dimension === DimensionType.DIM_2D ? new Canvas() : new Canvas3d(),
}, },
colors, colors,
workspace: isReview ? Workspace.REVIEW_WORKSPACE : Workspace.STANDARD, workspace: isReview ? Workspace.REVIEW_WORKSPACE : workspaceSelected,
}; };
} }
case AnnotationActionTypes.GET_JOB_FAILED: { case AnnotationActionTypes.GET_JOB_FAILED: {
@ -201,6 +212,11 @@ export default (state = defaultState, action: AnyAction): AnnotationState => {
...state.player.frame, ...state.player.frame,
fetching: false, fetching: false,
}, },
contextImage: {
loaded: false,
data: '',
hidden: state.player.contextImage.hidden,
},
}, },
}; };
} }
@ -243,6 +259,10 @@ export default (state = defaultState, action: AnyAction): AnnotationState => {
changeTime, changeTime,
delay, delay,
}, },
contextImage: {
...state.player.contextImage,
loaded: false,
},
}, },
annotations: { annotations: {
...state.annotations, ...state.annotations,
@ -1076,6 +1096,36 @@ export default (state = defaultState, action: AnyAction): AnnotationState => {
}, },
}; };
} }
case AnnotationActionTypes.HIDE_SHOW_CONTEXT_IMAGE: {
const { hidden } = action.payload;
const { loaded, data } = state.player.contextImage;
return {
...state,
player: {
...state.player,
contextImage: {
loaded,
data,
hidden,
},
},
};
}
case AnnotationActionTypes.GET_CONTEXT_IMAGE: {
const { context, loaded } = action.payload;
return {
...state,
player: {
...state.player,
contextImage: {
loaded,
data: context,
hidden: state.player.contextImage.hidden,
},
},
};
}
case AnnotationActionTypes.CLOSE_JOB: case AnnotationActionTypes.CLOSE_JOB:
case AuthActionTypes.LOGOUT_SUCCESS: { case AuthActionTypes.LOGOUT_SUCCESS: {
return { ...defaultState }; return { ...defaultState };

@ -6,6 +6,7 @@ import { ExtendedKeyMapOptions } from 'react-hotkeys';
import { Canvas, RectDrawingMethod } from 'cvat-canvas-wrapper'; import { Canvas, RectDrawingMethod } from 'cvat-canvas-wrapper';
import { IntelligentScissors } from 'utils/opencv-wrapper/intelligent-scissors'; import { IntelligentScissors } from 'utils/opencv-wrapper/intelligent-scissors';
import { MutableRefObject } from 'react'; import { MutableRefObject } from 'react';
import { Canvas3d } from 'cvat-canvas3d/src/typescript/canvas3d';
export type StringObject = { export type StringObject = {
[index: string]: string; [index: string]: string;
@ -332,6 +333,7 @@ export enum ActiveControl {
EDIT = 'edit', EDIT = 'edit',
OPEN_ISSUE = 'open_issue', OPEN_ISSUE = 'open_issue',
AI_TOOLS = 'ai_tools', AI_TOOLS = 'ai_tools',
PHOTO_CONTEXT = 'PHOTO_CONTEXT',
OPENCV_TOOLS = 'opencv_tools', OPENCV_TOOLS = 'opencv_tools',
} }
@ -381,7 +383,7 @@ export interface AnnotationState {
pointID: number | null; pointID: number | null;
clientID: number | null; clientID: number | null;
}; };
instance: Canvas; instance: Canvas | Canvas3d;
ready: boolean; ready: boolean;
activeControl: ActiveControl; activeControl: ActiveControl;
}; };
@ -404,6 +406,11 @@ export interface AnnotationState {
}; };
playing: boolean; playing: boolean;
frameAngles: number[]; frameAngles: number[];
contextImage: {
loaded: boolean;
data: string;
hidden: boolean;
};
}; };
drawing: { drawing: {
activeInteractor?: Model | OpenCVTool; activeInteractor?: Model | OpenCVTool;
@ -459,6 +466,7 @@ export interface AnnotationState {
} }
export enum Workspace { export enum Workspace {
STANDARD3D = 'Standard 3D',
STANDARD = 'Standard', STANDARD = 'Standard',
ATTRIBUTE_ANNOTATION = 'Attribute annotation', ATTRIBUTE_ANNOTATION = 'Attribute annotation',
TAG_ANNOTATION = 'Tag annotation', TAG_ANNOTATION = 'Tag annotation',

@ -4,11 +4,13 @@
import os import os
import os.path as osp import os.path as osp
import io
import shutil import shutil
import traceback import traceback
from datetime import datetime from datetime import datetime
from distutils.util import strtobool from distutils.util import strtobool
from tempfile import mkstemp from tempfile import mkstemp
import cv2
import django_rq import django_rq
from django.shortcuts import get_object_or_404 from django.shortcuts import get_object_or_404
@ -39,7 +41,7 @@ from cvat.apps.dataset_manager.serializers import DatasetFormatsSerializer
from cvat.apps.engine.frame_provider import FrameProvider from cvat.apps.engine.frame_provider import FrameProvider
from cvat.apps.engine.models import ( from cvat.apps.engine.models import (
Job, StatusChoice, Task, Project, Review, Issue, Job, StatusChoice, Task, Project, Review, Issue,
Comment, StorageMethodChoice, ReviewStatus, StorageChoice Comment, StorageMethodChoice, ReviewStatus, StorageChoice, DimensionType, Image
) )
from cvat.apps.engine.serializers import ( from cvat.apps.engine.serializers import (
AboutSerializer, AnnotationFileSerializer, BasicUserSerializer, AboutSerializer, AnnotationFileSerializer, BasicUserSerializer,
@ -391,7 +393,7 @@ class TaskViewSet(auth.TaskGetQuerySetMixin, viewsets.ModelViewSet):
@swagger_auto_schema(method='get', operation_summary='Method returns data for a specific task', @swagger_auto_schema(method='get', operation_summary='Method returns data for a specific task',
manual_parameters=[ manual_parameters=[
openapi.Parameter('type', in_=openapi.IN_QUERY, required=True, type=openapi.TYPE_STRING, openapi.Parameter('type', in_=openapi.IN_QUERY, required=True, type=openapi.TYPE_STRING,
enum=['chunk', 'frame', 'preview'], enum=['chunk', 'frame', 'preview', 'context_image'],
description="Specifies the type of the requested data"), description="Specifies the type of the requested data"),
openapi.Parameter('quality', in_=openapi.IN_QUERY, required=True, type=openapi.TYPE_STRING, openapi.Parameter('quality', in_=openapi.IN_QUERY, required=True, type=openapi.TYPE_STRING,
enum=['compressed', 'original'], enum=['compressed', 'original'],
@ -430,7 +432,7 @@ class TaskViewSet(auth.TaskGetQuerySetMixin, viewsets.ModelViewSet):
data_id = request.query_params.get('number', None) data_id = request.query_params.get('number', None)
data_quality = request.query_params.get('quality', 'compressed') data_quality = request.query_params.get('quality', 'compressed')
possible_data_type_values = ('chunk', 'frame', 'preview') possible_data_type_values = ('chunk', 'frame', 'preview', 'context_image')
possible_quality_values = ('compressed', 'original') possible_quality_values = ('compressed', 'original')
try: try:
@ -475,6 +477,23 @@ class TaskViewSet(auth.TaskGetQuerySetMixin, viewsets.ModelViewSet):
elif data_type == 'preview': elif data_type == 'preview':
return sendfile(request, frame_provider.get_preview()) return sendfile(request, frame_provider.get_preview())
elif data_type == 'context_image':
if db_task.dimension == DimensionType.DIM_3D:
data_id = int(data_id)
image = Image.objects.get(data_id=db_task.data_id, frame=data_id)
for i in image.related_files.all():
path = os.path.realpath(str(i.path))
image = cv2.imread(path)
success, result = cv2.imencode('.JPEG', image)
if not success:
raise Exception("Failed to encode image to '%s' format" % (".jpeg"))
return HttpResponse(io.BytesIO(result.tobytes()), content_type="image/jpeg")
return Response(data='No context image related to the frame',
status=status.HTTP_404_NOT_FOUND)
else:
return Response(data='Only 3D tasks support context images',
status=status.HTTP_400_BAD_REQUEST)
else: else:
return Response(data='unknown data type {}.'.format(data_type), status=status.HTTP_400_BAD_REQUEST) return Response(data='unknown data type {}.'.format(data_type), status=status.HTTP_400_BAD_REQUEST)
except APIException as e: except APIException as e:

@ -22,6 +22,7 @@ module.exports = (stagedFiles) => {
const cvatData = containsInPath('/cvat-data/', eslintFiles); const cvatData = containsInPath('/cvat-data/', eslintFiles);
const cvatCore = containsInPath('/cvat-core/', eslintFiles); const cvatCore = containsInPath('/cvat-core/', eslintFiles);
const cvatCanvas = containsInPath('/cvat-canvas/', eslintFiles); const cvatCanvas = containsInPath('/cvat-canvas/', eslintFiles);
const cvatCanvas3d = containsInPath('/cvat-canvas3d/', eslintFiles);
const cvatUI = containsInPath('/cvat-ui/', eslintFiles); const cvatUI = containsInPath('/cvat-ui/', eslintFiles);
const mapping = {}; const mapping = {};
@ -31,6 +32,7 @@ module.exports = (stagedFiles) => {
mapping['npm run precommit:cvat-data -- '] = cvatData.join(' '); mapping['npm run precommit:cvat-data -- '] = cvatData.join(' ');
mapping['npm run precommit:cvat-core -- '] = cvatCore.join(' '); mapping['npm run precommit:cvat-core -- '] = cvatCore.join(' ');
mapping['npm run precommit:cvat-canvas -- '] = cvatCanvas.join(' '); mapping['npm run precommit:cvat-canvas -- '] = cvatCanvas.join(' ');
mapping['npm run precommit:cvat-canvas3d -- '] = cvatCanvas3d.join(' ');
for (const command of Object.keys(mapping)) { for (const command of Object.keys(mapping)) {
const files = mapping[command]; const files = mapping[command];

@ -49,6 +49,7 @@
"precommit:cvat-data": "cd cvat-ui && eslint --fix", "precommit:cvat-data": "cd cvat-ui && eslint --fix",
"precommit:cvat-core": "cd cvat-ui && eslint --fix", "precommit:cvat-core": "cd cvat-ui && eslint --fix",
"precommit:cvat-canvas": "cd cvat-ui && eslint --fix", "precommit:cvat-canvas": "cd cvat-ui && eslint --fix",
"precommit:cvat-canvas3d": "cd cvat-ui && eslint --fix",
"precommit:cvat-ui": "cd cvat-ui && eslint --fix" "precommit:cvat-ui": "cd cvat-ui && eslint --fix"
}, },
"repository": { "repository": {

Loading…
Cancel
Save