From 5b46b516dd3657a0a238a83e285162a5d614b8c7 Mon Sep 17 00:00:00 2001 From: Andrey Zhavoronkov Date: Thu, 18 Mar 2021 15:02:28 +0300 Subject: [PATCH] Added backup/restore guide (#2964) --- CHANGELOG.md | 1 + cvat/apps/documentation/backup_guide.md | 76 +++++++++++++++++++++++++ cvat/apps/documentation/faq.md | 7 ++- docker-compose.yml | 2 - 4 files changed, 82 insertions(+), 4 deletions(-) create mode 100644 cvat/apps/documentation/backup_guide.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b0d4730..e9101a04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Kubernetes templates and guide for their deployment () - [WiderFace](http://shuoyang1213.me/WIDERFACE/) format support () - [VGGFace2](https://github.com/ox-vgg/vgg_face2) format support () +- [Backup/Restore guide](cvat/apps/documentation/backup_guide.md) () ### Changed diff --git a/cvat/apps/documentation/backup_guide.md b/cvat/apps/documentation/backup_guide.md new file mode 100644 index 00000000..e2664c61 --- /dev/null +++ b/cvat/apps/documentation/backup_guide.md @@ -0,0 +1,76 @@ +## About CVAT data volumes + +Docker volumes are used to store all CVAT data: + +- `cvat_db`: PostgreSQL database files, used to store information about users, tasks, projects, annotations, etc. + Mounted into `cvat_db` container by `/var/lib/postgresql/data` path. + +- `cvat_data`: used to store uploaded and prepared media data. + Mounted into `cvat` container by `/home/django/data` path. + +- `cvat_keys`: used to store user ssh keys needed for [synchronization with a remote Git repository](user_guide.md#task-synchronization-with-a-repository). + Mounted into `cvat` container by `/home/django/keys` path. + +- `cvat_logs`: used to store logs of CVAT backend processes managed by supevisord. + Mounted into `cvat` container by `/home/django/logs` path. + +- `cvat_events`: this is an optional volume that is used only when [Analytics component](../../../components/analytics) + is enabled and is used to store Elasticsearch database files. + Mounted into `cvat_elasticsearch` container by `/usr/share/elasticsearch/data` path. + +## How to backup all CVAT data + +All CVAT containers should be stopped before backup: + +```console +docker-compose stop +``` + +Please don't forget to include all the compose config files that were used in the docker-compose command +using the `-f` parameter. + +Backup data: + +```console +mkdir backup +docker run --rm --name temp_backup --volumes-from cvat_db -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_db.tar.bz2 /var/lib/postgresql/data +docker run --rm --name temp_backup --volumes-from cvat -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_data.tar.bz2 /home/django/data +# [optional] +docker run --rm --name temp_backup --volumes-from cvat_elasticsearch -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_events.tar.bz2 /usr/share/elasticsearch/data +``` + +Make sure the backup archives have been created, the output of `ls backup` command should look like this: + +```console +ls backup +cvat_data.tar.bz2 cvat_db.tar.bz2 cvat_events.tar.bz2 +``` + +## How to restore CVAT from backup + +Note: CVAT containers must exist (if no, please follow the [installation guide](installation.md#quick-installation-guide)). +Stop all CVAT containers: + +```console +docker-compose stop +``` + +Restore data: + +```console +cd +docker run --rm --name temp_backup --volumes-from cvat_db -v $(pwd):/backup ubuntu bash -c "cd /var/lib/postgresql/data && tar -xvf /backup/cvat_db.tar.bz2 --strip 4" +docker run --rm --name temp_backup --volumes-from cvat -v $(pwd):/backup ubuntu bash -c "cd /home/django/data && tar -xvf /backup/cvat_data.tar.bz2 --strip 3" +# [optional] +docker run --rm --name temp_backup --volumes-from cvat_elasticsearch -v $(pwd):/backup ubuntu bash -c "cd /usr/share/elasticsearch/data && tar -xvf /backup/cvat_events.tar.bz2 --strip 4" +``` + +After that run CVAT as usual: + +```console +docker-compose up -d +``` + +## Additional resources + +[Docker guide about volume backups](https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes) diff --git a/cvat/apps/documentation/faq.md b/cvat/apps/documentation/faq.md index 7e897c52..16cfd620 100644 --- a/cvat/apps/documentation/faq.md +++ b/cvat/apps/documentation/faq.md @@ -13,12 +13,12 @@ - [How to upload annotations to an entire task from UI when there are multiple jobs in the task](#how-to-upload-annotations-to-an-entire-task-from-ui-when-there-are-multiple-jobs-in-the-task) - [How to specify multiple hostnames for CVAT_HOST](#how-to-specify-multiple-hostnames-for-cvat_host) - [How to create a task with multiple jobs](#how-to-create-a-task-with-multiple-jobs) +- [How to transfer CVAT to another machine](#how-to-transfer-cvat-to-another-machine) ## How to update CVAT -Before upgrading, please follow the official docker -[manual](https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes) and backup all CVAT volumes. +Before upgrading, please follow the [backup guide](backup_guide.md) and backup all CVAT volumes. To update CVAT, you should clone or download the new version of CVAT and rebuild the CVAT docker images as usual. @@ -149,3 +149,6 @@ services: Set the segment size when you create a new task, this option is available in the [Advanced configuration](user_guide.md#advanced-configuration) section. + +## How to transfer CVAT to another machine +Follow the [backup/restore guide](backup_guide.md#how-to-backup-all-cvat-data). diff --git a/docker-compose.yml b/docker-compose.yml index daa74ee9..8eee1393 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -46,7 +46,6 @@ services: - cvat_data:/home/django/data - cvat_keys:/home/django/keys - cvat_logs:/home/django/logs - - cvat_models:/home/django/models cvat_ui: container_name: cvat_ui @@ -87,4 +86,3 @@ volumes: cvat_data: cvat_keys: cvat_logs: - cvat_models: