diff --git a/cvat/apps/documentation/mounting_cloud_storages.md b/cvat/apps/documentation/mounting_cloud_storages.md index 713ff2a7..6d3ab74e 100644 --- a/cvat/apps/documentation/mounting_cloud_storages.md +++ b/cvat/apps/documentation/mounting_cloud_storages.md @@ -1,68 +1,83 @@ - [Mounting cloud storage](#mounting-cloud-storage) - [AWS S3 bucket](#aws-s3-bucket-as-filesystem) - - [Ubuntu 20.04](#ubuntu-20.04) - - [Mount](#mount) - - [Cheking](#cheking) - - [Automounting](#automounting) - - [Using /etc/fstab]('#using-fstab') - - [Using systemd](#using-systemd) - - [Unmount](#unmount-filesystem) + - [Ubuntu 20.04](#aws_s3_ubuntu_2004) + - [Mount](#aws_s3_mount) + - [Automounting](#aws_s3_automounting) + - [Using /etc/fstab](#aws_s3_using_fstab) + - [Using systemd](#aws_s3_using_systemd) + - [Cheking](#aws_s3_cheking) + - [Unmount](#aws_s3_unmount_filesystem) - [Azure container](#microsoft-azure-container-as-filesystem) - - [Ubuntu 20.04](#ubuntu-20.04) - - [Mount](#mount) - - [Cheking](#cheking) - - [Automounting](#automounting) - - [Using /etc/fstab]('#using-fstab') - - [Using systemd](#using-systemd) - - [Unmount](#unmount-filesystem) + - [Ubuntu 20.04](#azure_ubuntu_2004) + - [Mount](#azure_mount) + - [Automounting](#azure_automounting) + - [Using /etc/fstab]('#azure_using_fstab') + - [Using systemd](#azure_using_systemd) + - [Cheking](#azure_cheking) + - [Unmount](#azure_unmount_filesystem) - [Google Drive](#google-drive-as-filesystem) - - [Ubuntu 20.04](#ubuntu-20.04) - - [Mount](#mount) - - [Cheking](#cheking) - - [Automounting](#automounting) - - [Using /etc/fstab]('#using-fstab') - - [Using systemd](#using-systemd) - - [Unmount](#unmount-filesystem) - - + - [Ubuntu 20.04](#google_drive_ubuntu_2004) + - [Mount](#google_drive_mount) + - [Automounting](#google_drive_automounting) + - [Using /etc/fstab]('#google_drive_using_fstab') + - [Using systemd](#google_drive_using_systemd) + - [Cheking](#google_drive_cheking) + - [Unmount](#google_drive_unmount_filesystem) + # Mounting cloud storage ## AWS S3 bucket as filesystem -### Ubuntu 20.04 -#### Mount +### Ubuntu 20.04 +#### Mount 1. Install s3fs + ```bash sudo apt install s3fs ``` 2. Enter your credentials in a file `${HOME}/.passwd-s3fs` and set owner-only permissions: + ```bash echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs chmod 600 ${HOME}/.passwd-s3fs ``` 3. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` - 4. Run s3fs, replace `bucket_name`, `mount_point`: + ```bash s3fs -o allow_other ``` For more details see [here](https://github.com/s3fs-fuse/s3fs-fuse). -#### Cheking -A file `/etc/mtab` contains records of currently mounted filesystems. +#### Automounting +Follow the 3 mounting steps above. + +##### Using fstab + +1. Create a shell script named aws_s3_fuse(e.g in /usr/bin, as root) with this content +(replace `user_name` on whose behalf the disk will be mounted, `backet_name`, `mount_point`, `/path/to/.passwd-s3fs`): + ```bash -cat /etc/mtab | grep 's3fs' +#!/bin/bash +sudo -u s3fs -o passwd_file=/path/to/.passwd-s3fs -o allow_other +exit 0 ``` -#### Automounting -##### Using fstab -TODO +2. Give it the exec permission: + +```bash +sudo chmod +x /usr/bin/aws_s3_fuse ``` -cat /etc/mtab | grep s3fs >> /etc/fstab -doesn't work + +3. Edit `/etc/fstab` adding a line like this, replace `mount_point`): + +``` +/absolute/path/to/aws_s3_fuse fuse allow_other,user,_netdev 0 0 ``` -##### Using systemd + +##### Using systemd 1. Create unit file `sudo nano /etc/systemd/system/s3fs.service`. Replace `user_name`, `bucket_name`, `mount_point`, `/path/to/.passwd-s3fs` @@ -73,7 +88,7 @@ Description=FUSE filesystem over AWS S3 bucket After=network.target [Service] -Environment=MOUNT_POINT= +Environment="MOUNT_POINT=" User= Group= ExecStart=s3fs ${MOUNT_POINT} -o passwd_file=/path/to/.passwd-s3fs -o allow_other @@ -86,20 +101,27 @@ WantedBy=multi-user.target ``` 2. Update the system configurations, enable unit autorun when the system boots. + ```bash sudo systemctl daemon-reload sudo systemctl enable s3fs.service +sudo systemctl start s3fs.service ``` +#### Cheking +A file `/etc/mtab` contains records of currently mounted filesystems. +```bash +cat /etc/mtab | grep 's3fs' +``` -#### Unmount filesystem +#### Unmount filesystem ```bash fusermount -u ``` ## Microsoft Azure container as filesystem -### Ubuntu 20.04 -#### Mount +### Ubuntu 20.04 +#### Mount 1. Set up the Microsoft package repository.(More [here](https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software#configuring-the-repositories)) ```bash @@ -107,45 +129,51 @@ wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod. sudo dpkg -i packages-microsoft-prod.deb sudo apt-get update ``` + 2. Install `blobfuse` and `fuse` + ```bash sudo apt-get install blobfuse fuse ``` For more details see [here](https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation) 3. Create enviroments(replace `account_name`, `account_key`, `mount_point`) + ```bash export AZURE_STORAGE_ACCOUNT= export AZURE_STORAGE_ACCESS_KEY= MOUNT_POINT= ``` -3. Create a folder for cache: +4. Create a folder for cache: ```bash sudo mkdir -p /mnt/blobfusetmp ``` -4. Make sure the file must be owned by the user who mounts the container: +5. Make sure the file must be owned by the user who mounts the container: ```bash sudo chown /mnt/blobfusetmp ``` -5. Create the mount point, if it doesn't exists: +6. Create the mount point, if it doesn't exists: ```bash mkdir -p ${MOUNT_POINT} ``` -6. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` +7. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` +8. Mount container(replace `your_container`): -7. Mount container(replace `your_container`): ```bash blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp -o allow_other ``` -#### Automounting +#### Automounting/a<> +Follow the 3 mounting steps above. +##### Using fstab + +1. Create configuration file `connection.cfg` with same content, change accountName, +select one from accountKey or sasToken and replace with your value -##### Using fstab -1. Create configuration file `connection.cfg` with same content, change accountName, select one from accountKey or sasToken and replace with your value ``` accountName # Please provide either an account key or a SAS token, and delete the other line. @@ -156,19 +184,26 @@ authType containerName ``` -2. create `mount.sh` with content below: +2. create `azure_fuse`(e.g in /usr/bin, as root) with content below +(replace `user_name` on whose behalf the disk will be mounted, `mount_point`, `/path/to/blobfusetmp`,`/path/to/connection.cfg`): + ``` #!/bin/bash -BLOBFS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -cd $BLOBFS_DIR/build -./blobfuse $1 --tmp-path=/mnt/blobfusetmp --use-attr-cache=true -o attr_timeout=240 --o entry_timeout=240 -o negative_timeout=120 -o allow_other --config-file=../connection.cfg +sudo -u blobfuse --tmp-path=/path/to/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other +exit 0 + ``` -4. Edit `/etc/fstab` with the blobfuse script. Add the following line: -`//mount.sh fuse _netdev,allow_other` +3. Give it the exec permission: +```bash +sudo chmod +x /usr/bin/azure_fuse +``` + +4. Edit `/etc/fstab` with the blobfuse script. Add the following line(replace paths): +`/absolute/path/to/mount.sh fuse _netdev,allow_other` + +##### Using systemd -##### Using systemd 1. Create unit file `sudo nano /etc/systemd/system/blobfuse.service`. Replace `user_name`, `mount_point`, `container_name`,`/path/to/connection.cfg` @@ -178,7 +213,7 @@ Description=FUSE filesystem over Azure container After=network.target [Service] -Environment=MOUNT_POINT= +Environment="MOUNT_POINT=" User= Group= ExecStart=blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other @@ -191,14 +226,21 @@ WantedBy=multi-user.target ``` 2. Update the system configurations, enable unit autorun when the system boots: + ```bash sudo systemctl daemon-reload sudo systemctl enable blobfuse.service +sudo systemctl blobfuse s3fs.service ``` - Or for more detail [see here](https://github.com/Azure/azure-storage-fuse/tree/master/systemd) -#### Unmount the filesystem +#### Cheking +A file `/etc/mtab` contains records of currently mounted filesystems. +```bash +cat /etc/mtab | grep 'blobfuse' +``` + +#### Unmount filesystem ```bash fusermount -u ``` @@ -207,12 +249,14 @@ If you have any mounting problems, check out the [answers](https://github.com/Az to common problems ## Google Drive as filesystem -### Ubuntu 20.04 -#### Mount +### Ubuntu 20.04 +#### Mount To mount a google drive as a filesystem in user space(FUSE) you can use [google-drive-ocamlfuse](https://github.com/astrada/google-drive-ocamlfuse) To do this follow the instructions below: + 1. Install google-drive-ocamlfuse + ``` sudo add-apt-repository ppa:alessandro-strada/ppa sudo apt-get update @@ -220,7 +264,8 @@ sudo apt-get install google-drive-ocamlfuse ``` 2. Run `google-drive-ocamlfuse` without parameters -``` + +```bash google-drive-ocamlfuse ``` This command will create the default application directory (~/.gdfuse/default), containing the configuration file config @@ -231,6 +276,7 @@ This will let you modify default configuration before mounting the filesystem. Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive). 3. Create the mount point, if it doesn't exist(replace mount_point): + ```bash mountpoint="" mkdir -p $mountpoint @@ -243,34 +289,33 @@ mkdir -p $mountpoint google-drive-ocamlfuse -o allow_other $mountpoint ``` -#### Cheking -A file `/etc/mtab` contains records of currently mounted filesystems. -```bash -cat /etc/mtab | grep 'google-drive-ocamlfuse' -``` +#### Automounting +Follow the 3 mounting steps above. +##### Using fstab + +1. Create a shell script named gdfuse(e.g in /usr/bin, as root) with this content +(replace `user_name` on whose behalf the disk will be mounted, `label`, `mount_point`): -#### Automounting -##### Using fstab -1. Create a shell script named gdfuse in /usr/bin (as root) with this content: ```bash #!/bin/bash - -su $USERNAME -l -c "google-drive-ocamlfuse -label $1 $*" +sudo -u google-drive-ocamlfuse -o allow_other -label