diff --git a/cvat/apps/documentation/mounting_cloud_storages.md b/cvat/apps/documentation/mounting_cloud_storages.md index 6d3ab74e..f8d1de55 100644 --- a/cvat/apps/documentation/mounting_cloud_storages.md +++ b/cvat/apps/documentation/mounting_cloud_storages.md @@ -2,26 +2,26 @@ - [AWS S3 bucket](#aws-s3-bucket-as-filesystem) - [Ubuntu 20.04](#aws_s3_ubuntu_2004) - [Mount](#aws_s3_mount) - - [Automounting](#aws_s3_automounting) + - [Automatically mount](#aws_s3_automatically_mount) - [Using /etc/fstab](#aws_s3_using_fstab) - [Using systemd](#aws_s3_using_systemd) - - [Cheking](#aws_s3_cheking) + - [Check](#aws_s3_check) - [Unmount](#aws_s3_unmount_filesystem) - [Azure container](#microsoft-azure-container-as-filesystem) - [Ubuntu 20.04](#azure_ubuntu_2004) - [Mount](#azure_mount) - - [Automounting](#azure_automounting) - - [Using /etc/fstab]('#azure_using_fstab') + - [Automatically mount](#azure_automatically_mount) + - [Using /etc/fstab](#azure_using_fstab) - [Using systemd](#azure_using_systemd) - - [Cheking](#azure_cheking) + - [Check](#azure_check) - [Unmount](#azure_unmount_filesystem) - [Google Drive](#google-drive-as-filesystem) - [Ubuntu 20.04](#google_drive_ubuntu_2004) - [Mount](#google_drive_mount) - - [Automounting](#google_drive_automounting) - - [Using /etc/fstab]('#google_drive_using_fstab') + - [Automatically mount](#google_drive_automatically_mount) + - [Using /etc/fstab](#google_drive_using_fstab) - [Using systemd](#google_drive_using_systemd) - - [Cheking](#google_drive_cheking) + - [Check](#google_drive_check) - [Unmount](#google_drive_unmount_filesystem) # Mounting cloud storage @@ -29,86 +29,86 @@ ### Ubuntu 20.04 #### Mount -1. Install s3fs +1. Install s3fs: -```bash -sudo apt install s3fs -``` + ```bash + sudo apt install s3fs + ``` -2. Enter your credentials in a file `${HOME}/.passwd-s3fs` and set owner-only permissions: +1. Enter your credentials in a file `${HOME}/.passwd-s3fs` and set owner-only permissions: -```bash -echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs -chmod 600 ${HOME}/.passwd-s3fs -``` + ```bash + echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs + chmod 600 ${HOME}/.passwd-s3fs + ``` -3. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` -4. Run s3fs, replace `bucket_name`, `mount_point`: +1. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` +1. Run s3fs, replace `bucket_name`, `mount_point`: -```bash -s3fs -o allow_other -``` + ```bash + s3fs -o allow_other + ``` For more details see [here](https://github.com/s3fs-fuse/s3fs-fuse). -#### Automounting -Follow the 3 mounting steps above. +#### Automatically mount +Follow the first 3 mounting steps above. ##### Using fstab -1. Create a shell script named aws_s3_fuse(e.g in /usr/bin, as root) with this content -(replace `user_name` on whose behalf the disk will be mounted, `backet_name`, `mount_point`, `/path/to/.passwd-s3fs`): +1. Create a bash script named aws_s3_fuse(e.g in /usr/bin, as root) with this content + (replace `user_name` on whose behalf the disk will be mounted, `backet_name`, `mount_point`, `/path/to/.passwd-s3fs`): -```bash -#!/bin/bash -sudo -u s3fs -o passwd_file=/path/to/.passwd-s3fs -o allow_other -exit 0 -``` + ```bash + #!/bin/bash + sudo -u s3fs -o passwd_file=/path/to/.passwd-s3fs -o allow_other + exit 0 + ``` -2. Give it the exec permission: +1. Give it the execution permission: -```bash -sudo chmod +x /usr/bin/aws_s3_fuse -``` + ```bash + sudo chmod +x /usr/bin/aws_s3_fuse + ``` -3. Edit `/etc/fstab` adding a line like this, replace `mount_point`): +1. Edit `/etc/fstab` adding a line like this, replace `mount_point`): -``` -/absolute/path/to/aws_s3_fuse fuse allow_other,user,_netdev 0 0 -``` + ```bash + /absolute/path/to/aws_s3_fuse fuse allow_other,user,_netdev 0 0 + ``` ##### Using systemd -1. Create unit file `sudo nano /etc/systemd/system/s3fs.service`. -Replace `user_name`, `bucket_name`, `mount_point`, `/path/to/.passwd-s3fs` +1. Create unit file `sudo nano /etc/systemd/system/s3fs.service` + (replace `user_name`, `bucket_name`, `mount_point`, `/path/to/.passwd-s3fs`): -``` -[Unit] -Description=FUSE filesystem over AWS S3 bucket -After=network.target - -[Service] -Environment="MOUNT_POINT=" -User= -Group= -ExecStart=s3fs ${MOUNT_POINT} -o passwd_file=/path/to/.passwd-s3fs -o allow_other -ExecStop=fusermount -u ${MOUNT_POINT} -Restart=always -Type=forking - -[Install] -WantedBy=multi-user.target -``` + ```bash + [Unit] + Description=FUSE filesystem over AWS S3 bucket + After=network.target -2. Update the system configurations, enable unit autorun when the system boots. + [Service] + Environment="MOUNT_POINT=" + User= + Group= + ExecStart=s3fs ${MOUNT_POINT} -o passwd_file=/path/to/.passwd-s3fs -o allow_other + ExecStop=fusermount -u ${MOUNT_POINT} + Restart=always + Type=forking -```bash -sudo systemctl daemon-reload -sudo systemctl enable s3fs.service -sudo systemctl start s3fs.service -``` + [Install] + WantedBy=multi-user.target + ``` + +1. Update the system configurations, enable unit autorun when the system boots, mount the bucket: -#### Cheking + ```bash + sudo systemctl daemon-reload + sudo systemctl enable s3fs.service + sudo systemctl start s3fs.service + ``` + +#### Check A file `/etc/mtab` contains records of currently mounted filesystems. ```bash cat /etc/mtab | grep 's3fs' @@ -119,122 +119,130 @@ cat /etc/mtab | grep 's3fs' fusermount -u ``` -## Microsoft Azure container as filesystem -### Ubuntu 20.04 -#### Mount -1. Set up the Microsoft package repository.(More [here](https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software#configuring-the-repositories)) +If you used [systemd](#aws_s3_using_systemd) to mount a bucket: ```bash -wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -sudo dpkg -i packages-microsoft-prod.deb -sudo apt-get update +sudo systemctl stop s3fs.service +sudo systemctl disable s3fs.service ``` -2. Install `blobfuse` and `fuse` - -```bash -sudo apt-get install blobfuse fuse -``` -For more details see [here](https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation) +## Microsoft Azure container as filesystem +### Ubuntu 20.04 +#### Mount +1. Set up the Microsoft package repository.(More [here](https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software#configuring-the-repositories)) -3. Create enviroments(replace `account_name`, `account_key`, `mount_point`) + ```bash + wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb + sudo dpkg -i packages-microsoft-prod.deb + sudo apt-get update + ``` -```bash -export AZURE_STORAGE_ACCOUNT= -export AZURE_STORAGE_ACCESS_KEY= -MOUNT_POINT= -``` +1. Install `blobfuse` and `fuse`: -4. Create a folder for cache: -```bash -sudo mkdir -p /mnt/blobfusetmp -``` + ```bash + sudo apt-get install blobfuse fuse + ``` + For more details see [here](https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation) -5. Make sure the file must be owned by the user who mounts the container: -```bash -sudo chown /mnt/blobfusetmp -``` +1. Create enviroments(replace `account_name`, `account_key`, `mount_point`): -6. Create the mount point, if it doesn't exists: -```bash -mkdir -p ${MOUNT_POINT} -``` - -7. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` -8. Mount container(replace `your_container`): - -```bash -blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp -o allow_other -``` + ```bash + export AZURE_STORAGE_ACCOUNT= + export AZURE_STORAGE_ACCESS_KEY= + MOUNT_POINT= + ``` -#### Automounting/a<> -Follow the 3 mounting steps above. -##### Using fstab +1. Create a folder for cache: + ```bash + sudo mkdir -p /mnt/blobfusetmp + ``` -1. Create configuration file `connection.cfg` with same content, change accountName, -select one from accountKey or sasToken and replace with your value +1. Make sure the file must be owned by the user who mounts the container: + ```bash + sudo chown /mnt/blobfusetmp + ``` -``` -accountName -# Please provide either an account key or a SAS token, and delete the other line. -accountKey -#change authType to specify only 1 -sasToken -authType -containerName -``` +1. Create the mount point, if it doesn't exists: + ```bash + mkdir -p ${MOUNT_POINT} + ``` -2. create `azure_fuse`(e.g in /usr/bin, as root) with content below -(replace `user_name` on whose behalf the disk will be mounted, `mount_point`, `/path/to/blobfusetmp`,`/path/to/connection.cfg`): +1. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` +1. Mount container(replace `your_container`): -``` -#!/bin/bash -sudo -u blobfuse --tmp-path=/path/to/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other -exit 0 + ```bash + blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp -o allow_other + ``` -``` +#### Automatically mount +Follow the first 7 mounting steps above. +##### Using fstab -3. Give it the exec permission: +1. Create configuration file `connection.cfg` with same content, change accountName, + select one from accountKey or sasToken and replace with your value: + + ```bash + accountName + # Please provide either an account key or a SAS token, and delete the other line. + accountKey + #change authType to specify only 1 + sasToken + authType + containerName + ``` + +1. Create a bash script named `azure_fuse`(e.g in /usr/bin, as root) with content below + (replace `user_name` on whose behalf the disk will be mounted, `mount_point`, `/path/to/blobfusetmp`,`/path/to/connection.cfg`): + + ```bash + #!/bin/bash + sudo -u blobfuse --tmp-path=/path/to/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other + exit 0 + ``` + +1. Give it the execution permission: + ```bash + sudo chmod +x /usr/bin/azure_fuse + ``` + +1. Edit `/etc/fstab` with the blobfuse script. Add the following line(replace paths): ```bash -sudo chmod +x /usr/bin/azure_fuse +/absolute/path/to/azure_fuse fuse allow_other,user,_netdev ``` -4. Edit `/etc/fstab` with the blobfuse script. Add the following line(replace paths): -`/absolute/path/to/mount.sh fuse _netdev,allow_other` - ##### Using systemd -1. Create unit file `sudo nano /etc/systemd/system/blobfuse.service`. -Replace `user_name`, `mount_point`, `container_name`,`/path/to/connection.cfg` - -``` -[Unit] -Description=FUSE filesystem over Azure container -After=network.target - -[Service] -Environment="MOUNT_POINT=" -User= -Group= -ExecStart=blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other -ExecStop=fusermount -u ${MOUNT_POINT} -Restart=always -Type=forking - -[Install] -WantedBy=multi-user.target -``` - -2. Update the system configurations, enable unit autorun when the system boots: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable blobfuse.service -sudo systemctl blobfuse s3fs.service -``` -Or for more detail [see here](https://github.com/Azure/azure-storage-fuse/tree/master/systemd) - -#### Cheking +1. Create unit file `sudo nano /etc/systemd/system/blobfuse.service`. + (replace `user_name`, `mount_point`, `container_name`,`/path/to/connection.cfg`): + + ```bash + [Unit] + Description=FUSE filesystem over Azure container + After=network.target + + [Service] + Environment="MOUNT_POINT=" + User= + Group= + ExecStart=blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other + ExecStop=fusermount -u ${MOUNT_POINT} + Restart=always + Type=forking + + [Install] + WantedBy=multi-user.target + ``` + +1. Update the system configurations, enable unit autorun when the system boots, mount the container: + + ```bash + sudo systemctl daemon-reload + sudo systemctl enable blobfuse.service + sudo systemctl start blobfuse.service + ``` + Or for more detail [see here](https://github.com/Azure/azure-storage-fuse/tree/master/systemd) + +#### Check A file `/etc/mtab` contains records of currently mounted filesystems. ```bash cat /etc/mtab | grep 'blobfuse' @@ -245,6 +253,13 @@ cat /etc/mtab | grep 'blobfuse' fusermount -u ``` +If you used [systemd](#azure_using_systemd) to mount a container: + +```bash +sudo systemctl stop blobfuse.service +sudo systemctl disable blobfuse.service +``` + If you have any mounting problems, check out the [answers](https://github.com/Azure/azure-storage-fuse/wiki/3.-Troubleshoot-FAQ) to common problems @@ -255,99 +270,103 @@ To mount a google drive as a filesystem in user space(FUSE) you can use [google-drive-ocamlfuse](https://github.com/astrada/google-drive-ocamlfuse) To do this follow the instructions below: -1. Install google-drive-ocamlfuse +1. Install google-drive-ocamlfuse: -``` -sudo add-apt-repository ppa:alessandro-strada/ppa -sudo apt-get update -sudo apt-get install google-drive-ocamlfuse -``` + ```bash + sudo add-apt-repository ppa:alessandro-strada/ppa + sudo apt-get update + sudo apt-get install google-drive-ocamlfuse + ``` -2. Run `google-drive-ocamlfuse` without parameters +1. Run `google-drive-ocamlfuse` without parameters: -```bash -google-drive-ocamlfuse -``` -This command will create the default application directory (~/.gdfuse/default), containing the configuration file config -(see the [wiki](https://github.com/astrada/google-drive-ocamlfuse/wiki) page for more details about configuration). -And it will start a web browser to obtain authorization to access your Google Drive. -This will let you modify default configuration before mounting the filesystem. + ```bash + google-drive-ocamlfuse + ``` -Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive). + This command will create the default application directory (~/.gdfuse/default), + containing the configuration file config (see the [wiki](https://github.com/astrada/google-drive-ocamlfuse/wiki) + page for more details about configuration). + And it will start a web browser to obtain authorization to access your Google Drive. + This will let you modify default configuration before mounting the filesystem. -3. Create the mount point, if it doesn't exist(replace mount_point): + Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive). -```bash -mountpoint="" -mkdir -p $mountpoint -``` +1. Create the mount point, if it doesn't exist(replace mount_point): -4. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` + ```bash + mountpoint="" + mkdir -p $mountpoint + ``` -5. Mount the filesystem -```bash -google-drive-ocamlfuse -o allow_other $mountpoint -``` +1. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` +1. Mount the filesystem: + + ```bash + google-drive-ocamlfuse -o allow_other $mountpoint + ``` -#### Automounting -Follow the 3 mounting steps above. +#### Automatically mount +Follow the first 4 mounting steps above. ##### Using fstab -1. Create a shell script named gdfuse(e.g in /usr/bin, as root) with this content -(replace `user_name` on whose behalf the disk will be mounted, `label`, `mount_point`): +1. Create a bash script named gdfuse(e.g in /usr/bin, as root) with this content + (replace `user_name` on whose behalf the disk will be mounted, `label`, `mount_point`): -```bash -#!/bin/bash -sudo -u google-drive-ocamlfuse -o allow_other -label