From c378989ce1d87d54ce696f148f890873d1df897d Mon Sep 17 00:00:00 2001 From: Maya Date: Fri, 30 Oct 2020 20:12:15 +0300 Subject: [PATCH] Added guide how mount cloud storages as fuse --- cvat/apps/documentation/installation.md | 3 + .../documentation/mounting_cloud_storages.md | 306 ++++++++++++++++++ 2 files changed, 309 insertions(+) create mode 100644 cvat/apps/documentation/mounting_cloud_storages.md diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index ca536ea5..8abd316d 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -373,6 +373,9 @@ You can change the share device path to your actual share. For user convenience we have defined the environment variable \$CVAT_SHARE_URL. This variable contains a text (url for example) which is shown in the client-share browser. +You can [mount](/cvat/apps/documentation/mounting_cloud_storages.md) +your cloud storage as a FUSE and use it later as a share. + ### Email verification You can enable email verification for newly registered users. diff --git a/cvat/apps/documentation/mounting_cloud_storages.md b/cvat/apps/documentation/mounting_cloud_storages.md new file mode 100644 index 00000000..713ff2a7 --- /dev/null +++ b/cvat/apps/documentation/mounting_cloud_storages.md @@ -0,0 +1,306 @@ +- [Mounting cloud storage](#mounting-cloud-storage) + - [AWS S3 bucket](#aws-s3-bucket-as-filesystem) + - [Ubuntu 20.04](#ubuntu-20.04) + - [Mount](#mount) + - [Cheking](#cheking) + - [Automounting](#automounting) + - [Using /etc/fstab]('#using-fstab') + - [Using systemd](#using-systemd) + - [Unmount](#unmount-filesystem) + - [Azure container](#microsoft-azure-container-as-filesystem) + - [Ubuntu 20.04](#ubuntu-20.04) + - [Mount](#mount) + - [Cheking](#cheking) + - [Automounting](#automounting) + - [Using /etc/fstab]('#using-fstab') + - [Using systemd](#using-systemd) + - [Unmount](#unmount-filesystem) + - [Google Drive](#google-drive-as-filesystem) + - [Ubuntu 20.04](#ubuntu-20.04) + - [Mount](#mount) + - [Cheking](#cheking) + - [Automounting](#automounting) + - [Using /etc/fstab]('#using-fstab') + - [Using systemd](#using-systemd) + - [Unmount](#unmount-filesystem) + - +# Mounting cloud storage +## AWS S3 bucket as filesystem +### Ubuntu 20.04 +#### Mount + +1. Install s3fs +```bash +sudo apt install s3fs +``` + +2. Enter your credentials in a file `${HOME}/.passwd-s3fs` and set owner-only permissions: +```bash +echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs +chmod 600 ${HOME}/.passwd-s3fs +``` + +3. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` + +4. Run s3fs, replace `bucket_name`, `mount_point`: +```bash +s3fs -o allow_other +``` + +For more details see [here](https://github.com/s3fs-fuse/s3fs-fuse). + +#### Cheking +A file `/etc/mtab` contains records of currently mounted filesystems. +```bash +cat /etc/mtab | grep 's3fs' +``` + +#### Automounting +##### Using fstab +TODO +``` +cat /etc/mtab | grep s3fs >> /etc/fstab +doesn't work +``` +##### Using systemd + +1. Create unit file `sudo nano /etc/systemd/system/s3fs.service`. +Replace `user_name`, `bucket_name`, `mount_point`, `/path/to/.passwd-s3fs` + +``` +[Unit] +Description=FUSE filesystem over AWS S3 bucket +After=network.target + +[Service] +Environment=MOUNT_POINT= +User= +Group= +ExecStart=s3fs ${MOUNT_POINT} -o passwd_file=/path/to/.passwd-s3fs -o allow_other +ExecStop=fusermount -u ${MOUNT_POINT} +Restart=always +Type=forking + +[Install] +WantedBy=multi-user.target +``` + +2. Update the system configurations, enable unit autorun when the system boots. +```bash +sudo systemctl daemon-reload +sudo systemctl enable s3fs.service +``` + + +#### Unmount filesystem +```bash +fusermount -u +``` + +## Microsoft Azure container as filesystem +### Ubuntu 20.04 +#### Mount +1. Set up the Microsoft package repository.(More [here](https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software#configuring-the-repositories)) + +```bash +wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb +sudo dpkg -i packages-microsoft-prod.deb +sudo apt-get update +``` +2. Install `blobfuse` and `fuse` +```bash +sudo apt-get install blobfuse fuse +``` +For more details see [here](https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation) + +3. Create enviroments(replace `account_name`, `account_key`, `mount_point`) +```bash +export AZURE_STORAGE_ACCOUNT= +export AZURE_STORAGE_ACCESS_KEY= +MOUNT_POINT= +``` + +3. Create a folder for cache: +```bash +sudo mkdir -p /mnt/blobfusetmp +``` + +4. Make sure the file must be owned by the user who mounts the container: +```bash +sudo chown /mnt/blobfusetmp +``` + +5. Create the mount point, if it doesn't exists: +```bash +mkdir -p ${MOUNT_POINT} +``` + +6. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` + +7. Mount container(replace `your_container`): +```bash +blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp -o allow_other +``` + +#### Automounting + +##### Using fstab +1. Create configuration file `connection.cfg` with same content, change accountName, select one from accountKey or sasToken and replace with your value +``` +accountName +# Please provide either an account key or a SAS token, and delete the other line. +accountKey +#change authType to specify only 1 +sasToken +authType +containerName +``` + +2. create `mount.sh` with content below: +``` +#!/bin/bash +BLOBFS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +cd $BLOBFS_DIR/build +./blobfuse $1 --tmp-path=/mnt/blobfusetmp --use-attr-cache=true -o attr_timeout=240 +-o entry_timeout=240 -o negative_timeout=120 -o allow_other --config-file=../connection.cfg +``` + +4. Edit `/etc/fstab` with the blobfuse script. Add the following line: +`//mount.sh fuse _netdev,allow_other` + +##### Using systemd +1. Create unit file `sudo nano /etc/systemd/system/blobfuse.service`. +Replace `user_name`, `mount_point`, `container_name`,`/path/to/connection.cfg` + +``` +[Unit] +Description=FUSE filesystem over Azure container +After=network.target + +[Service] +Environment=MOUNT_POINT= +User= +Group= +ExecStart=blobfuse ${MOUNT_POINT} --container-name= --tmp-path=/mnt/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other +ExecStop=fusermount -u ${MOUNT_POINT} +Restart=always +Type=forking + +[Install] +WantedBy=multi-user.target +``` + +2. Update the system configurations, enable unit autorun when the system boots: +```bash +sudo systemctl daemon-reload +sudo systemctl enable blobfuse.service +``` + +Or for more detail [see here](https://github.com/Azure/azure-storage-fuse/tree/master/systemd) + +#### Unmount the filesystem +```bash +fusermount -u +``` + +If you have any mounting problems, check out the [answers](https://github.com/Azure/azure-storage-fuse/wiki/3.-Troubleshoot-FAQ) +to common problems + +## Google Drive as filesystem +### Ubuntu 20.04 +#### Mount +To mount a google drive as a filesystem in user space(FUSE) +you can use [google-drive-ocamlfuse](https://github.com/astrada/google-drive-ocamlfuse) +To do this follow the instructions below: +1. Install google-drive-ocamlfuse +``` +sudo add-apt-repository ppa:alessandro-strada/ppa +sudo apt-get update +sudo apt-get install google-drive-ocamlfuse +``` + +2. Run `google-drive-ocamlfuse` without parameters +``` +google-drive-ocamlfuse +``` +This command will create the default application directory (~/.gdfuse/default), containing the configuration file config +(see the [wiki](https://github.com/astrada/google-drive-ocamlfuse/wiki) page for more details about configuration). +And it will start a web browser to obtain authorization to access your Google Drive. +This will let you modify default configuration before mounting the filesystem. + +Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive). + +3. Create the mount point, if it doesn't exist(replace mount_point): +```bash +mountpoint="" +mkdir -p $mountpoint +``` + +4. Uncomment `user_allow_other` in the `/etc/fuse.conf` file: `sudo nano /etc/fuse.conf` + +5. Mount the filesystem +```bash +google-drive-ocamlfuse -o allow_other $mountpoint +``` + +#### Cheking +A file `/etc/mtab` contains records of currently mounted filesystems. +```bash +cat /etc/mtab | grep 'google-drive-ocamlfuse' +``` + +#### Automounting +##### Using fstab +1. Create a shell script named gdfuse in /usr/bin (as root) with this content: +```bash +#!/bin/bash + +su $USERNAME -l -c "google-drive-ocamlfuse -label $1 $*" +exit 0 +``` +2. Give it the exec permission: +```bash +sudo chmod +x /usr/bin/gdfuse +``` + +3. Edit `/etc/fstab` adding a line like this, replace `label`(dafualt `label=default`): +``` +gdfuse#