Digital Ocean 'Spaces' mounting and scripts

If you are using DO (Digital Ocean) Spaces feature then you are probably thinking of scripting or even mounting this "share" on your pc or even on Droplet itself.

First, I would like to mention that I recently switch this blog to Digital Ocean and pointed my domain registrar to DO DNS service which is accessible to you as soon as you open your DO account and you can use this DNS completely free. For account creation you need 5$ which later you can use to pay 1 VM (virtual machine) for 1 month. In your account later, you can create your referral links so anyone who registers their account using the link will get 10$ on their account and later if they spend 25$ you get also 25$ on your account. So in light of that, here is mine and you can use it to make your DO account and get your 10$ :)

In short, DO has one of the best interfaces for managing your VMs or, as they call them, Droplets. True, I haven't seen much of them, but I'm sure this is one of the best. You can ofcourse take snapshot of your Droplet for 1$/month. I believe in background are KVM or Xen hypervisors. But this post isn't about all the features on DO so I won't go into much details cause DO documentation is great on every subject you need. Now to mention storage side...

Besides local space on your Droplet you can create block device and mount them to you Droplet. Block device is called Volumes on DO, and basic price is 10$/month for 100GB. You can even go as low as 1$/month for 10G but then losing the point of storage space. Minimum local space on Droplet itself is 25G.

Next thing is Spaces. Spaces are an object storage implementation. Basically you can transfer you data using HTTP API. If you need more details here is great documentation on object and block storage ^[1]. And now this post is about 2 tools for using Spaces with your linux machine. One is s3cmd command line utility and the other is s3fs for mounting it.

s3cmd

s3cmd ^[2] is available on Debian repos but it's on version 1.6.1. So we will manually install it. If you find s4cmd on package list don't bother cause it doesn't work with DO Spaces s3 implementation.

First download latest version using wget:

wget https://github.com/s3tools/s3cmd/releases/download/v2.0.2/s3cmd-2.0.2.tar.gz

Then unpack it for example into /opt/s3cmd and create symlink in /usr/local/bin:

sudo ln -s /opt/s3cmd/s3cmd /usr/local/bin/s3cmd

Now on your DO account create your API access keys if you havent already done so. Here is the guide if needed ^[3].

After just run s3cmd --configure for initial configuration. It will create .s3cfg file in your home path. Most values stay at default except few of them. Here it is in short:

[default]
access_key = [your_access_key]
gpg_command = /usr/bin/gpg #or whereve it is on your system
# you can user ```date | md5sum``` to generate gpg
gpg_passphrase = 15165dd2e3b97af3ebd00e329730e9bc
# you can get endpoint address on your Spaces interfaces
host_base = nyc3.digitaloceanspaces.com 
# just add endpoint address after %(bucket)s
host_bucket = %(bucket)s.nyc3.digitaloceanspaces.com
secret_key = [your_secret_key]
use_https = True
...

Now you can list all your Spaces with:

s3cmd ls
2018-08-07 14:21  s3://space1

Or list all content on your Space:

s3cmd ls s3://space1
                       DIR   s3://space1/test/
                       DIR   s3://space1/test2/
2018-08-08 13:42   1537768   s3://space1/MegaCli-8.07.07-1.noarch.rpm
2018-08-08 10:15  12749576   s3://space1/teamviewer_13.1.3026_amd64.deb
2018-08-08 11:42 1073741824   s3://space1/testfile

For command reference please use s3cmd -h any you will get all the commands available for uploading, downloading etc.

This is a great tool to work with your Spaces on DO from command line cause it's easy to use in your scripts for example. Soon as I rewrite my backup script I will share it with you.

s3fs

If you wanna mount your DO Spaces, you can do it using s3fs ^[4].

Clone git repo:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git

And then build it all:

./autogen.sh 
./configure --prefix=/opt --with-openssl
sudo make
sudo make install

I installed it with prefix /opt so just link it's executable to get it into your path:

ln -s /opt/s3fs/bin/s3fs /usr/local/bin/

Now create in your home .passwd-s3fs file and input access key and secret key:

echo [your_access_key]:[your_secret_key] >  ~/.passwd-s3fs
chmod 600  ~/.passwd-s3fs

Then mount it through command line:

sudo s3fs space1 /mnt/space1 -ourl=https://nyc3.digitaloceanspaces.com -o passwd_file=~/.passwd-s3fs -o allow_other

Or using fstab:

s3fs#space1 /mnt/space1    fuse   _netdev,allow_other,passwd_file=/home/user1/.passwd-s3fs,url=https://nyc3.digitaloceanspaces.com/ 0 0

That's it. But do keep in mind that object storage is not ideal for every day r/w tasks. But, for example, it's good for backup. Not to repeat, as it is said in this article on object vs block storage ^[1:1]:

...on a filesystem, you can easily append a single line to the end of a log file. On an object storage system, you'd need to retrieve the object, add the new line, and write the entire object back...