Recommended Tools

STRG speaks the S3 API, so a great many tools built for S3 work with it straight away. This page collects the ones we have found especially useful, grouped by the job you want to get done, each with a short explanation and a working example you can adapt.

The list is growing, so it is worth checking back from time to time.

Every example here signs its requests with an access key from the dashboard, and reaches your bucket through the S3 endpoint of the bucket's region, https://s3.<region>.strg.com - an endpoint being simply the address a tool sends its requests to. The S3 API overview lists every region and its endpoint. The examples use eu-central-1 and a bucket called my-bucket.

Data Migration

Moving data you already have - on a local disk, a server, or another storage service - into STRG.

• rclone

rclone

What it is. rclone is a free, open-source command-line program for copying and synchronising files between storage systems. It knows how to talk to local folders, SFTP servers and dozens of cloud storage services, and treats each of them as a remote: a named connection you set up once and then refer to as name:path. To synchronise (sync) means making one place an exact copy of another - copying what is new or changed, and removing what is no longer there. rclone compares files by size and modification time - or, with --checksum, by checksum, a short fingerprint calculated from a file's contents that changes whenever the contents do - so a second run transfers only what changed.

When it helps. rclone is a good fit whenever existing data needs to come into STRG: moving an archive of photos or documents off a local disk or a NAS, migrating a bucket from another storage service in one command, or keeping a folder on a server mirrored into a bucket as a nightly backup. Because an interrupted run can simply be started again and picks up where the data differs, it also handles large migrations that take hours or days, and its --transfers option lets it move many files in parallel.

Using it with STRG. rclone reaches STRG through a remote in its configuration file (rclone config file prints its location). This one reads the access key from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, so the secret never has to be written into the file:

rclone.conf

[strg]
type = s3
provider = Other
env_auth = true
region = eu-central-1
endpoint = https://s3.eu-central-1.strg.com
force_path_style = true
no_check_bucket = true

no_check_bucket = true stops rclone from trying to create the bucket before it uploads, since the bucket already exists in the dashboard. region and endpoint name the bucket's own region: a bucket in another region is answered NoSuchBucket.

A key for a migration needs these permissions, and can be limited to the one bucket it fills:

Permission	Label	Why rclone needs it
objects:list	List objects	To see what is already in the bucket
objects:read	Read objects	To read an object's stored modification time when comparing
objects:write	Write objects	To upload, including large files in parts, and - for sync - to delete

With the key in the environment, rclone copy copies a local folder into the bucket, leaving anything already there alone:

export AWS_ACCESS_KEY_ID=<your access key ID>
export AWS_SECRET_ACCESS_KEY=<your secret access key>

rclone copy ./archive strg:my-bucket/archive --progress

rclone sync goes one step further and makes the destination match the source exactly, which means it deletes objects in the bucket that the source does not have. --dry-run shows what a run would copy and delete without changing anything, which makes it a good first step:

rclone sync ./archive strg:my-bucket/archive --dry-run
rclone sync ./archive strg:my-bucket/archive --progress

Moving from another storage service works the same way, with that service set up as a second remote through rclone config - called old here - as the source:

rclone sync old:source-bucket strg:my-bucket --dry-run

Deleted objects keep their versions

Every STRG bucket keeps previous versions, so an object rclone sync deletes stays stored, and billed, as a previous version. A lifecycle rule can clear previous versions away after a number of days you choose.

Data Ingestion

Getting files that keep arriving - from cameras, scanners, partners' systems or a website's visitors - into a bucket, from senders that cannot speak S3 themselves.

• Rilavek

Rilavek

What it is. Rilavek is a managed file-transfer bridge: it receives files over the protocols older devices and many business systems already use, and streams them on into S3-compatible storage as they arrive, without keeping them on its own servers. Those protocols are FTP, the classic file-transfer protocol, FTPS (FTP with encryption), SFTP (file transfer over an encrypted SSH connection), and HTTP resumable uploads through the Tus protocol - uploads that can carry on from where they stopped after a dropped connection, rather than starting over. Each flow is a Pipe: one or more of those inputs on one side, and one or more storage destinations, which Rilavek calls Data Stores, on the other.

When it helps. Rilavek fits wherever files need to land in a bucket but the sender only knows how to upload to a server: a scanner, a factory machine or a partner's system that can only use FTP; a digital camera sending photos straight from a shoot; IP cameras and CCTV recorders backing up their footage. It also gives you an SFTP or FTP endpoint in front of your bucket without running a server yourself, resumable uploads from a website's visitors (it works with the Uppy upload widget), copies of each file sent to several destinations at once for backups, and signed webhooks - a message Rilavek sends to an address of yours each time a file arrives, so another system can start working on it straight away.

Using it with STRG. In Rilavek, your STRG bucket is a Data Store. The first step is an access key for Rilavek alone, limited to the one bucket it delivers to. These are the permissions it needs:

Permission	Label	Why Rilavek needs it
objects:write	Write objects	To upload files, large ones in parts, and to clean up an upload that was cancelled
objects:list	List objects	To show FTP and SFTP clients a folder listing; uploads over HTTP work without it

Rilavek has no need to read files back, so the key can leave Read objects out. Adding a Data Store then works like this in Rilavek's dashboard:

1. Data Stores in the sidebar, then New Data Store.

2. The tile for generic S3-compatible storage.

3. The details of your bucket and key:

   Rilavek's setting	Value
   Name	Any name you will recognise when choosing it in a Pipe, such as STRG my-bucket
   Bucket Name	my-bucket
   Region	eu-central-1 - the bucket's region
   Endpoint URL	https://s3.eu-central-1.strg.com - the endpoint of that region
   Access Key & Secret	The access key ID and secret access key of the key above
   Path	Optional - a folder in the bucket, such as incoming/, for every file this store receives

4. Save Data Store. Rilavek checks right away that it can write to the bucket, without changing anything already there, and shows an error if a setting needs another look.

The Data Store can then be chosen as a destination of a Pipe, and every file a sender uploads to that Pipe arrives in my-bucket. Rilavek's documentation walks through Pipes and senders in detail.

Bucket names without dots

A bucket whose name contains a dot is reached through path-style addressing only (Addressing explains why), so for Rilavek a bucket name without dots is the simplest choice.