# Recommended Tools

STRG speaks the S3 API, so a great many tools built for S3 work with it straight away. This page
collects the ones we have found especially useful, grouped by the job you want to get done, each
with a short explanation and a working example you can adapt.

The list is growing, so it is worth checking back from time to time.

Every example here signs its requests with an [access key](https://docs.strg.com/access-keys) from the dashboard,
and reaches your bucket through the S3 endpoint of the bucket's region,
`https://s3.<region>.strg.com` - an *endpoint* being simply the address a tool sends its requests
to. [The S3 API overview](https://docs.strg.com/s3-api/overview#regions-and-endpoints) lists every region and
its endpoint. The examples use `eu-central-1` and a bucket called `my-bucket`.

## Data Migration

Moving data you already have - on a local disk, a server, or another storage service - into STRG.

- [rclone](https://docs.strg.com/recommended-tools#rclone)

### rclone

**What it is.** [rclone](https://rclone.org) is a free, open-source command-line program for
copying and synchronising files between storage systems. It knows how to talk to local folders,
SFTP servers and dozens of cloud storage services, and treats each of them as a *remote*: a named
connection you set up once and then refer to as `name:path`. To *synchronise* (sync) means making
one place an exact copy of another - copying what is new or changed, and removing what is no
longer there. rclone compares files by size and modification time - or, with `--checksum`, by
*checksum*, a short fingerprint calculated from a file's contents that changes whenever the
contents do - so a second run transfers only what changed.

**When it helps.** rclone is a good fit whenever existing data needs to come into STRG: moving
an archive of photos or documents off a local disk or a NAS, migrating a bucket from another
storage service in one command, or keeping a folder on a server mirrored into a bucket as a
nightly backup. Because an interrupted run can simply be started again and picks up where the
data differs, it also handles large migrations that take hours or days, and its
`--transfers` option lets it move many files in parallel.

**Using it with STRG.** rclone reaches STRG through a remote in its configuration file
(`rclone config file` prints its location). This one reads the access key from the
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables, so the secret never has to
be written into the file:

```ini title="rclone.conf"
[strg]
type = s3
provider = Other
env_auth = true
region = eu-central-1
endpoint = https://s3.eu-central-1.strg.com
force_path_style = true
no_check_bucket = true
```

`no_check_bucket = true` stops rclone from trying to create the bucket before it uploads, since
the bucket already exists in the dashboard. `region` and `endpoint` name the bucket's own region:
a bucket in another region is answered `NoSuchBucket`.

A key for a migration needs these permissions, and can be limited to the one bucket it fills:

| Permission | Label | Why rclone needs it |
| --- | --- | --- |
| `objects:list` | List objects | To see what is already in the bucket |
| `objects:read` | Read objects | To read an object's stored modification time when comparing |
| `objects:write` | Write objects | To upload, including large files in parts, and - for `sync` - to delete |

With the key in the environment, `rclone copy` copies a local folder into the bucket, leaving
anything already there alone:

```sh
export AWS_ACCESS_KEY_ID=<your access key ID>
export AWS_SECRET_ACCESS_KEY=<your secret access key>

rclone copy ./archive strg:my-bucket/archive --progress
```

`rclone sync` goes one step further and makes the destination match the source exactly, which
means it **deletes** objects in the bucket that the source does not have. `--dry-run` shows what
a run would copy and delete without changing anything, which makes it a good first step:

```sh
rclone sync ./archive strg:my-bucket/archive --dry-run
rclone sync ./archive strg:my-bucket/archive --progress
```

Moving from another storage service works the same way, with that service set up as a second
remote through `rclone config` - called `old` here - as the source:

```sh
rclone sync old:source-bucket strg:my-bucket --dry-run
```

**Note: Deleted objects keep their versions**

Every STRG bucket keeps previous versions, so an object `rclone sync` deletes stays stored, and
billed, as a previous version. A [lifecycle rule](https://docs.strg.com/lifecycle-policies) can clear previous
versions away after a number of days you choose.

## Data Ingestion

Getting files that keep arriving - from cameras, scanners, partners' systems or a website's
visitors - into a bucket, from senders that cannot speak S3 themselves.

- [Rilavek](https://docs.strg.com/recommended-tools#rilavek)

### Rilavek

**What it is.** [Rilavek](https://rilavek.com) is a managed file-transfer bridge: it receives
files over the protocols older devices and many business systems already use, and streams them
on into S3-compatible storage as they arrive, without keeping them on its own servers. Those
protocols are *FTP*, the classic file-transfer protocol, *FTPS* (FTP with encryption), *SFTP*
(file transfer over an encrypted SSH connection), and HTTP *resumable uploads* through the Tus
protocol - uploads that can carry on from where they stopped after a dropped connection, rather
than starting over. Each flow is a *Pipe*: one or more of those inputs on one side, and one or more
storage destinations, which Rilavek calls *Data Stores*, on the other.

**When it helps.** Rilavek fits wherever files need to land in a bucket but the sender only knows
how to upload to a server: a scanner, a factory machine or a partner's system that can only use
FTP; a digital camera sending photos straight from a shoot; IP cameras and CCTV recorders backing
up their footage. It also gives you an SFTP or FTP endpoint in front of your bucket without
running a server yourself, resumable uploads from a website's visitors (it works with the Uppy
upload widget), copies of each file sent to several destinations at once for backups, and
signed *webhooks* - a message Rilavek sends to an address of yours each time a file arrives, so
another system can start working on it straight away.

**Using it with STRG.** In Rilavek, your STRG bucket is a Data Store. The first step is an
[access key](https://docs.strg.com/access-keys) for Rilavek alone, limited to the one bucket it delivers to.
These are the permissions it needs:

| Permission | Label | Why Rilavek needs it |
| --- | --- | --- |
| `objects:write` | Write objects | To upload files, large ones in parts, and to clean up an upload that was cancelled |
| `objects:list` | List objects | To show FTP and SFTP clients a folder listing; uploads over HTTP work without it |

Rilavek has no need to read files back, so the key can leave *Read objects* out. Adding a Data
Store then works like this in Rilavek's dashboard:

1. **Data Stores** in the sidebar, then **New Data Store**.
2. The tile for generic S3-compatible storage.
3. The details of your bucket and key:

   | Rilavek's setting | Value |
   | --- | --- |
   | Name | Any name you will recognise when choosing it in a Pipe, such as `STRG my-bucket` |
   | Bucket Name | `my-bucket` |
   | Region | `eu-central-1` - the bucket's region |
   | Endpoint URL | `https://s3.eu-central-1.strg.com` - the endpoint of that region |
   | Access Key & Secret | The access key ID and secret access key of the key above |
   | Path | Optional - a folder in the bucket, such as `incoming/`, for every file this store receives |

4. **Save Data Store**. Rilavek checks right away that it can write to the bucket, without
   changing anything already there, and shows an error if a setting needs another look.

The Data Store can then be chosen as a destination of a Pipe, and every file a sender uploads to
that Pipe arrives in `my-bucket`. [Rilavek's documentation](https://rilavek.com/docs) walks
through Pipes and senders in detail.

**Tip: Bucket names without dots**

A bucket whose name contains a dot is reached through path-style addressing only
([Addressing](https://docs.strg.com/s3-api/overview#addressing) explains why), so for Rilavek a bucket name
without dots is the simplest choice.