Server admin

Fast S3 storage.

A storage location can be an S3 bucket on AWS or on any S3-compatible service. Farwing writes each file straight into the bucket while it arrives, in many parts at once, and reads large files back in parallel ranges. Nothing is copied to the server's own disk on the way. Writing to a bucket is part of the Team plan and above.

How a file is written

Each incoming file becomes one multipart upload in the bucket. Farwing holds each part in memory until all of its bytes are in, then uploads it while the rest of the file is still arriving. Parts are sent several at a time, and a part that was slowed by lost packets does not hold up the parts after it.

Every part is checked by the bucket. Farwing sends a SHA-256 checksum with each part, and the bucket refuses a part that was damaged on the way.
The whole file is checked before it appears. Farwing compares the whole file with the sender's checksum, as it does on local disk. Only when it matches does Farwing finish the upload. Until then the object does not exist in the bucket, so nobody can see or download half a file.
A failed transfer leaves nothing behind. If the check fails or the transfer is cancelled, Farwing aborts the upload and the bucket discards its parts.
A busy bucket is waited out. When the bucket asks Farwing to slow down, Farwing retries after a growing, slightly random wait and sends fewer parts at once to that bucket for a while. A transfer does not fail because the bucket was busy.

A bucket allows at most 10,000 parts in one object. For a very large file Farwing raises the part size on its own so the file fits.

How a file is read

Large files are downloaded from the bucket as several byte ranges at once, ahead of what the recipient needs, and sent on in order. Reads share the same memory budget as writes, and get the same slow-down handling.

Keep the server near the bucket

Every part is a separate request to the bucket, so the distance between the Farwing server and the bucket sets the pace. Run the server in the same region as the bucket. A server on another continent spends most of each part waiting for replies, and a single file can then move far slower than the network allows.

Run the Speed check to see how fast your server reads and writes each bucket.

Performance settings

Each bucket has a Performance section under Admin → Storage → Locations. The defaults suit most buckets. Run the Speed check before and after a change.

Setting	Default	Range	What it does
Part size	64 MB	16–512 MB	How much of a file goes in each part. Larger parts mean fewer requests and more memory.
Parts at once, per file	16	1–256	How many parts of one file upload at the same time.
Ranges at once, per file	16	1–256	How many byte ranges of one file download at the same time.
Part checksums	SHA-256	SHA-256 or off	Turn off only for a service that refuses the checksum. The whole-file check still runs.

Farwing never renames your files to make them faster to store.

Memory

Parts wait in memory until they are uploaded, so the server sets aside one memory budget that every bucket shares. By default it is a quarter of the server's memory, or of the container's memory limit when that is lower, up to 4 GB.

When the budget is full, Farwing slows the incoming transfer through its normal flow control until parts have been uploaded. No data is dropped and the transfer does not fail. The live view then shows Limited by: S3 upload (memory buffer full).

As a rough guide, the memory a transfer needs is its speed multiplied by the time the bucket takes to accept one part. At 10 Gbit/s that is about 2 to 3 GB. To set the budget yourself, give the server this environment variable, in megabytes:

FARWING_S3_MEMORY_MB=2048

The Performance section shows how much memory one file can hold at once with your settings: the part size times the parts at once.

Clean up unfinished uploads

An upload that never finishes still keeps its parts in the bucket, and the provider charges for them even though they do not show in a listing. Farwing aborts every unfinished upload under the bucket's prefix that was started more than 7 days ago, and records each one in the server log. This includes uploads that another program started, so give Farwing its own bucket or prefix if other software writes large files to the same bucket.

As a second safety net, add a lifecycle rule to the bucket that does the same. The Performance section has a Copy rule button with this rule, already filled in with the bucket's prefix:

{
  "Rules": [
    {
      "ID": "farwing-abort-unfinished-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

To apply it with the AWS command line, save it as lifecycle.json and run this command with your bucket's name. It replaces the bucket's lifecycle rules, so add any rules the bucket already has to the same file first:

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-farwing-bucket \
  --lifecycle-configuration file://lifecycle.json

Other providers accept the same rule in their console or command line.

Virus scanning with a bucket

A scanner needs the whole file. When virus scanning applies to a file going to a bucket, Farwing writes it to a hidden staging area inside the same bucket, under .farwing/staging/, and the scanner reads it from there. A clean file is then copied to its real place by the bucket itself, part by part for a large file, so the bytes do not travel through the server again. An infected file never becomes a visible object.

When scanning does not apply, the file goes straight to its real place and appears once the whole-file check passes.

Received and stored

Every file has two moments:

Received: the file is complete, its checksum matches, it has been scanned where scanning applies, and it is visible and downloadable. The file.arrived event fires.
Stored: the file is in its final storage. The file.stored event fires.

On local disk and on a bucket written as this page describes, both happen at the same moment. Use file.stored for an event rule that deletes or moves the source of a file, so the source only goes once the file is safely kept.

What the live view shows

When the bucket is what holds a transfer back, the live view says so:

S3 upload (memory buffer full): parts are waiting for memory. Raise the budget, or check the bucket's speed.
S3 upload (bucket asked to slow down): the bucket is limiting requests. Farwing is waiting it out.
S3 upload backlog: parts are waiting to be uploaded.
S3 read: the bucket is not sending data as fast as the recipient can take it.