#s3 - S3-compatible object storage

Import with import "s3.j" as s3;. An object-storage client for Amazon S3 and every S3-compatible store - the endpoint is configurable, so one module serves AWS S3, MinIO, Cloudflare R2, and Backblaze B2 (a selectable backend, not a module per vendor). Every request is signed with AWS Signature Version 4 (HMAC-SHA256 key-chaining), built on hash.hmac + hash.compute + encoding (hex) + time (the request timestamp) + http. Needs the default jennifer binary (net via http).

import "s3.j" as s3;
import "http.j" as http;

def c as s3.Client init s3.connect(
    "https://s3.us-east-1.amazonaws.com", "us-east-1", accessKey, secretKey);

def put as http.Response init s3.put($c, "mybucket", "hello.txt", "hi there");
def obj as http.Response init s3.get($c, "mybucket", "hello.txt");
io.printf("%d\n%s\n", $obj.status, $obj.body);

Runnable: examples/modules/s3_demo.j.

#Client

s3.connect(endpoint, region, accessKey, secretKey) returns a value-semantic s3.Client. The endpoint is any S3-compatible base URL (scheme://host, no trailing slash); addressing is path-style ({endpoint}/{bucket}/{key}), which works uniformly across AWS and self-hosted stores.

Every request carries a timeout so a hung S3 endpoint fails instead of blocking forever (the classic way a slow store exhausts a worker pool). connect defaults Client.timeout to 30 000 ms; set it to change the bound, or to 0 to disable it:

def c as s3.Client init s3.connect(endpoint, region, key, secret);
$c.timeout = 5000;   # fail a request that stalls for 5 s

#Operations

Every call returns an http.Response (status / headers / body); reading it needs import "http.j". A non-2xx status is a value to branch on, not an error.

(The list op is listObjects, not list, because list is a reserved type keyword.)

# Page through every object (S3 caps a listing at 1000 keys per page).
def body as string init s3.listObjects($c, "mybucket").body;
repeat {
    for (def k in s3.objectKeys($body)) {
        io.printf("%s\n", $k);
    }
    if (s3.isTruncated($body)) {
        $body = s3.listObjectsFrom($c, "mybucket", s3.nextContinuationToken($body)).body;
    }
} until (not s3.isTruncated($body));

#Presigned URLs

s3.presign(client, method, bucket, key, expiresSeconds) builds a URL that grants time-limited access to an object without exposing the secret key, using SigV4 query-signing (the X-Amz-* query parameters, UNSIGNED-PAYLOAD). The returned URL works from any HTTP client (a browser, curl) until it expires. This is pure string work - no request is sent - so it runs on both binaries.

# A link that lets anyone GET the object for one hour.
def url as string init s3.presign($c, "GET", "mybucket", "report.pdf", 3600);
# A PUT upload link (hand it to a browser to upload directly to S3).
def up as string init s3.presign($c, "PUT", "mybucket", "upload.bin", 900);

The maximum validity is 604800 seconds (7 days).

#Multipart upload

For objects beyond the ~5 GB single-PUT limit (or to stream parts as they are produced), upload in parts:

def uid as string init s3.createMultipartUpload($c, "mybucket", "big.iso", "application/octet-stream");
def etags as list of string init [];
$etags[] = s3.uploadPart($c, "mybucket", "big.iso", $uid, 1, $part1); # each part >= 5 MiB except the last
$etags[] = s3.uploadPart($c, "mybucket", "big.iso", $uid, 2, $part2);
def r as http.Response init s3.completeMultipartUpload($c, "mybucket", "big.iso", $uid, $etags);
# On any failure, discard the parts so they do not accrue storage cost:
#   s3.abortMultipartUpload($c, "mybucket", "big.iso", $uid);

uploadPart returns each part's ETag; pass them to completeMultipartUpload in part-number order (part 1 first).

#Signing

Requests are signed with SigV4 for service s3: the canonical request covers the method, the URI-encoded path (object keys keep their /), the canonical query (parameters sorted by key), the signed headers, and the SHA-256 of the body; the string-to-sign is HMAC-chained through AWS4<secret> -> date -> region -> s3 -> aws4_request to the signature. The base signed set is host / x-amz-content-sha256 / x-amz-date; putWith / putBytesWith add content-type and each x-amz-meta-* header to the signed set, and copy adds x-amz-copy-source, all sorted in. The payload hash is a real SHA-256 of the body (not UNSIGNED-PAYLOAD) for the request-signing path, so the whole request is integrity-covered; presigned URLs use UNSIGNED-PAYLOAD (the standard for query-signing). The signature is pinned in the tests against independent SigV4 implementations - the header-signing over the wire (TestS3Requests, which re-derives the signature from the actual signed-header set) and the presigned URL against a Python reference.

#Scope

• Path-style, SigV4, s3 service. Virtual-hosted addressing and the older SigV2 are out of scope.
• String and byte bodies. put / get carry text; putBytes / getBytes (and putBytesWith) carry raw bytes for binary objects. A single body is held in memory, so objects beyond ~5 GB use the multipart path.
• Object ops + multipart + presign. Get / put / delete / head / copy, content-type + metadata, multipart upload, and presigned URLs are covered. Bucket create / delete and ACL / policy management are not.
• listObjects returns the raw XML (plus objectKeys); pagination is supported through isTruncated / nextContinuationToken / listObjectsFrom, but full metadata parsing (size, etag, last-modified) is a follow-on.

#See also

• hash.md - the hmac / compute primitives SigV4 builds on.
• http.md - the client transport requests go through.
• webhook.md / totp.md - the other hash.hmac-based modules.
• modules/index.md - the module catalog and import rules.