Large files

#What this is for

Print files get large: a catalogue full of photos, a banner of several metres, a book. Seshat takes them in parts, so a large file never has to fit in one request, and an upload that stopped halfway goes on where it was.

For most apps there is nothing to do. The packages send a large file in parts by themselves. This page is for when you want to show how far an upload is, or resume one after a deploy.

#The limits

Every part goes straight to storage, and Seshat reads the file from there. A path is read from your disk one part at a time, so the file is never whole in memory, on your side or on Rekhu's.

#Large files by themselves

upload, check, enhance, fillTemplate and textPositions send a file larger than 20 MB in parts, wait for its safety check, and then work on it by its file_id. Your code stays the same:

$check = Rekhu::seshat()->for($user)->check('/data/catalogue.pdf', $order);

const check = await rekhu.seshat({ user }).check('/data/catalogue.pdf', { order });

Give a path rather than the bytes for a large file. The package then streams it from disk.

#Send a file in parts yourself

Call uploadInParts when you want the progress, for example to show a bar to the customer, or to keep the upload id for a resume.

1. Start the upload. Call uploadInParts() with an upload, a path, or the bytes with a name, and a progress function.
2. Follow the progress. The function gets the bytes sent, the total and the upload id: once before the first part, then after each part. Keep the upload id.
3. Wait for the safety check. When all parts are in, Rekhu checks the file for safety. The method waits for that check, polling every 2 seconds for up to 30 minutes.
4. Use the file_id. You get file_id, name, bytes and safety. Pass the file_id to check, fix, fillTemplate and the rest.

use Rekhu\Core\Rekhu;

$file = Rekhu::seshat()->uploadInParts('/data/catalogue.pdf', 'catalogue.pdf', function (int $sent, int $total, string $uploadId) {
    cache()->put('upload:'.auth()->id(), ['id' => $uploadId, 'done' => $sent / $total]);
});

$check = Rekhu::seshat()->check($file['file_id'], $order);

const seshat = rekhu.seshat({ user });

const file = await seshat.uploadInParts('/data/catalogue.pdf', {
    onProgress: (sent, total, uploadId) => progress.set(user.id, { uploadId, done: sent / total }),
});

const check = await seshat.check(file.file_id, { order });

A part that fails is sent again, up to three times. The Node.js package sends up to three parts at once; the Laravel package sends them one after the other.

To wait longer or poll less often, pass the time:

$file = Rekhu::seshat()->uploadInParts($path, progress: $progress, timeoutSeconds: 3600, everySeconds: 5);

const file = await seshat.uploadInParts(path, { onProgress, timeout: 3_600_000, every: 5000 });

Laravel takes seconds; Node.js takes milliseconds.

Note: uploading is free, also in parts. Only the work on the file (a check, a fix) is charged.

#Resume an upload that stopped

A deploy, a crash or a dropped connection can stop an upload halfway. Rekhu keeps the parts it has for 24 hours.

1. Keep the upload id. Your progress function gets it before the first part; store it with the order.
2. Call resumeUpload. Pass the upload id and the same file.
3. Only the missing parts go. Then the file is checked for safety, and you get the same answer as from uploadInParts.

$file = Rekhu::seshat()->resumeUpload($uploadId, '/data/catalogue.pdf');

const file = await seshat.resumeUpload(uploadId, '/data/catalogue.pdf');

The file must be the same size as the one the upload started with; another file is refused before anything is sent. resumeUpload takes the same progress function and times as uploadInParts.

When the safety check took longer than the wait, the method throws an error that says the file is still being checked. Call resumeUpload again later: it sends nothing and waits for the check.

#Drop an upload

When the customer cancels, drop the upload and the parts sent so far:

Rekhu::seshat()->abortUpload($uploadId);

await seshat.abortUpload(uploadId);

#Work on the file by its file_id

Every call that takes a file also takes a file_id. Upload once, then check, fix, preview and fill as often as you need within the day:

$seshat = Rekhu::seshat()->for($user);

$check = $seshat->check($file['file_id'], $order);
$fixed = $seshat->fix($file['file_id'], 'auto', $order);
$seshat->downloadTo($fixed['file_id'], storage_path('app/print/catalogue.pdf'));

const check = await seshat.check(file.file_id, { order });
const fixed = await seshat.fix(file.file_id, 'auto', order);
await seshat.downloadTo(fixed.file_id, 'storage/print/catalogue.pdf');

Use downloadTo for a large result: it writes the file straight to disk instead of holding it in memory.

Seshat checks and fixes files of up to 2 GB in one piece. A larger file is taken and kept, but a check or a fix on it is refused with a message that says so.

#An unsafe file

When the safety check refuses the file, uploadInParts and resumeUpload throw UnsafeFile. Its safety property holds the report, with why. A refused file is never kept. See File safety.

use Rekhu\Core\Exceptions\UnsafeFile;

try {
    $file = Rekhu::seshat()->uploadInParts($path);
} catch (UnsafeFile $e) {
    return back()->withErrors(['artwork' => $e->getMessage()]);
}

import { UnsafeFile } from '@rekhu/core';

try {
    await seshat.uploadInParts(path);
} catch (error) {
    if (error instanceof UnsafeFile) return res.status(422).json({ message: error.message, safety: error.safety });
    throw error;
}

#Next

• File safety: what the safety check looks for, and what to do with its verdict.
• Fixing files: repair the file once it is in.