Downloads, library and watchlist

This page is about keeping things rather than just reading them: putting video and images on your own disk, finding them again months later, and collecting the same target on a timer. After reading it you will know what each of the three pages costs, where the bytes actually go, what the size ceiling will take away from you, and how to stop it taking the one post you meant to keep.

The three pages at a glance

The console splits “what I have seen” from “what I am holding” from “what I keep collecting”, and they are separate for a reason: the first is free, the second fills a disk, and the third spends the identity pool forever.

Page	Path	What it holds	What it costs
Library	/library	The archive: one row per post this instance has ever parsed — title, author, tags, the media manifest	Nothing. It answers from Postgres; no identity is spent and nothing is fetched
Downloads	/downloads	The download index: one row per request to store a post’s media, plus what landed on disk	Disk, and one identity-backed request per post that has to be fetched first
Watchlist	/watchlist	Standing instructions: collect this author or this post every N hours, forever	The identity pool, on a schedule, for as long as the entry exists

A fourth page, Transcriptions (/transcriptions), holds the text made from stored videos when the optional transcriber is on; it costs CPU on this machine and nothing upstream. See Turning a stored video into text.

All of them live in the Content group in the sidebar. A post can be in the archive without its media being on disk (the usual case), and a download can outlive its archive row — the downloads table still describes a real directory on the volume even if the post was later deleted from the library.

The media downloader is optional

Storing bytes is done by a separate container, the downloader sidecar. It is an opt-in Compose profile, and an instance without it is correctly configured, not broken — the archive, the API, the console and the watchlist all work unchanged. Only the byte-fetching half is missing.

Start it, from the repository root:

echo 'DTK_DOWNLOADER_URL=http://downloader:9100' >> .env
docker compose -p dtk -f docker/compose.yml --profile downloader up -d --build

Without DTK_DOWNLOADER_URL, POST /api/v1/downloads answers 501 with NOT_CONFIGURED and the message “this instance has no media downloader; start the downloader compose profile and set DTK_DOWNLOADER_URL”. The Downloads page shows the same thing as a card at the top and keeps working as a read-only index of whatever was stored earlier.

The sidecar is the container that dials arbitrary CDN hosts, so it is deliberately the emptiest one in the stack: a scratch image holding one static Go binary, no shell, no package manager, a read-only root filesystem, all capabilities dropped, no-new-privileges, and exactly one writable path. It is on the edge network only — it has no route to Postgres or Redis — and Compose blanks DTK_SECRET_KEY, DTK_DATABASE_URL and DTK_REDIS_URL for it even though the shared env file carries them. It is never handed a cookie, a proxy, an API key or a database address; it receives a list of mirrors, the host allowlist for that one platform, and a byte ceiling.

Its own limits, all read from .env:

Variable	Default	What it does
DTK_DOWNLOADER_URL	(empty)	Read by api and worker. Empty disables media downloads entirely
DTK_DOWNLOADER_TOKEN	(empty)	Shared secret. Empty means the sidecar checks nothing; it listens on an internal network no other container is on, so this is defence in depth rather than the boundary
DTK_DOWNLOADER_WORKERS	4	Jobs pulled in parallel
DTK_DOWNLOADER_ITEM_WORKERS	4	Files pulled in parallel within one job
DTK_DOWNLOADER_QUEUE	64	Jobs it will accept before answering “the download queue is full”
DTK_DOWNLOADER_TIMEOUT_SECONDS	900	Ceiling on one transfer
DTK_DOWNLOADER_HISTORY	500	Finished jobs kept in memory for progress reporting
DTK_DOWNLOADER_MAX_REDIRECTS	5	Redirect hops a mirror may take

The ceiling on concurrent transfers is the product of the two worker counts — 16 with the defaults. Raise them for a fast connection; they cost goroutines and a 32 KiB buffer each, not much else. The container is capped at 512 MB of memory and 2 CPUs because transfers are streamed, so resident memory is a handful of buffers rather than the size of the files.

The job history is in memory and is lost on restart. That has one visible consequence: after restarting the sidecar, live progress bars for jobs it has forgotten disappear, and the row draws without one. The durable record is written when a job settles, so nothing is lost except the bar.

Where the files land

Files go to the named Docker volume media-data, mounted at /var/lib/dtk/media. The downloader mounts it read-write and is the only thing that writes to it; the api container mounts it read-only, which is how the console can hand you a stored file without the downloader ever becoming a relay. The path inside the containers is fixed — if you want the media somewhere else on the host, change the volume definition in docker/compose.yml, not a setting.

The layout is one directory per post:

/var/lib/dtk/media/<platform>/<author_uid>/<content_id>/
├── video.mp4        # or .mov, .webm, .m4v
├── cover.jpg        # one, never two
├── image-01.jpg     # image posts: one file per slide
├── image-01.mp4     # a live photo's clip, beside its still
├── image-02.jpg
└── meta.json

meta.json is written beside the files and is the reason a folder found two years later still says what it is. It carries the platform, post id, kind, web_url, title, description, publish time, duration, author uid and nickname, music, tags, location and the file list. It deliberately carries no signed CDN URLs — those expire within hours and would be the one misleading part of it.

What the planner decides before anything is fetched:

• One video per post. Every bitrate the platform offers is treated as a mirror of the same file — the first that lands wins. Storing four copies of one clip is not a feature. The parser’s order is kept, and it puts the watermark-free stream first. A Bilibili video is one progressive MP4 rather than separate audio and video tracks, so it is one file too; a video in several parts is one post, and the file is the first part unless you ask for another (see One part of a multi-part video). Measured with a guest identity, that file is up to 1080p on older uploads and 720p on a current one.
• A live photo keeps its clip. A Douyin slide that is a live photo carries a few seconds of MP4 of its own, and it is stored right after its still under the same name — image-01.mp4 beside image-01.webp. An album can mix them with ordinary stills, which have no clip. The clip is not the post’s video: the library does not play it as the post and the transcriber does not read it. An album archived before 5.1.6 has no clips in its stored manifest, so its first download after the upgrade re-parses the post first, however fresh its links are.
• One cover, added last, and only if the post has not already hit the item cap.
• At most 128 files per post. A Douyin image album tops out around 35 slides, and with every one a live photo that is 71 files with the cover; the cap guards against a malformed payload becoming a thousand-item job.
• Images are capped at 32 MiB each, or media.max_file_bytes if that is lower. A 500 MB “image” is a sign something is wrong. A live photo’s clip is held to the same cap: it measured about 0.6 MB at 720p.
• Extensions are taken from the URL only when recognised — mp4, mov, webm, m4v for video and jpg, jpeg, png, webp, heic, gif for images — otherwise the kind’s default is used. A query string ending in .php cannot name a file on your disk.
• Every mirror is checked against the media host allowlist before a job exists, and anything dropped is reported back as a skipped reason.

A transfer in flight is written to <name>.part and renamed into place when it completes, so a name only ever appears when the file behind it is whole. A .part counts toward the volume total while it exists.

Starting a download

The Downloads page has a mode tab row at the top. Each mode takes a different shape of input, which is why they cannot share one submit button — a post link and a profile link are different things, and a form that guessed would quietly download the wrong one.

Mode	Takes	Result
A post	One share link, the whole share caption around one, or a bare post id	The post’s media on the server’s volume
Many posts	A paste of many links	A parse result per link, which can then be saved in one go
Comments	A post link or id	A JSON file in your browser — comments are data, not media
An author’s posts	A profile link or a bare author id	One queued download per post from their feed

The Skip what is already downloaded checkbox is shared by every mode that can act on more than one thing (all but Comments). It hands back the copy this instance already has instead of fetching the post again — which is what re-running a feed wants: the author added three posts and the other forty are here already. An evicted copy does not count; there are no bytes left to skip.

One post

Paste a link or a post id. Digits, and a Bilibili BV… or av… id, are treated as an id; anything else as a link. The platform menu beside the box is only consulted for bare digits and is disabled while a link is in the box, because the link says which platform it is — and if you send both and they disagree, the API refuses rather than guessing. A BV… or av… id names its own platform, so it goes to Bilibili and the menu shows that; plain digits with Bilibili picked are read as an av number and converted to the bvid.

Three things can happen, and the toast tells you which:

• Started. The post was already in the archive, the plan was built from the stored mirrors, and a job is queued. The response is 202 with a download_id and the task_id running it.
• Fetching the post first. This instance has never seen the post. The worker fetches it through the same pool, scheduler and request log as any other read, then downloads what comes back. Slower first response; you do not have to parse it yourself first.
• Already downloading / already downloaded. The request was answered with an existing download (reused: "in_flight" or "already_stored", HTTP 200). Joining an in-flight download is never optional, whatever skip_existing says: two downloads of one post write into the same directory and the loser of that race renames a file the winner has already moved, and comes back partial with a file missing.

Short links (v.douyin.com/..., vm.tiktok.com/..., b23.tv/...) are refused here, by design: resolving one means following it, and this endpoint makes no network calls. Send them to POST /api/v1/parse — which expands them in the background — and the post is archived by the time you come back. The Many posts mode accepts short links directly, because it goes through the parse pipeline.

If the post is archived and has nothing downloadable, you get a 400 with the reasons now, rather than a task that fails a minute later.

One part of a multi-part video

A Bilibili upload can come in several parts (分P). A plain download stores the first part, the one the post’s link plays. To store another, pass part (counted from 1) to POST /api/v1/downloads, or paste the link a player is on — ?p=3 in a Bilibili link means part 3 and is read the same way. part: 1 is the post’s own download, whichever way it is spelt. Asking for a part past the last one fails with the number of parts the video has. Douyin and TikTok posts have no parts, and part there is a 400.

On the Downloads page, a Bilibili post in the A post mode gets a Part menu under the box. It starts on the first part, or on the one a pasted link’s ?p= names, and List the parts fills it with the rest, by title and length — on request rather than as you type, because listing costs a real request through the pool and most videos have one part. A part’s row and drawer are marked P3.

Each part is its own download, with its own download_id, stored in a directory beside the post’s: bilibili/<mid>/<bvid>-p3/. Beside, not inside, because removing a download removes its directory, and a part nested in the post’s own would go whenever the first part’s files did. Two parts of one post can download at the same time; a second request for the same part joins the one in flight, as for any post. Saved to your computer, a part’s files are named for it (dtk-bilibili-<bvid>-p3-video.mp4), so parts saved to one folder do not overwrite each other.

The Library shows what the post’s link plays, so it counts the first part’s download as the post’s stored media and not a later part’s. A transcription follows the download, so transcribing part 3 is transcribing part 3’s download. GET /api/v1/bilibili/video/parts lists the parts with their numbers, titles and lengths (API).

Many posts (batch parse and download)

This mode replaces the one-line form with a textarea, its own results table and its own toolbar. Paste up to 500 links; they are submitted to the parse pipeline in chunks of 50.

The input you actually have is the thing this is built around. A Douyin share caption looks like 2.84 nqe:/ <title> https://v.douyin.com/L4FJNR3/ <sentence> — one link surrounded by prose. Splitting that on whitespace produces four rows, three of them nonsense. Instead, as you type (after a 300 ms pause) the paste is sent to POST /api/v1/tools/parse-batch, which extracts URLs using the same host allowlist the rest of the system uses and reports what it found. That endpoint touches no network and spends no identity; it accepts up to 512 KiB of text and returns at most 1000 items, saying truncated when it dropped some.

Each line comes back as one of:

Kind	Submittable	Meaning
link	yes	A recognised Douyin, TikTok or Bilibili URL
short_link	yes	A share link that needs following; the parse pipeline expands it
content_id	no	A bare post id with no platform attached to it — or a bare Bilibili bvid, which names its platform but is still not a link; this mode submits links only
bad_id	no	Looks like a post id but cannot be one — Douyin and TikTok mint ids whose high bits are the second they were issued
unknown	no	No Douyin, TikTok or Bilibili link in this line

The rejected kinds are answered locally, without spending a task, and appear in the table as failed rows with a reason.

Submission is one request per chunk; tracking is per row. Each row then lives its own life — its own state, its own error, its own retry — because a synchronous bulk endpoint would smear one dead link into a single error for the whole paste. Rows are polled every 1.5 seconds, 8 at a time, round-robin so the tail of a long paste is not starved.

The results table columns are: state, link, title, author, platform, kind, content id, plays, likes, and the error code. Comments, shares and published time are available but hidden by default — use the table’s column chooser. Clicking a finished row opens a drawer with the video player, the images, the counters, per-file download buttons and the raw result payload. A live photo plays its clip in place of the still, and gets a download button of its own for it.

The toolbar:

• Save N to this instance — makes exactly the same request the single-post mode makes, once per row, so these rows land in the archive with the same progress, retry, dedupe and skip-what-is-already-here behaviour as everything else. One failure does not abandon the rest; the toast reports how many were queued, how many were already here and how many could not be queued.
• Export CSV — one row per parse result, saved as dtk-parse-<timestamp>.csv. The columns are url, state, error_code, platform, kind, content_id, web_url, title, author, author_uid, created_at, duration_ms, play_count, digg_count, comment_count, share_count, collect_count, video_url. Cells that contain a comma, a quote or a newline are quoted, and a leading =, +, - or @ is prefixed with an apostrophe so a spreadsheet cannot execute it.
• Export JSON — the same rows with the full result payload, as dtk-parse-<timestamp>.json.
• Retry failed (N) — retries only the rows whose error code is not permanent. A dead link is not worth another identity.
• Clear — empties the table. Nothing on the server is touched; these rows are browser state and do not survive a reload.

The drawer’s own download buttons are a different thing from “Save to this instance”: they fetch from the platform CDN straight to your browser, so the file lands in your downloads folder and the server never relays media. Mirrors are tried in order. Some CDNs refuse cross-origin reads, and when all of them do, the first mirror is opened in a new tab and the toast says so — that is CORS or hotlink protection, not a dead file.

An author’s recent posts

Paste a profile link, or a bare author id: the sec_user_id the library and playground hand you on Douyin, the secUid on TikTok (both start MS4wLjABAAAA), or the numeric mid on Bilibili. A TikTok profile link names the @handle, which the post list does not take, so the page reads the profile once to get the secUid. The How many posts menu offers the newest 20, 50, 100, 200, or everything the feed will give up. It is a menu rather than a number field on purpose: every post is a download, every page of the feed is a read through the identity pool, and a free-text depth invites “99999” typed without a thought. Choosing everything asks first.

The page walks the feed 20 posts at a time until it has enough, stopping at 40 pages, then queues one download per post, and reports both under the form as it goes: the page it is reading, then how many are queued, already here and failed, with the first failure’s reason. Stop ends the run between any two requests. Failures on individual posts do not abandon the rest.

Douyin pages an author’s posts past the first only for a signed-in identity: without one, page 2 answers IDENTITY_LOGIN_REQUIRED. The run keeps what the earlier pages gave, queues those, and says that later pages need a signed-in identity, with a link to the signed-in identities of that platform. A first page a guest answered may also leave out the author’s newest posts; the report says so when that happened.

One trap worth knowing: a bare author id does not say which platform it belongs to — MS4wLjABAAAA… looks the same on Douyin and TikTok, and a mid is plain digits — so it is looked up on whichever platform the menu shows. The menu is enabled for exactly that case and disabled while a profile link is in the box, because the link carries the platform.

For deeper history than the feed will hand over in one pass, use Archive their history in the library’s post drawer, which is a different job with a different cost.

A post’s comments

Reads up to 5 pages of comments, 50 to a page where the platform allows it (35 on TikTok; Bilibili sets its own page size), through the ordinary read endpoint and hands your browser a JSON file named comments-<platform>-<id>.json. Each page is a real request through the identity pool. Comments are data rather than media, so nothing is written to the server’s volume and the downloader is not involved.

Bilibili shows a guest only the three hottest comments and no second page. When a guest answered, or a later page needed a signed-in identity, the comments read so far are still saved, and the page says why there are not more.

The downloads table

Column	What it shows
Post	The stored cover as a thumbnail, the archive’s title, and the author and platform beneath. Falls back to the content id when the archive knows nothing about the post
State	queued, running, done, partial, failed or cancelled, plus a live progress bar while it is running
Platform	douyin, tiktok or bilibili
Files	How many files completed
Size	Bytes on disk, or Cleaned up for a row whose files were removed — by hand or by the ceiling
Finished	When it settled
Keep	The pin toggle
Retry	Present only on a settled row that is worth retrying

partial is its own state rather than a success or a failure: a post whose video landed and whose third image 404’d is neither, and calling it done would hide the gap while calling it failed would hide a video that is on the disk.

Filtering by state and by platform happens on the server, so “the most recent TikTok downloads” means the most recent of all of them and not the most recent of whatever page happened to load. Sorting, by contrast, is done in the browser over the rows already loaded — the default is Finished, newest first. The table shows 100 rows a page, with Previous and Next and where the page sits in the whole (101-200 of 340), and the search box above it matches the post’s title or author as the archive holds them, or its id (q on GET /api/v1/downloads). The filters, the search and the open drawer are kept in the address, so /downloads?id=<download id> opens one download from anywhere.

The table polls every 2 seconds while anything is in flight and every 10 seconds otherwise. The progress bar is read live from the sidecar for at most 8 in-flight rows at a time. It shows the bytes written so far against the download’s size, the percentage, the speed over the last few seconds and the time left — for example 38.2 MB / 96.7 MB · 39% over 1.4 MB/s · 42 s left. The size is what each file’s server declared in Content-Length. The sidecar never trusts that number for the ceiling, which is enforced on bytes actually written, but it is good enough to draw a bar against. A file that has not started yet has declared nothing, so until every file of a download has begun — the later images of an album, say — the bar counts files instead, with the bytes so far beside it.

Clicking a row opens a drawer, titled by the post, with the player, the per-file list, each file’s SHA-256, and a save link per completed file. It links to the post in the Library when the archive has it, and when the video already has a transcript, Transcribe video becomes Open the transcript. The drawer follows the row as the table refreshes: opened on a running download, it shows each file’s own progress and speed, and the stored files once it lands.

The progress object

Over the API the bar is progress, on every row of GET /api/v1/downloads and on GET /api/v1/downloads/{download_id}. It is read live from the sidecar and never stored, so it is null on a settled row, while the downloader is not configured or not answering, for a job the sidecar has forgotten (it keeps a bounded history and loses it on a restart), and on in-flight rows past the first 8 of a page. A null is “no bar”, never “no progress”.

Field	Type	What it is
bytes	integer	Bytes written so far, summed over the files
bytes_expected	integer or null	What the whole download will weigh: a finished file’s real size, a running file’s declared Content-Length, nothing for a failed file. null while any file has not started or its server declared no length — count files then, as the console does
bytes_per_second	integer	Speed over the last few seconds, summed over the files that are running
files_done, files_total	integer	Files finished, and files in the job
state	string	The sidecar’s own state for the job: queued, running, done, partial, failed or cancelled. It can settle a moment before the row does
items	array	One entry per file, in the job’s order: name, kind (video, image or cover), state (queued, running, done or failed), bytes, bytes_expected (null when the server declared none) and bytes_per_second (0 unless it is running)

bytes_expected is for display only: the size ceiling is enforced in the sidecar on bytes actually written, never on what a server declared.

Retrying, and why there is no resume across attempts

Retry is offered on failed, partial and cancelled rows. It calls POST /api/v1/downloads/retry with the download_id and starts the same post over from scratch, re-parsing the post first if the stored mirrors have gone stale (older than media.mirror_max_age_seconds, 600 seconds by default).

There is no resume across attempts, and there could not usefully be one. The sidecar writes to <name>.part and truncates it on every attempt, because media URLs are signed and expire: bytes fetched an hour ago cannot be continued against a link that now answers 403. Asking again is what works.

Within one transfer it is different. Some CDN edges end a transfer early: measured on Bilibili in October 2026, an Akamai edge serving a video it had not cached yet stopped after exactly 2 MiB with “unexpected EOF”, every time, while the same request a minute later came back whole. So when a mirror stops short of the length it declared, the sidecar asks the same mirror for the rest — a Range request from the byte it stopped at — up to three times, seconds later, against the same link. It only accepts a 206 covering exactly the rest of the same file (If-Range carries the first answer’s validator), so a file replaced in between is refused rather than spliced on. An edge that answers the request for the rest with a server error — measured, a 504 while it was still pulling the file from its origin — is asked again after one, two and four seconds. The downloader log’s item stored line says how many times a transfer had to be resumed.

Retry is refused while a download is still running — a second attempt would race the first into the same directory. A download that gets stuck is failed by the maintenance pass two hours after it was created (the pass runs every five minutes), after which the row is retryable. Nothing on disk is touched by that: files that did land stay where they are.

Pinning and the size ceiling

This is the part that deletes things, so it is worth reading before you need it.

Stored media is capped by media.max_bytes, 2 GiB by default. When the volume goes over it, the maintenance pass removes stored downloads oldest-first until it is back under. What that pass does, precisely:

• The total comes from the volume, measured by the sidecar, not from the database. Files removed by hand or restored from a backup are therefore accounted for; a sweep that trusted its own bookkeeping would delete to satisfy a number nothing else agrees with.
• Pinned downloads are skipped, always. This is the only exemption there is.
• Only the files go. The row keeps its file list, sizes and digests and reports on_disk: false, so the console can still tell you what was collected and that it was cleaned up. That distinction matters: “collected and later evicted” can be undone by asking again; “never fetched” cannot be undone by anything.
• Every sweep that removes something raises a media_evicted alert (throttled to one every six hours). It is the only notice you will get that your disk policy just ran — see Operations for wiring up notification channels.
• If everything above the ceiling is pinned, nothing is removed and a warning is logged saying there is nothing evictable.
• media.max_bytes = 0 disables eviction entirely. That is a choice you are allowed to make; it means nothing is ever cleaned up.

Pinning is done from the Keep column, or in bulk after selecting rows. It exempts the whole download from the sweep for good. Pin anything you would be upset to lose, particularly a post the platform has since taken down — that one cannot be fetched again.

Separately from the media ceiling, a capacity guard watches actual disk usage on the volumes this instance can see. Past capacity.warn_percent (80) it raises a warning; past capacity.hard_stop_percent (92) new download jobs are refused with Retry-After: 300 and scheduled collection stands down, while every read carries on. Nothing is deleted by the guard. A full disk should degrade an instance, not take it offline.

The four tiles at the top of the page report: bytes stored (against the ceiling, turning amber at 75% and red at 90%), total downloads and how many are in flight, how many are pinned, and how many have been cleaned up.

Duplicates, cancelling, and what actually deletes bytes

Duplicate downloads. Re-downloading a post is normal — the first attempt failed, the mirrors went stale, the file was evicted — and each attempt leaves a row. The successful ones leave a second complete copy. The Duplicate downloads card runs a dry run first and shows the count in a confirmation dialog, because the count is the whole decision. Confirming keeps the newest copy of each post that still has its files and removes the rest. A directory is only deleted when no surviving row still uses it, so in the ordinary case where two attempts wrote to the same place, the records go and the file stays — which is correct, since there was only ever one copy on disk. It requires the downloader to be reachable if any directory has to be removed; otherwise it refuses rather than leaving bytes nothing points at.

Cancelling. DELETE /api/v1/downloads/{id} stops a transfer that has not finished: it tells the sidecar to cancel and marks the row cancelled. It does not delete anything. Files that already completed stay where they are, and a download that has already settled is returned unchanged.

Deleting a download’s files. The Downloads page’s bulk Delete calls DELETE /api/v1/downloads/{id}/files once per selected row that still has files on disk. It removes the files and keeps the record, which then shows Cleaned up and counts in that tile — exactly as when the ceiling evicts it, and asking for the download again brings the files back. The downloads of one post share a directory, so the directory goes and every record in it is marked. A download that is pinned, or still running, is left out of the selection and refused by the endpoint: a pin is the promise that the files stay, and a running download is still writing them. Before this endpoint existed the button called the cancel above, which reported every finished row deleted while removing nothing.

What removes stored files, then, is one of four things:

1. Delete on the Downloads page, above: the files go, the records stay and read as cleaned up.
2. Deleting the post from the Library (its bulk Delete), which removes the archive row, its collection memberships, the download records and the directories on disk. Because the records go too, these do not appear under Cleaned up — there is nothing left to count.
3. The size-ceiling sweep, for anything not pinned.
4. The duplicate removal above, for redundant copies.

Saving a stored file to your own computer

A stored file can be handed to your browser. In the downloads drawer each completed file has a Save this file link; in the library drawer, Export the video file and one link per stored image (and live photo clip) or cover.

This goes through GET /api/v1/downloads/{download_id}/files/{name} and requires the media:read scope. The endpoint fetches nothing: it rebuilds the path from the stored row — the download’s own directory plus a name that has to appear in that row’s file list — resolves it, and checks the result still lands inside the media root. A name that is not in the record is a 404 whether or not something of that name exists on disk. A download whose files were evicted answers 404 with reason: "evicted" rather than going back to the platform; ask for the download again to restore it.

The file arrives with the content type the downloader recorded and a Content-Disposition that renames it:

dtk-<platform>-<content_id>-<stored name>

for example dtk-douyin-7408915107113127220-video.mp4. On the volume every post owns a directory, so files can be called video.mp4 and never collide. Your downloads folder has no such directory — fifty saved posts would be fifty files called video.mp4, the second of them video (1).mp4, with nothing in either name saying which post it came from. Characters outside A-Za-z0-9._- become hyphens and each part is capped at 64 characters. The name is produced by the server, so curl, the console and any other client all get the same one.

Note what this is not: the server does not proxy media on demand. Saving a file serves a copy that is already on the disk, behind an authenticated scope. “Store on this instance” and “save this file to my computer” are two different destinations, which is why the console labels them differently.

Turning a stored video into text

A finished download’s video can be transcribed on this machine. The transcriber sidecar runs FunASR models — Paraformer to recognise the speech, FSMN-VAD to find where someone is speaking, CT-Transformer to put the punctuation back — and it sits on a network with no route out, so the audio never leaves the host. It is optional and off by default: the transcriber profile turns it on, and Preparing the FunASR ONNX models gets the ~500 MB of models it needs. The guided installer does both if you say yes to transcription.

Know what it is for before you queue a hundred: the models are the general Mandarin ones, so Chinese speech comes out well and other languages poorly. The result is one whole text per video — no timestamps, no speaker labels. A Bilibili download is transcribed exactly like a Douyin or TikTok one; for a video in several parts, it is the part that download stored — the first, unless you downloaded another.

Starting one

From	How
Console	Downloads → open a finished download → Transcribe video; or Library → open a saved video post → Transcribe video
REST	POST /api/v1/transcriptions with {"download_id": "<uuid>"}; add ?wait=30 to hold the connection until it finishes
MCP	transcribe_video(download_id) — from a link, the chain is parse_url → download_media → transcribe_video

The download has to be finished and its video still on disk. One the size ceiling evicted is refused with a message saying so: download it again, then transcribe.

Asking twice is cheap. A finished transcription of the same file is handed back (200, with reused saying so), and one already running is joined (202) rather than started again. refresh: true forces a new one. normalize (on by default) rewrites spoken numbers as digits; if that last step fails, DTK_TRANSCRIBER_NORMALIZATION_FAILURE_POLICY decides whether you get the text without it (fallback, the default) or a failed transcription (fail).

The Transcriptions page

/transcriptions, in the same Content group, lists every transcription newest first with its state, a preview of the text, the video’s duration, the model and whether numbers were normalised. A running one shows its stage — Decoding audio, Finding speech, Recognising speech, Adding punctuation, Writing numbers as digits — and its details drawer shows the text recognised so far, marked provisional until punctuation and numbers are applied at the end. A finished one can be exported as a .txt.

The list pages 100 at a time and searches the text, the post’s title and its id; its filters and the open drawer are kept in the address (/transcriptions?id=<transcription id>). A transcript’s drawer links to the download it was made from and to the post in the Library. While the transcriber is off — the default — the Transcribe buttons are disabled and say how to turn it on, and so does this page when it is empty.

Cancel stops queued or running work. Delete removes settled records — the text and its metadata — and never the video they came from.

What it costs

CPU, and only while a job runs: the models stay loaded at about 750 MiB, and a job keeps two cores busy. Measured on 2026-10-16, a 3 min 13 s clip took 3.7 seconds on two cores to itself. Sharing those cores with the browser container while it mints identities is slower — see When transcription and the browser share a host.

The sidecar runs one job at a time (DTK_TRANSCRIBER_WORKERS) with eight waiting (DTK_TRANSCRIBER_QUEUE); past that a transcription is marked failed with the reason, and starting it again later is the fix. Transcriptions also count against the worker queue’s ceiling like downloads do, so a bulk run can answer QUEUE_FULL before it reaches the sidecar.

The Library page

/library is the archive: everything this instance has parsed, answered entirely from local storage. Nothing here spends an identity, nothing is rate limited upstream, and a post that has since been deleted is still here with its availability saying so. That is the whole point of keeping an archive.

The tiles across the top show total posts, authors, how many posts are marked Gone (deleted upstream), and a count per platform. Each count carries a “N of M downloaded” footer that is also a link: clicking it filters the list to the posts whose media is on this disk. Seen and kept are different numbers, and reading the first as the second is how an archive of 66 posts gets mistaken for 66 videos on a volume.

Two views:

• Covers (the default) — a wall of cover tiles with duration and, for saved posts, a size chip. The cover comes from this instance’s disk when the post has been stored and from the platform otherwise, in that order, because an archived cover_url is a CDN link that expires and may be hotlink-protected. The grid can be grouped by nothing, by author, or by the day the media was saved.
• Table — sortable columns for the questions a grid cannot answer: post, kind, length, availability, published, last seen.

Paging is by cursor, never by page number, because the table is written to while you walk it and an offset silently skips and repeats rows. Previous walks back through the cursors you have already visited. A page is 50 posts.

Selection is by post key and survives a page turn — selecting a few here, a few there, then acting on all of them is the intended workflow. A bulk bar appears at the bottom of the viewport while anything is selected. Besides collections and Delete, it can Save media (one download per post; what is already stored is counted, not fetched again), Transcribe (each selected video whose media is stored here; image posts and unsaved posts are skipped and counted) and Export selected (the selected posts as NDJSON, through POST /api/v1/archive/export). Save and Transcribe go one post at a time with a running count and a Stop, and report what was done, skipped and failed.

Filters, search and views

Filter	Values	Applies
Search	Substring of the title or description	On submit
Platform	douyin, tiktok, bilibili	On change
Kind	Video, image album	On change
Length	Under a minute (< 60 s), under three minutes (< 180 s), long (≥ 180 s), unknown	On change
Availability	Live, deleted, private, unknown	On change
Collection	Any of your collections	On change
On this disk	Downloaded or not / downloaded only / not downloaded	On change
Author	Click an author’s name on a card, in the table or in a post’s drawer	At once, shown as a removable chip
Tag	Click a #tag in a post’s drawer	At once, shown as a removable chip

Filters combine, and every one of them — with the open post — is kept in the address: /library?author=<uid>, /library?tag=<tag> and /library?platform=<p>&content_id=<id>, the links the other pages use, work from anywhere. The menus apply the moment they change, because picking “douyin” from a closed list is the decision; the text box waits for the button, because typing is not a decision until you stop and firing a query per keystroke would page the archive on the way to a word.

Search is substring matching, not word matching, and that is a deliberate trade-off rather than an oversight: Postgres’ full-text search does not segment Chinese, so a word search would silently match nothing on the primary platform. The cost is that a search for a short string can match more than you meant.

“On this disk” describes what was kept, not the post: an evicted download does not count as downloaded, because there is nothing left to play. The counter beside Clear tells you how many filters are narrowing the list, which is usually the answer to “why does a library of 660 posts show four”.

Collections

A collection is a named set you make by hand. It is the one grouping the library offers that is not derived from the posts themselves — author, platform, length and save date all come out of the record; this one comes out of you deciding.

Open Collections beside Clear to create, list and delete them. A name is at most 80 characters and is unique case-insensitively (enforced by the database, so two console tabs cannot race); the optional note is at most 500 characters and is never parsed.

Add posts by selecting them and using Add to collection in the bulk bar. Remove from this collection appears only while you are filtering by one — “remove from which collection” has no answer otherwise. Adding a post that is already in a collection is a no-op rather than an error, and a post that is not in the archive is skipped, so one stale card does not refuse the other nineteen. A single call takes at most 500 posts.

Deleting a collection removes the label only. The posts stay in the archive and their files stay on disk. It is the one delete on this page that does not touch a post or a byte.

Saving media from the Library

Open a post to get its drawer. The footer carries three actions:

• Archive their history — a backfill: walks this author’s posts past the first page and archives every one. Deliberately separate from the watchlist, because a watchlist entry is for what is new and runs forever, while a backfill is a one-off with a very different cost. The console asks for the server default of 5 pages; the API accepts 1–20 and the worker caps it at 20 pages of 20 posts. It stops at the first empty page, at the ceiling, or when the platform says there is no more history. It costs one upstream request per page and needs that platform’s read scope.
• Store on this instance — starts a download for this post. If the media is already stored the button reads Saved · fetch again and its tooltip says how large the stored copy is; using it fetches a fresh copy and replaces what is there. (This button does not set skip_existing, which is why it re-fetches rather than doing nothing.) Downloading is a queued task, so the row is not stored the instant the toast appears — the page re-checks a few seconds later, and the download itself appears immediately on the Downloads page.
• Export the video file — hands your browser the stored video, when there is one. Image posts have no video, so this button does not appear for them; their slides — and a live photo’s clip, next to its still — get one link each in the drawer body instead, because a browser will not start thirty downloads from a single click and pretending otherwise would silently save one file out of thirty.

The drawer also plays the stored video inline, from this instance’s disk rather than from the platform: no CDN link to expire, no hotlink check to fail, and nobody outside this machine is told what is being watched. It preloads metadata only, so a 240 MB video does not start transferring because a drawer opened.

Under the player, a video post has Transcribe video, the same action as in the downloads drawer — Open the transcript when it already has one. It reads the stored file, never the platform’s, so on a post not saved yet it is disabled and says to store it first. A saved post also links to its download. See Turning a stored video into text.

Exporting the archive

Export in the page header streams the archive as newline-delimited JSON and saves it as archive-YYYY-MM-DD.ndjson.

It is streamed a page at a time rather than assembled server-side, so the size of your own data is not what breaks the export. One post per line, in the same shape the list endpoint returns, without the media manifest — use the list endpoint or GET /api/v1/archive/{platform}/{content_id} when you need the mirrors. A single export is capped at 50 000 rows; past that, page through the list endpoint with cursor.

Two things to know before relying on it:

• It needs the archive:export scope, which ordinary read keys do not carry. This is the one call that turns a read key into a copy of the database, so it is scoped separately on purpose. A console session carries every scope, so the button works for any signed-in user.
• The export applies every filter on the page, the collection and “on this disk” included. It used to ignore those two, so exporting a collection exported the whole archive. To export a hand-picked set instead, select the posts and use Export selected.

For a copy of the whole instance, including identities and settings, see the backup tooling in Operations; this export is the archive only.

Re-checking what still exists

Re-check in the page header answers “which of the things I saved are gone”. Each post is one real request through the identity pool, so the pass runs in the background as a task and is bounded.

Defaults, all settings you can change: archive.recheck_batch posts per pass (25, hard-capped at 200), only posts whose last check is older than archive.recheck_after_days (7), and archive.recheck_pause_seconds (3) between individual requests. That pause is measured rather than chosen: at 0.5 s a live sweep sent 20 TikTok requests in 13 seconds and every one was rejected, while a single manual request a minute later succeeded — the scheduler paces per identity, which does nothing about a platform limit that counts requests per address. Nobody is waiting on this pass, so slow is free.

A deleted post answers the platform’s own not-found, and that is the finding, not a failure: the row is kept and marked deleted, so it stays searchable and exportable with the truth attached. A private post is marked private. Any other error leaves the row untouched, because a risk-control response says nothing about whether the post exists. If an endpoint’s circuit opens mid-pass the sweep stops early rather than marking the rest unchecked.

The worker queues a recheck by itself roughly every six hours, so the button is for when you want the answer now. Setting archive.recheck_after_days to 0 turns rechecking off entirely.

Because it spends the identity pool, this endpoint requires a platform read scope (douyin:read, tiktok:read or bilibili:read) rather than archive:read — reading the archive is promised to cost nothing, and a recheck is a platform read wearing an archive name.

For the same reason the recheck and the author backfill queue like reads. Once the worker queue is at sched.queue_max (500) they answer 503 with QUEUE_FULL and a Retry-After of sched.max_wait_seconds (10): wait that long and ask again. Asking while an identical one is still queued joins it rather than queueing a second. The worker’s own scheduled recheck is submitted directly and never meets that ceiling.

Deleting posts and their media

Select posts and press Delete in the bulk bar. This is the one call in the archive that destroys something: it is not reversible and there is no trash. The archive rows go, their collection memberships go with them, and unless you call the API with media: false the download records and the directories on disk go too.

The files are removed first and the rows second. The other order can leave a record pointing at a directory that is already gone — a download the console offers and cannot deliver; this order can at worst leave bytes with no record, which the storage panel already reports.

If the posts have stored media and the downloader is unreachable — not running, or not answering — the call refuses and deletes nothing, rather than orphaning bytes on a volume nothing can account for. If you genuinely want the record gone and the files kept, that is media: false on POST /api/v1/archive/delete; the console always sends true.

A single call takes at most 500 posts, and it needs media:write (or admin): the same scope that may start a download and cancel one may also remove what it produced, and nothing weaker can.

The posts stay on the platform. Deleting here deletes your copy.

The Watchlist page

/watchlist is the only place in the console that creates standing work. Everything else happens once and finishes; an entry here spends the identity pool every few hours for as long as it exists. The page is built around making that cost legible: the interval is a first-class column, the failure count is visible, and Pause all is a button rather than something you assemble out of per-row toggles.

What it is for is one table: the snapshot history. Until something collects on a schedule, that history holds whatever a human happened to parse at whatever moments they happened to do it — a scatter of unrelated observations rather than a series. This is what makes growth curves real.

Nothing on this page fetches. When an entry comes due, the worker submits it as an ordinary task: the same queue, the same scheduler, the same identity pool, the same archive and snapshot writes at the end. Scheduled collection therefore queues behind interactive requests, which is the correct priority — somebody is waiting on those — and it cannot outrun the rate limits, because there is only one set of them.

Column	Meaning
Target	The name, or the id, with the kind and platform beneath; linked to what it has collected in the Library (not for a TikTok author, whom the archive keys by the numeric id rather than the secUid that is watched)
Interval	How often it runs
Next run	Relative time, or Paused
Runs	Observations collected since it was added
Last run	Relative time, or a red “N failures in a row” carrying the last error as a tooltip
Running	The enable/pause switch
(unnamed)	Edit (name and interval) and Remove, which asks first

The three tiles show how many targets exist and how many are running, total runs, and how many are currently failing.

Adding a target

Five fields: platform, what to watch, the target, an optional name, and the interval. The target is a profile or post link — its platform wins over the menu — or the bare id. A name given here, or later under Edit, is kept; left empty, the first successful run fills in the author’s nickname, and clearing it under Edit hands it back to the platform.

Kind	Target id	What one run collects
An author	The stable author id — sec_user_id on Douyin, secUid on TikTok (both start MS4wLjAB), the numeric mid on Bilibili	One page of their posts (the newest 20). That page carries the author record on every item, so a single request answers both “what is new” and “how many followers now”
A post	The post id, as the archive holds it	That post’s detail, which is how its counters move over time

A TikTok profile link names the @handle, which the post-list endpoint refuses, so the page reads the profile once and stores the secUid. A bare @handle typed into the box still will not work: nothing checks the shape of an id when you add it, so the entry is created — and every run then fails, because the post list has no parameter that takes a handle and refuses it by name rather than sending it somewhere it cannot work. Watching the same platform + kind + target twice is refused, as a duplicate.

The first run is due immediately. Somebody who just added an author wants to see it collect, and holding the first run back by the interval would only make the feature look broken for six hours.

What an entry collects is not configurable from this page, and that is deliberate — see the table above for why one request answers two questions. The API’s pages field (1–10, default 1) is stored on the entry and returned by the list endpoint, but a scheduled run submits a single request for the newest 20 posts; deeper history is what the library’s Archive their history backfill is for.

Intervals, the floor, and a failing entry

The console offers 15 minutes, 1 hour, 6 hours, 12 hours and 24 hours, hiding anything below the server’s floor. New entries default to watchlist.default_interval_seconds (6 hours), and the menu always lists the value the form will send.

The floor is watchlist.min_interval_seconds, 900 seconds by default, and it cannot be set below 60 whatever you do. An interval under the floor is refused rather than silently raised to a number you did not ask for. The reason is not politeness: a target collected every few seconds does not produce a better time series, it burns the whole identity pool on one author and starves everything else. Nothing on any of the platforms moves fast enough to need more than 15 minutes.

Cost, so you can budget: one entry at 6 hours is 4 requests a day. A hundred watched authors at 6 hours is 400 requests a day, one page each. Shortening an interval takes effect immediately rather than after the old one elapses.

Failures back the entry off rather than the task. After 2 consecutive failures the interval starts doubling per additional failure, capped at 8 hours. A watched author whose id was mistyped would otherwise fail every run forever; this turns that into a few attempts a day and leaves the row visible, with its error, for whoever comes to fix it. A platform outage ends, so it is still retried. Re-enabling a paused entry clears the backoff and its error and makes it due now — you are asserting the problem is fixed, and making you wait out an eight-hour backoff to find out would be its own bug.

Two more bounds worth knowing:

• The watcher ticks once a minute and queues at most watchlist.batch_size (10) due entries per tick. The rest stay due and go on the following tick, so a hundred entries coming due on the same minute are spread over minutes rather than landing in front of whoever is using the API.
• The next run is scheduled from submission, not from completion. A task that never finishes would otherwise stop the entry forever, and “collect every six hours” should keep meaning that even when one run is lost.
• Past the capacity hard stop, scheduled collection stands down entirely. Nobody is waiting on it, and standing down the writers is the whole point of the guard.

Pausing, removing and turning it all off

Three different switches, in increasing order of scope:

Action	Effect	Reversal
The Running switch on a row	Pauses that entry. It keeps its history, its runs count and its label	Flip it back; the backoff and last error are cleared and it becomes due now
Pause all in the page header	Disables every enabled entry at once. The button for “something is wrong and I want the pool back”	Per entry, deliberately — there is no “resume all”
The watchlist.enabled setting	Stops scheduled collection globally, leaving every entry exactly as configured. The page shows a banner saying so	Turn the setting back on; nothing else changes

Remove deletes the schedule and nothing else. Everything already collected stays in the archive and in the snapshot history — deleting the instruction is not a request to delete what it produced.

Changing, pausing, adding and removing entries all require the operator role or higher. Viewing the list only needs a signed-in console session.

Settings reference

All of these are runtime settings, editable from the console’s Settings page or the settings API; see Configuration for how settings scopes and editing work.

Setting	Default	What it does
media.enabled	true	Allow media downloads. Off refuses new jobs and leaves stored files alone; nothing is deleted by turning it off
media.max_bytes	2147483648 (2 GiB)	How much disk stored media may occupy. Past it the oldest unpinned downloads are removed until the volume is back under, and an alert says what went. 0 disables eviction entirely
media.max_file_bytes	536870912 (512 MiB)	Ceiling for one file, counted on bytes written rather than on what the server claims. A transfer that reaches it is refused and leaves nothing behind. Must be 0 or at least 1 MiB
media.mirror_max_age_seconds	600	How old an archived media link may be before a download re-parses the post for fresh ones. 0 always re-parses
capacity.warn_percent	80	Disk usage percent at which to raise a warning
capacity.hard_stop_percent	92	Disk usage percent at which new download jobs and scheduled collection pause. Interactive reads are never paused and nothing is deleted
archive.enabled	true	Keep parsed posts and authors after the task result expires. Off makes the instance stateless beyond its logs
archive.store_raw	false	Also keep each platform’s untouched payload. Off by default: it is the largest single thing this instance can choose to store
archive.recheck_after_days	7	How old a post’s last existence check may be before it is checked again. 0 turns rechecking off
archive.recheck_batch	25	How many posts one recheck pass verifies
archive.recheck_pause_seconds	3	Seconds between the individual requests a recheck or backfill makes
watchlist.enabled	true	Run scheduled collection. Off leaves the entries and their history alone and simply stops submitting runs
watchlist.min_interval_seconds	900	Shortest interval an entry may be given. Cannot be set below 60
watchlist.default_interval_seconds	21600 (6 h)	Interval a new entry gets when none is given
watchlist.batch_size	10	How many due entries one tick may queue

Scopes and roles

The API surfaces behind these three pages are scoped separately on purpose: seeing what was collected, seeing what is on the operator’s filesystem, and putting something there are different questions.

Scope	Grants
archive:read	Search the archive, read its stats, read one post
archive:export	The bulk NDJSON export, and only that
media:read	List downloads, read storage usage, fetch a stored file
media:write	Start, retry, pin and cancel a download; delete a download’s files; deduplicate; create and change collections; delete archived posts
douyin:read / tiktok:read / bilibili:read	The availability recheck and the author backfill, because both spend the identity pool
transcription:write	Start, refresh, cancel and delete transcriptions
transcription:read	List transcriptions and read their text
identity:manage	The watchlist endpoints (/api/v1/admin/watchlist/*); operator role or above for anything that writes
admin	Everything, the watchlist endpoints included

Two things follow from this that are easy to get wrong:

• An API key minted with archive:read cannot export the archive and cannot see a single byte on your disk. That is the intent. Give a key the narrowest set that does its job — see Users and API keys.
• A console session carries every scope, bounded by the account’s role rather than by scopes. Only the watchlist endpoints that write additionally check the role (operator or above). So any signed-in console user — including a viewer — can start a download or delete an archived post from the UI. If that is not what you want, do not hand out console accounts you would not hand out media:write to.

Where to go next

• Configuration — every setting named here, and how to change one.
• Concepts — the identity pool, the scheduler and the task queue these pages spend.
• Console overview — the rest of the pages and how they fit together.
• Playground and tools — parsing one link, signing a request, and the URL identifier this page’s inputs use.
• Users and API keys — minting a key with the right scopes.
• Operations — alerts, notification channels, disk and backups.
• REST API guide — the endpoints behind every button here.
• Troubleshooting — when a download fails, a watchlist entry keeps failing, or the disk fills up.