#Content Pipeline

The publish pipeline (packages/pipeline/publish.mjs) turns your source content into everything the app reads: chapter JSON, narrated MP3 audio, per-word timing files, cover images, and a library manifest — uploaded to the brand's R2 bucket. It is brand-neutral: your site owns the parser that reads your content shape; the pipeline handles TTS, stitching, hashing, upload, manifest, and notify.

your source (markdown, etc.)
   │  --parser <your-module.mjs>            (site-owned; produces canonical shape)
   ▼
packages/pipeline/publish.mjs
   ├─ diff by content hash (only changed chapters proceed)
   ├─ chapter JSON                          books/<id>/chapters/<ch>.<hash>.json
   ├─ Azure neural TTS per block → stitch   books/<id>/audio/<ch>.<hash>.mp3
   │    + word timings                      books/<id>/timings/<ch>.<hash>.json
   ├─ covers (hashed)                       books/<id>/covers/cover.<hash>.<ext>
   ├─ manifest.json (uploaded LAST)         manifest.json
   └─ POST /api/admin/publish → web push

#Command

node packages/pipeline/publish.mjs --brand <id> --source <path> --parser <module> [flags]

The three required flags:

The root npm run publish script only passes --brand storylark, so it will exit with the usage message on its own. Append the rest after --, e.g. npm run publish -- --source examples/demo --parser examples/demo/parser.mjs.

#Optional flags

#Environment

ffmpeg and ffprobe must be on PATH for the audio stitch step (packages/pipeline/stitch.mjs).

#The parser contract

The pipeline never assumes your content format. You provide an ESM module whose default export (or a named parse export) is:

export async function parse(sourceRepo, previousChapters, siteOrigin) {
  // sourceRepo       — the --source path
  // previousChapters — state.chapters from the last publish, keyed "bookId/chapterId"
  //                     (use it to keep block IDs stable across edits)
  // siteOrigin       — marketing origin (appOrigin with the `app.` label dropped),
  //                     used to resolve root-relative image srcs
  return {
    books: [
      {
        book: { id, title, author, description?, order?, series?, /* ... */ },
        chapters: [
          {
            id, title, label?, blocks, charLength, wordCount, readingTime?, setting?
          }
        ]
      }
    ]
  };
  // A bare array of the same `{ book, chapters }` items is also accepted.
}

blocks are StoryLark content blocks (paragraph with em/strong spans, scene-break, display-beat, message-block, image, end-marker). You don't have to build these by hand: packages/pipeline/lib/md.mjs exports helpers the bundled parser uses —

• readFrontmatter(source) — flat key: value frontmatter.
• parseBlocks(body, { siteOrigin }) — markdown prose → blocks (conventions below).
• chapterCharLength(blocks), countWords(blocks).
• stabilizeBlockIds(blocks, previousBlocks) — reuses a block's prior ID when its text is unchanged, so bookmarks and reading positions survive edits elsewhere.

#Markdown block conventions (parseBlocks)

See examples/demo/parser.mjs for a complete, working parser that reads the public-domain markdown stories in examples/demo/books/ — the fastest way to try the pipeline end to end:

node packages/pipeline/publish.mjs --brand storylark \
  --source examples/demo --parser examples/demo/parser.mjs \
  --no-audio --local app/dist

#Incremental, content-hash publishing

Every chapter is hashed (contentHash({ blocks, title }), first 8 hex chars). The pipeline keeps a per-brand state file at .storylark/state/<brand>.json recording each chapter's last hash, audio info, and publish date. On each run it:

1. Parses everything and computes each chapter's current hash.
2. Publishes only chapters whose hash changed (others are untouched — no re-TTS, no re-upload). --dry-run prints this plan without acting.
3. Writes chapter JSON, and (unless --no-audio) synthesizes + stitches audio and timings.
4. Uploads artifacts under content-hashed, immutable keys (<ch>.<hash>.json / .mp3), so a republished chapter gets a new key and old downloads keep working until re-fetched.
5. Uploads manifest.json last (short TTL), so readers never see a manifest pointing at objects that aren't uploaded yet.
6. If ADMIN_KEY is set and not --local, POSTs the new version to /api/admin/publish to fan out push notifications.

Before any of that, a conflict check. The deployment stores its own copy of your source (so the admin portal can edit it), which means a laptop's working tree and the live site can drift apart. Every publish reads the live manifest first and refuses — exit code 2, naming each chapter and every hash involved — when the live content differs both from what this machine last published and from what this run is about to write. The ordinary case (nobody else has touched it) stays silent; --pull reconciles a real divergence by fetching the deployment's source down first; --force publishes over the check when you're sure. Chapter order and book metadata mismatches are warnings, not refusals.

Covers are handled similarly: a book's coverSource (art in your source repo's public/) or brands/<id>/assets/covers/<bookId>.<ext> is hashed and uploaded under covers/cover.<hash>.<ext>; books with no art fall back to the brand icon.

#TTS and word timings

Audio is generated at publish time, once per chapter revision (not on demand). The voice named in deployment/<id>/deployment.json tts.voice picks the provider:

• Bundled local voices (the default — free, no account). Kokoro voice ids (af_heart, bm_fable, …) run the Apache-licensed Kokoro-82M model on your own machine via packages/pipeline/tts-kokoro.mjs. 28 English voices ship with it; the model (~90 MB) downloads on first use and is cached. Word timings are estimated: each sentence's duration is exact, and words inside it are apportioned by length — accurate enough for read-along highlighting. af_heart is StoryLark's default narrator.
• Azure neural TTS (optional premium tier — bring your own key). packages/pipeline/tts.mjs calls Azure one block at a time, using voices like en-US-Ava:DragonHDOmniLatestNeural, and collects a WordBoundary event per spoken word (character offset + 100 ns audio offset). Requests are spaced to respect the Azure F0 free-tier rate limit (20 req/min) with retry/backoff on transient throttling. Requires AZURE_SPEECH_KEY / AZURE_SPEECH_REGION.
• packages/pipeline/stitch.mjs concatenates the per-block MP3 chunks into one chapter file (with a short beat of silence for scene breaks) and shifts each block's word times by the measured (ffprobe) chunk offsets — trailing silence would otherwise drift a naive sum.
• The result is a timings JSON of [charStart, charEnd, startMs, endMs] per word, which the reader uses for word-synced highlighting and tap-to-seek. See read-along.md.

A monthly character budget (hard stop at 450K, under the Azure F0 500K/month limit) is tracked in the state file's charLedger; a publish that would exceed it aborts with guidance to use --no-audio or wait for the next month. The budget applies to Azure voices only — the bundled local voices are unmetered.

Re-narration is per block, not per chapter. A content-addressed cache of each block's synthesized audio (keyed by that block's type and spoken text — the same key stabilizeBlockIds matches on) means only the blocks whose words actually changed get re-synthesized; the rest are spliced back in from the previous run, with word timings re-derived from measured durations so highlighting still lines up. Editing one paragraph re-narrates one paragraph, not the chapter it lives in — and it's charged that way against the monthly budget, too. --renarrate-all bypasses the cache when you genuinely want everything redone (a narrator-voice change, for example).

Chapters published --no-audio (hasAudio: false in the manifest) fall back to the device's own Web Speech synthesis for Listen mode — always available, lower quality.

#Output layout (R2 / --local dir)

manifest.json                              library catalog + version (~60s cache)
books/<bookId>/
  chapters/<chapterId>.<hash>.json         blocks + metadata (immutable)
  audio/<chapterId>.<hash>.mp3             48kHz/96kbps mono (immutable)
  timings/<chapterId>.<hash>.json          per-word timing (immutable)
  covers/cover.<hash>.<ext>                cover art (immutable)

The bucket is named <brand>-content; an R2 custom domain serves the bucket root at the brand's contentOrigin, which is exactly what the app fetches from (packages/core/src/brand.ts contentUrl()). More on the storage/caching model in data-model.md.

Found a gap? StoryLark is open source — improve these docs on GitHub.