01 · Ingest
Read the open web
A Kestra flow per source — RSS, arXiv, GitHub trending, HN, Reddit, YouTube transcripts, Exa semantic search. Each one is idempotent, retried with backoff, and recorded for replay.
ingest.rss…
Lighthouse is four steps: ingest, process, deliver, observe. Each one is a Kestra flow described in plain YAML — versioned, retried, replayed, observed. We did not invent any of that. Kestra handed it to us.
01 · Ingest
A Kestra flow per source — RSS, arXiv, GitHub trending, HN, Reddit, YouTube transcripts, Exa semantic search. Each one is idempotent, retried with backoff, and recorded for replay.
ingest.rss…
02 · Process
Documents are embedded into pgvector, near-duplicates are clustered, and an LLM classifies what matters for your topic. Cluster summaries are the source material for the brief.
process.embed_dedup…
03 · Deliver
Your AI provider drafts the Markdown brief from the cluster summaries. Lighthouse auto-formats and posts it to email (HTML), Slack (mrkdwn), Discord (native MD), Telegram (HTML), or Notion — whichever channels you configured.
deliver.brief…
04 · Observe
Kestra’s UI shows every step that ran, every retry, every artifact. We forward alerts and let you replay any execution if a source breaks or a model is down.
monitors.alerts…
Adding a new source is a 20-line file you can review in a PR. No DSL, no framework lock-in.
Flaky RSS feed? Slow LLM call? Kestra retries with backoff and you wake up to a brief, not an outage.
A namespace per stage, logs per task, a Gantt per execution. Debugging looks like a normal job.
Every ingestion source is a small flow with three or four tasks. Tasks call HTTP, run Python, talk to LLMs, and write to Postgres — orchestrated by Kestra’s engine.
# flows/ingest/rss.yaml
id: rss
namespace: company.team.lighthouse.ingest
inputs:
- id: topic_id
type: STRING
triggers:
- id: every-morning
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 7 * * *"
tasks:
- id: load_profile
type: io.kestra.plugin.scripts.python.Script
script: |
profile = read_topic("{{ inputs.topic_id }}")
- id: pull_feeds
type: io.kestra.plugin.fs.http.Request
uri: "{{ outputs.load_profile.vars.feed }}"
retry:
type: constant
maxAttempt: 3
interval: PT30S
- id: upsert
type: io.kestra.plugin.jdbc.postgresql.Query
sql: "INSERT INTO lh.documents …"Every box in Lighthouse is something somebody else figured out first. We borrow heavily and link out loudly — please go support these projects.
The reason this entire pipeline is YAML you can read. Replays, retries, scheduling, observability — out of the box.
Visit project
Sign-up, settings storage, row-level security and realtime. The whole user surface in one weekend.
Visit project
BYOK works because LiteLLM speaks OpenAI on top of every provider you'd actually pick.
Visit project
Server components for the slow parts, client components for the interactive parts. One Node host.
Visit project
The topic picker and every hover hint use Floating UI’s collision-aware positioning.
Visit project
Aurora, ShineBorder, BorderBeam, MagicCard, AnimatedGridPattern — the visual layer of this page is shamelessly enabled by Magic UI.
Visit project
Every pixel on this page. Utility-first, dark-first, a11y-friendly defaults.
Visit project
Documents, embeddings and briefs live in the same database. One join away.
Visit project
The hero text, page transitions, marquee, magnet cursors — all backed by Framer Motion.
Visit project
Renders the sample brief safely with custom components — no innerHTML.
Visit project
The dashboard exposes lighthouse-* tools via document.modelContext so browser agents can select topics, run briefs, and build custom profiles without scraping the DOM.
Visit project
RSS, arXiv, GitHub, HN, Reddit, YouTube transcripts — none of this works without their generous public surfaces.
Visit project
The interesting part is the brief that lands tomorrow morning. Pick a topic and bring your AI key.