TechStar Asia - Tech News for Builders and Operators

Most "email analytics" demos reach straight for the open pixel. Drop a 1x1 transparent GIF in the body, watch the loads roll in, call it engagement. That worked in 2015. Today Apple's Mail Privacy Protection pre-fetches every image the instant a message hits the inbox, so a chunk of your "opens" are Apple's proxy warming a cache, not a human reading anything. Gmail proxies images too. Corporate gateways strip them. The number you get back is noise wearing a confidence interval.

There's a signal sitting right next to it that nobody can fake or block: did they reply? A reply is a human deciding your message was worth typing back to. It's lower-volume than an open and it's slower to arrive, but it's true. And if your sender is a Nylas Agent Account — a mailbox your agent owns, sends from, and receives into — reply rate is something you can actually compute, because every send and every inbound reply land in the same threaded data plane you already have read access to.

This post is the build. I work on the Nylas CLI, so the terminal commands below are the exact ones I reach for, and I'll show the raw HTTP next to each so you can wire it into whatever language your agent runs in.

What you're actually measuring

Reply rate is one fraction:

reply rate = replied_threads / sent_threads   (per segment, per time window)

sent_threads is the count of distinct conversations your agent started in some window. replied_threads is how many of those got at least one inbound message back from a real recipient. Segment by campaign, by recipient cohort, by prompt version — whatever dimension you want to compare — and watch it move over time.

The mechanism that makes this computable is threading. When the agent sends, Nylas stamps the outbound message with a Message-ID and returns a thread_id. When the recipient replies, their client sets In-Reply-To and References pointing back at that Message-ID, Nylas stitches the reply into the same thread, and the inbound message.created webhook carries the same thread_id. So the entire correlation is: remember which thread_id you sent into, and when an inbound reply shows up on that thread, count it. No subject-line guessing, no header parsing by hand. The email threading doc goes deep on the header chain if you want it; for analytics you only need the thread_id.

One thing to get straight up front, because it shapes the whole design: Agent Accounts don't support custom message metadata. On a regular grant you might stuff a campaign key into the send and filter on it later. You can't here. The mapping from thread_id to "which campaign / which run / which segment" lives in your database. That's not a limitation so much as a clarification of where the line is — Nylas owns the email data plane, your app owns the analytics.

Why this beats open tracking

Opens are noisy and increasingly blocked. Apple MPP, image proxying, and gateway stripping inflate or suppress the count depending on the recipient's setup. You can't tell a real open from a proxy pre-fetch.
A reply is unforgeable intent. Nobody's mail client replies on their behalf. The signal is sparse but it means exactly what you think it means.
It's the right loop for an agent. An autonomous agent doesn't need a dashboard a human reads quarterly; it needs a number it can optimize against. Reply rate per prompt version tells the agent which copy actually works.
Nothing new on the data plane. An Agent Account is just a grant with a grant_id. Sends, threads, messages, webhooks — all the same grant-scoped endpoints you'd use for any mailbox. The agent abstraction adds zero new API surface to learn.

Honest tradeoff, stated plainly: reply rate is a truer signal but a lower-volume one. On cold outreach you might see single-digit reply percentages where opens read 40%. If you need a high-frequency proxy for "message got displayed," opens still have a (degraded) place. If you need to know whether your agent's email is actually working, reply rate is the metric that won't lie to you. You can run both — they answer different questions.

Before you begin

You need an Agent Account to send from. It's provisioned in two equivalent ways.

The API call is POST /v3/connect/custom with provider: nylas and a settings.email on a domain you've registered (a real custom domain, or a *.nylas.email trial subdomain):

curl --request POST \
  --url 'https://api.us.nylas.com/v3/connect/custom' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "nylas",
    "settings": { "email": "[email protected]" },
    "name": "Outbound Agent"
  }'

The CLI does the same in one line:

nylas agent account create [email protected] --name "Outbound Agent"

Either way you get back a grant_id. No OAuth dance, no refresh token — the agent owns the mailbox outright. Hold onto that grant_id; it's the identifier in every endpoint below. The Agent Accounts docs cover domain warm-up and the free-plan limits (200 messages per account per day, worth knowing before you batch a campaign).

You'll also want a small table in your own store. The schema is boring on purpose:

-- one row per agent send
CREATE TABLE agent_sends (
  thread_id    TEXT PRIMARY KEY,   -- returned by the send call
  message_id   TEXT NOT NULL,
  segment      TEXT NOT NULL,      -- "q2-trial-nudge", "prompt-v3", etc.
  sent_at      TIMESTAMPTZ NOT NULL,
  replied_at   TIMESTAMPTZ         -- NULL until a reply lands
);

-- dedup ledger so we never double-count a webhook
CREATE TABLE seen_notifications (
  notification_id TEXT PRIMARY KEY
);

That's the entire persistence layer. Everything else is reads against Nylas.

Send, and record the thread

When the agent sends, capture the thread_id off the response and write a row. The send endpoint is POST /v3/grants/{grant_id}/messages/send:

curl --request POST \
  --url 'https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages/send' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "to": [{ "email": "[email protected]" }],
    "subject": "Following up on your trial",
    "body": "Hi there — checking in on how the trial is going."
  }'

The response carries id (the message id) and thread_id. That thread_id is the key you store against your segment:

INSERT INTO agent_sends (thread_id, message_id, segment, sent_at)
VALUES ('<thread_id>', '<message_id>', 'q2-trial-nudge', now());

The CLI equivalent, with --json so you can pipe the response into your recorder:

nylas email send [email protected] \
  --to [email protected] \
  --subject "Following up on your trial" \
  --body "Hi there — checking in on how the trial is going." \
  --json

Pass the account as the first positional grant-id argument (the email resolves to the grant). The --json output includes the same thread_id; grab it with jq and run your insert. If the agent is replying into an existing thread rather than starting one, use --reply-to <message-id> (or reply_to_message_id in the API body) — Nylas sets the threading headers so the reply lands on the original thread, and your thread_id mapping is already correct.

The rule: record on every send, no exceptions. A send you forgot to log is a thread that can never count as sent, which silently deflates your denominator and inflates your rate. Tag every send, even ones you don't think you'll measure.

Subscribe to inbound replies

Inbound mail fires the standard message.created webhook. One important fact about Nylas webhooks: they're application-scoped, not grant-scoped. You subscribe once at the app level, and events for every grant in the app arrive at that one endpoint, each payload carrying the grant_id it belongs to. You don't create a webhook per Agent Account.

The subscription is POST /v3/webhooks:

curl --request POST \
  --url 'https://api.us.nylas.com/v3/webhooks' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "trigger_types": ["message.created"],
    "webhook_url": "https://your-app.example.com/nylas/inbound",
    "description": "Reply-rate inbound signal"
  }'

Or from the CLI:

nylas webhook create \
  --url https://your-app.example.com/nylas/inbound \
  --triggers message.created \
  --description "Reply-rate inbound signal"

While building, you don't need a public URL at all — nylas webhook server --tunnel cloudflared --secret <WEBHOOK_SECRET> stands up a local receiver behind a tunnel and prints each event as it arrives, which is how I test the handler before deploying it. The --secret is required whenever --tunnel is set, since the server verifies each event's signature; pass --no-tunnel instead if you just want a loopback-only receiver for local dev. Nylas signs every webhook with an X-Nylas-Signature header (hex HMAC-SHA256 of the raw body using your webhook secret); verify it in production, and nylas webhook verify checks a signature locally if you want to sanity-check your comparison logic.

Correlate: match each inbound reply to a sent thread

Here's the loop the whole post builds toward. For each message.created event:

Dedup on the notification id. Nylas guarantees at-least-once delivery — the same event can arrive up to three times. The top-level notification id is constant across all retries of one event, so it's your dedup key. If you've seen it, drop the event.
Filter out the agent's own sends. message.created fires for outbound messages too. You only care about real replies, so skip any message whose from is the agent's own address.
Match thread_id against your agent_sends table. If the inbound reply's thread_id is one the agent sent into, that thread just earned a reply.

import os, hmac, hashlib

def handle_message_created(raw_body: bytes, signature: str, db):
    # Signature check: HMAC-SHA256 of the raw body. Guard equal length
    # before constant-time compare — timingSafeEqual-style compares throw
    # on a length mismatch.
    expected = hmac.new(
        os.environ["NYLAS_WEBHOOK_SECRET"].encode(),
        raw_body, hashlib.sha256,
    ).hexdigest()
    if len(expected) != len(signature) or not hmac.compare_digest(expected, signature):
        return 401

    event = json.loads(raw_body)

    # 1. Dedup on the top-level notification id (constant across retries)
    notif_id = event["id"]
    if db.already_seen(notif_id):
        return 200
    db.mark_seen(notif_id)

    obj = event["data"]["object"]

    # 2. Real replies only — skip the agent's own outbound sends
    agent_addr = "[email protected]"
    senders = [p.get("email", "").lower() for p in obj.get("from", [])]
    if agent_addr in senders:
        return 200

    # 3. Match the reply's thread to a recorded send
    thread_id = obj["thread_id"]
    send = db.find_send_by_thread(thread_id)
    if send and send["replied_at"] is None:
        db.mark_replied(thread_id, when=obj["date"])  # first reply wins

    return 200

A note on the payload body. The Nylas docs aren't fully consistent on whether message.created ships the full body inline, so use the safe framing: don't rely on the webhook payload for the body — fetch the full message by id when you need it, and branch on message.created.truncated. For reply-rate counting you usually don't need the body at all (the thread_id, from, and date on the summary are enough), but if you want the reply text to feed back into the agent, fetch it:

# Full message by id
curl --request GET \
  --url 'https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages/<MESSAGE_ID>' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>'

nylas email read <message-id> [email protected]

When the event arrives as message.created.truncated (the payload exceeded ~1 MB), the body isn't inline and the fetch-by-id above is mandatory rather than optional. Branch on the trigger type and re-fetch.

If you'd rather reconstruct the whole conversation — say, to confirm the reply really is a back-and-forth and not an auto-responder — read the thread instead of the single message:

# Full thread by id
curl --request GET \
  --url 'https://api.us.nylas.com/v3/grants/<GRANT_ID>/threads/<THREAD_ID>' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>'

nylas email threads show <thread-id> [email protected]

The thread object gives you message_ids, participants, and both latest_message_received_date and latest_message_sent_date — handy as a cross-check that a received message genuinely arrived after your send.

Aggregate the rate yourself

The counting is done in your database, not in Nylas. There's no Nylas endpoint that returns "reply rate for campaign X" — and there shouldn't be, because the segmentation is your domain logic. With the two tables above, the rate per segment is a single query:

SELECT
  segment,
  count(*)                              AS sent_threads,
  count(replied_at)                     AS replied_threads,
  round(100.0 * count(replied_at) / count(*), 1) AS reply_rate_pct
FROM agent_sends
WHERE sent_at >= now() - interval '30 days'
GROUP BY segment
ORDER BY reply_rate_pct DESC;

Bucket by date_trunc('week', sent_at) instead of one window and you've got the trend line. Because replies trickle in over days, give each cohort time to mature before you trust its rate — a campaign sent yesterday will look worse than it really is until its replies land. I anchor every cohort to its sent_at rather than the wall clock for exactly that reason. The message timestamp on each event (obj["date"], also what date -u formats if you're stamping your own records) keeps sent and replied on the same clock.

That's the full loop: send and record the thread_id, receive message.created and dedup on the notification id, filter the agent's own sends, match the reply's thread, and divide replied by sent per segment. Threading is what makes a reply attributable to a specific send, and attribution is what turns "we got some replies" into a per-campaign rate the agent can steer on.

Guardrails and gotchas

Don't double-count multi-reply threads. A prospect might reply twice. The replied_at IS NULL guard means the first reply flips the thread to "replied" and later ones are ignored — exactly what you want, since the unit is threads that got a reply, not reply messages.
Auto-responders look like replies. An out-of-office bounce arrives as a real inbound message.created. If your reply rate suddenly spikes on a cold list, sample the threads — you may be counting vacation autoresponders. Filtering those is heuristic (subject patterns, known sender domains); decide whether they count for your use case.
Webhooks are at-least-once, so the dedup ledger isn't optional. Without it, three delivery attempts of one reply count as one reply (because replied_at only flips once) but still cost you three handler runs and three signature checks. Dedup on the notification id first, before any work.
The denominator is the fragile part. Reply rate breaks far more often from undercounted sends than from miscounted replies. Make the send-recording write part of the same code path as the send itself, not a follow-up job that can fail independently.
Reconcile against threads periodically. Webhooks can be missed during an extended grant outage. A nightly pass that lists recent threads for the grant and checks latest_message_received_date against your agent_sends rows catches anything the live path dropped.

What's next

Track email reply rates for AI agents — the cookbook recipe this post is built on, including the polling-vs-webhook tradeoff table
Email threading for agents — the header chain behind thread_id and why it beats subject matching
Agent Accounts — provisioning, supported endpoints, and plan limits
Nylas CLI command reference — every nylas email, nylas webhook, and nylas agent subcommand used above