- The Ad Resolution Pipeline Build an ad resolution pipeline with targeting filters, budget checks, frequency caps, and weighted selection to pick the best creative fast.
- Real-Time Ad Analytics with Server-Sent Events Live analytics dashboard architecture: event tracking, SSE streaming, and reactive charts for KPI visualization.
- Budget Pacing & Campaign Scheduling Budget pacing algorithms, daily resets, campaign lifecycle automation, and server-side scheduling patterns for ad systems in SvelteKit.
The Gap Between Working and Production-Ready
The previous six articles built a complete ad campaign system - domain model, CRUD management, resolution pipeline, real-time analytics, A/B testing, and budget pacing. Every piece works correctly in development. None of it is ready for production as written.
Production readiness is not a feature you add at the end. It is a set of properties (performance, security, reliability, and legal compliance) that the codebase either has or does not.
This article addresses the most critical gaps in the system we have built: slow queries against the events table, per-request frequency-cap lookups that do not scale, unvalidated HTML creatives that are an XSS vector, a tracking endpoint with no protection against click spam, and a data collection pipeline that may violate GDPR without explicit user consent.
Each section below is a standalone concern. Read them in the order that matches your immediate priorities.
Database Indexing: The Events Table
The ad_events table grows continuously. Every impression, every click, every conversion adds a row. After one month on a moderately trafficked site, you are looking at millions of rows. At that point, unindexed queries against this table become the slowest thing in your system.
Look at the queries we run most often:
- Metrics aggregation:
WHERE campaign_id = $1 AND type = 'impression' - Frequency caps:
WHERE session_id = $1 AND ad_group_id = $2 AND type = 'impression' AND timestamp >= $3 - Timeseries:
WHERE campaign_id = $1 AND timestamp >= $2 - Top creatives:
WHERE campaign_id = $1 GROUP BY creative_id, type
Every one of these filters on campaign_id, session_id, ad_group_id, type, and timestamp. Without indexes on these columns, PostgreSQL performs a full sequential scan through every row in the table.
-- migrations/add-ad-events-indexes.sql
-- Run this before going live. Existing tables can be indexed with CONCURRENTLY
-- so the operation does not lock the table during execution.
-- Primary analytics query: campaign + type + timestamp
CREATE INDEX CONCURRENTLY idx_ad_events_campaign_type_time
ON ad_events (campaign_id, type, timestamp DESC);
-- Frequency cap query: session + ad_group + type + timestamp
CREATE INDEX CONCURRENTLY idx_ad_events_session_group_type_time
ON ad_events (session_id, ad_group_id, type, timestamp DESC);
-- Top creatives query: campaign + creative + type
CREATE INDEX CONCURRENTLY idx_ad_events_campaign_creative_type
ON ad_events (campaign_id, creative_id, type);
-- Dashboard timeseries: campaign + timestamp
-- (already covered by idx_ad_events_campaign_type_time, but a covering
-- index on just campaign_id + timestamp is slightly faster for range scans)
CREATE INDEX CONCURRENTLY idx_ad_events_campaign_time
ON ad_events (campaign_id, timestamp DESC); One thing worth understanding about composite indexes is that column order determines which queries they can accelerate. PostgreSQL can use a composite index (a, b, c) for queries that filter on a, on a + b, or on a + b + c - but not for queries that filter only on b or only on c without also filtering on a. This is the leftmost prefix rule. It explains why campaign_id leads every index here: all four query patterns start with a campaign filter. type and timestamp follow because they narrow the result further within a campaign. Adding type before timestamp is intentional - filtering by event type eliminates a large fraction of rows before the range scan on timestamp begins.
In Drizzle, you declare indexes in the schema file rather than raw SQL:
// src/lib/server/db/schema.ts (additions)
import { index } from 'drizzle-orm/pg-core'
export const adEvents = pgTable(
'ad_events',
{
// ... existing columns unchanged
},
(table) => ({
// Analytics: campaign breakdown by event type and time
campaignTypeTimeIdx: index('idx_ad_events_campaign_type_time')
.on(table.campaignId, table.type, table.timestamp)
.desc(),
// Frequency caps: how many times has this session seen this ad group?
sessionGroupTypeTimeIdx: index('idx_ad_events_session_group_type_time')
.on(table.sessionId, table.adGroupId, table.type, table.timestamp)
.desc(),
// Top creatives table: what's the CTR per creative?
campaignCreativeTypeIdx: index('idx_ad_events_campaign_creative_type').on(
table.campaignId,
table.creativeId,
table.type
),
// Timeseries range scans
campaignTimeIdx: index('idx_ad_events_campaign_time')
.on(table.campaignId, table.timestamp)
.desc()
})
) Events Tables Grow Fast - Plan for PartitioningAt scale, even indexed tables slow down as row counts climb into the hundreds of millions. PostgreSQL table partitioning by month (
PARTITION BY RANGE (timestamp)) lets you drop old partitions instead of running expensiveDELETEoperations and keeps index sizes manageable. Set this up before you need it - retrofitting partitioning onto an existing large table is painful.
Redis-Backed Frequency Caps
The frequency cap check in Part 3 queries the database once per candidate ad group. With ten candidates, that is ten sequential database queries on the hot path. Each query takes 5–20ms; ten queries take 50–200ms. Your target is under 10ms for the entire resolution pipeline.
Redis solves this with a single pipeline call. Instead of querying ad_events for impression counts, we maintain a Redis counter per sessionId:adGroupId:date key and check them in a single batch.
// src/lib/server/ads/frequency-redis.ts
import { Redis } from '@upstash/redis'
import { env } from '$env/static/private'
const redis = new Redis({
url: env.UPSTASH_REDIS_REST_URL,
token: env.UPSTASH_REDIS_REST_TOKEN
})
/**
* Return the impression count key for a session + ad group + today.
* The key expires automatically at midnight UTC.
*/
function freqKey(sessionId: string, adGroupId: string): string {
const today = new Date().toISOString().slice(0, 10) // YYYY-MM-DD
return `freq:${today}:${sessionId}:${adGroupId}`
}
/**
* Batch-fetch impression counts for all candidate ad groups in a single
* Redis pipeline call. Returns a map of adGroupId → count.
*
* Replaces the N individual database queries from the original implementation.
*/
export async function batchGetFrequencyCounts(
sessionId: string,
adGroupIds: string[]
): Promise<Map<string, number>> {
if (adGroupIds.length === 0) return new Map()
const keys = adGroupIds.map((id) => freqKey(sessionId, id))
// Pipeline: one round-trip for all keys
const results = await redis.mget<number[]>(...keys)
const map = new Map<string, number>()
for (let i = 0; i < adGroupIds.length; i++) {
map.set(adGroupIds[i], results[i] ?? 0)
}
return map
}
/**
* Increment the impression counter for a session + ad group.
* Sets the TTL to expire at midnight UTC so counters reset daily automatically.
* Call this from the event tracker after recording an impression.
*/
export async function incrementFrequencyCount(sessionId: string, adGroupId: string): Promise<void> {
const key = freqKey(sessionId, adGroupId)
// INCR and EXPIRE are batched in a pipeline for two reasons: performance
// (one round-trip instead of two) and safety (if the process crashes
// between a bare INCR and a bare EXPIRE, the key would never expire and
// would accumulate indefinitely. Pipelining does not give true atomicity
// in Redis, but it ensures both commands are sent together and the server
// processes them sequentially without another client interleaving).
const midnight = new Date()
midnight.setUTCHours(24, 0, 0, 0)
const ttl = Math.floor((midnight.getTime() - Date.now()) / 1000)
const pipeline = redis.pipeline()
pipeline.incr(key)
pipeline.expire(key, ttl)
await pipeline.exec()
} Update the resolver’s Stage 4 to use the batched lookup:
// In resolver.ts - Stage 4 (frequency cap check)
import { batchGetFrequencyCounts } from './frequency-redis'
// ─── Stage 4: Apply frequency caps (batched) ──────────────────────────────
const adGroupIds = withinBudget.map((g) => g.adGroup.id)
const frequencyCounts = await batchGetFrequencyCounts(context.sessionId, adGroupIds)
const underCap = withinBudget.filter((group) => {
const cap = group.adGroup.targeting.frequencyCap
if (cap === null) return true // no cap configured
const count = frequencyCounts.get(group.adGroup.id) ?? 0
return count < cap
}) Update the tracker to increment the Redis counter alongside the database insert:
// In tracker.ts - after inserting the event row:
import { incrementFrequencyCount } from './frequency-redis'
// Increment the Redis frequency counter for impression events
if (input.type === 'impression') {
await incrementFrequencyCount(input.sessionId, input.adGroupId)
} Why Upstash Redis?Upstash Redis works with Vercel’s edge and serverless functions because it communicates over HTTP rather than a persistent TCP connection. Standard Redis clients that maintain connection pools do not work in serverless environments where each function invocation is stateless. If you run a persistent server (a VPS, a container), a standard Redis client is fine.
The performance difference is significant:
| Approach | Queries | Latency |
|---|---|---|
| Per-candidate DB query (original) | N queries (sequential) | 50–200ms |
| Redis pipeline (new) | 1 round-trip | 2–8ms |
Rate Limiting the Tracking Endpoint
The /api/ads/track endpoint is publicly accessible and unauthenticated. Without rate limiting, a single bot or a malicious advertiser competitor can hammer the endpoint and inflate impression counts, exhaust daily budgets, or trigger your fraud detection thresholds.
A sliding window counter in Redis is the standard approach:
// src/lib/server/ads/rate-limiter.ts
import { Redis } from '@upstash/redis'
import { env } from '$env/static/private'
const redis = new Redis({
url: env.UPSTASH_REDIS_REST_URL,
token: env.UPSTASH_REDIS_REST_TOKEN
})
interface RateLimitResult {
allowed: boolean
remaining: number
resetAt: number // Unix timestamp when the window resets
}
/**
* Sliding window rate limiter.
*
* @param key Unique identifier for this rate limit bucket (e.g. IP address)
* @param limit Maximum allowed requests in the window
* @param windowMs Window duration in milliseconds
*/
export async function checkRateLimit(
key: string,
limit: number,
windowMs: number
): Promise<RateLimitResult> {
const now = Date.now()
const windowStart = now - windowMs
const redisKey = `rl:${key}`
// A sorted set (zset) is used rather than a simple INCR counter because we need
// a true sliding window, not a fixed window. A simple counter resets at a fixed
// boundary (e.g. every full minute: 10:00:00, 10:01:00). That means a burst of
// 99 requests at 10:00:59 and another 99 at 10:01:01 - all within two seconds -
// passes the limit. A sorted set stores each request timestamp as a score;
// ZREMRANGEBYSCORE removes entries older than `now - windowMs` before counting,
// so the window rolls continuously with real time regardless of clock boundaries.
const pipeline = redis.pipeline()
pipeline.zremrangebyscore(redisKey, 0, windowStart)
pipeline.zadd(redisKey, { score: now, member: `${now}-${Math.random()}` })
pipeline.zcard(redisKey)
pipeline.expire(redisKey, Math.ceil(windowMs / 1000))
const results = await pipeline.exec()
const count = (results[2] as number) ?? 0
return {
allowed: count <= limit,
remaining: Math.max(0, limit - count),
resetAt: now + windowMs
}
} Apply it in the tracking endpoint:
// src/routes/api/ads/track/+server.ts (updated)
import { checkRateLimit } from '$lib/server/ads/rate-limiter'
export const GET: RequestHandler = async ({ url, cookies, request, getClientAddress }) => {
// Rate limit: 100 tracking events per minute per IP
const ip = getClientAddress()
const limit = await checkRateLimit(`track:${ip}`, 100, 60_000)
if (!limit.allowed) {
return new Response('Too Many Requests', {
status: 429,
headers: {
'Retry-After': String(Math.ceil((limit.resetAt - Date.now()) / 1000)),
'X-RateLimit-Remaining': '0'
}
})
}
// ... rest of existing tracking logic unchanged
} Rate Limiting Alone Is Not Fraud DetectionRate limiting stops obvious automated abuse - bots hitting the endpoint at thousands of requests per second. It does not catch sophisticated click fraud where requests are throttled to look human. True fraud detection requires additional signals: user agent analysis, IP reputation databases, session behaviour patterns, and statistical anomaly detection. These are large topics in their own right; consider a dedicated fraud detection service (ClickCease, TrafficGuard) if your revenue depends on accurate click data.
HTML Creative Sanitisation
Article 2 noted that HTML creatives require sanitisation before rendering. Article 3 sandboxed them in an iframe. Sandboxing alone is not sufficient - the allow-popups sandbox permission we specified permits popups to escape the sandbox, which is necessary for click-through but also means a malicious creative could open arbitrary windows.
Sanitise HTML creatives on save, not on render. The right moment to strip dangerous content is when the creative is created or edited; rendering should trust what is in the database:
// src/lib/server/ads/sanitise.ts
/**
* Sanitise HTML creative markup on the server.
*
* Uses the Web Standard DOMParser available in Node 18+.
* For Node 16 or non-standard environments, swap for the
* `sanitize-html` npm package with equivalent options.
*
* Rules:
* - Allowed elements: a, img, p, div, span, strong, em, br, h1–h6
* - Allowed attributes: href (a), src/alt/width/height (img), class, style
* - Strip everything else: script, iframe, object, embed, form, input
* - Enforce https:// on all href and src attributes
*/
export function sanitiseHtmlCreative(markup: string): string {
// Parse into a document fragment
const doc = new DOMParser().parseFromString(`<body>${markup}</body>`, 'text/html')
sanitiseNode(doc.body)
return doc.body.innerHTML
}
const ALLOWED_TAGS = new Set([
'a',
'img',
'p',
'div',
'span',
'strong',
'em',
'br',
'h1',
'h2',
'h3',
'h4',
'h5',
'h6',
'ul',
'ol',
'li'
])
const ALLOWED_ATTRS: Record<string, string[]> = {
a: ['href', 'class', 'style'],
img: ['src', 'alt', 'width', 'height', 'class', 'style'],
'*': ['class', 'style']
}
function sanitiseNode(node: Element | Document): void {
const children = Array.from(node.childNodes)
for (const child of children) {
if (child.nodeType === Node.TEXT_NODE) continue
if (child.nodeType !== Node.ELEMENT_NODE) {
// Remove comments, processing instructions, etc.
node.removeChild(child)
continue
}
const el = child as Element
const tag = el.tagName.toLowerCase()
if (!ALLOWED_TAGS.has(tag)) {
// Replace the element with its children (unwrap rather than remove)
while (el.firstChild) node.insertBefore(el.firstChild, el)
node.removeChild(el)
continue
}
// Strip disallowed attributes
const allowed = [...(ALLOWED_ATTRS[tag] ?? []), ...(ALLOWED_ATTRS['*'] ?? [])]
const attrNames = Array.from(el.attributes).map((a) => a.name)
for (const attr of attrNames) {
if (!allowed.includes(attr)) {
el.removeAttribute(attr)
}
}
// Enforce https:// on href and src
for (const attr of ['href', 'src']) {
const value = el.getAttribute(attr)
if (value && !value.startsWith('https://') && !value.startsWith('/')) {
el.removeAttribute(attr) // remove unsafe URLs entirely
}
}
// Recurse into allowed children
sanitiseNode(el)
}
} Call this in the creative form action before persisting:
// In the create/update creative form action:
import { sanitiseHtmlCreative } from '$lib/server/ads/sanitise'
// After Valibot validation passes:
if (result.output.content.type === 'html') {
result.output.content.markup = sanitiseHtmlCreative(result.output.content.markup)
} DOMParser in Node.js
DOMParseris available natively in Node 18.7+ (via theundiciglobals) and in all modern edge runtimes. For Node 16 or environments where it is absent, thesanitize-htmlpackage is a battle-tested alternative with a declarative allowlist API. The principle is identical; the implementation differs.
SSRF Prevention for Impression Pixels
Ad creatives can include a third-party impression tracking URL (impressionUrl). This URL is fetched server-side to notify the advertiser’s analytics platform. Without validation, an attacker can set the impression URL to an internal service address - http://169.254.169.254/metadata (AWS instance metadata), http://localhost:5432 (your database), or any other internal endpoint - turning your tracking server into an SSRF (Server-Side Request Forgery) proxy.
The fix is to validate the impression URL before storing it and before fetching it:
// src/lib/server/ads/validate-url.ts
const BLOCKED_HOSTS = [
'localhost',
'127.0.0.1',
'::1',
'0.0.0.0',
'169.254.169.254', // AWS / Azure instance metadata
'metadata.google.internal', // GCP instance metadata
'100.100.100.200' // Alibaba Cloud metadata
]
const BLOCKED_PRIVATE_RANGES = [
/^10\./,
/^172\.(1[6-9]|2[0-9]|3[0-1])\./,
/^192\.168\./,
/^fc[0-9a-f]{2}:/i, // IPv6 ULA
/^fe80:/i // IPv6 link-local
]
/**
* Validate that a URL is safe to fetch server-side.
* Returns the validated URL string, or throws if the URL is unsafe.
*/
export function validateExternalUrl(raw: string): string {
let parsed: URL
try {
parsed = new URL(raw)
} catch {
throw new Error(`Invalid URL: ${raw}`)
}
// Only allow HTTPS
if (parsed.protocol !== 'https:') {
throw new Error('Only HTTPS URLs are allowed for impression pixels')
}
const hostname = parsed.hostname.toLowerCase()
// Block known internal hostnames
if (BLOCKED_HOSTS.includes(hostname)) {
throw new Error(`Blocked host: ${hostname}`)
}
// Block private IP ranges
for (const pattern of BLOCKED_PRIVATE_RANGES) {
if (pattern.test(hostname)) {
throw new Error(`Private IP ranges are not allowed: ${hostname}`)
}
}
// Prevent URL redirects by removing any query params that look like redirect chains
// (This is defence-in-depth; the main protection is blocking private ranges above)
return parsed.toString()
} Use this in the creative validation schema:
// In src/lib/server/validation/ads.ts - update CreativeSchema:
import { validateExternalUrl } from '$lib/server/ads/validate-url'
// Replace the raw url() check on impressionUrl with a custom validator:
impressionUrl: v.optional(
v.pipe(
v.string(),
v.url('Must be a valid URL'),
v.transform((url) => {
try {
return validateExternalUrl(url)
} catch (err) {
throw new Error((err as Error).message)
}
})
)
) And when you fetch the pixel server-side, validate again at the call site:
// When dispatching impression pixels from the tracker:
import { validateExternalUrl } from './validate-url'
if (creative.impressionUrl) {
try {
const safeUrl = validateExternalUrl(creative.impressionUrl)
// Use a short timeout - third-party pixels should be fast
await fetch(safeUrl, {
method: 'GET',
signal: AbortSignal.timeout(2_000)
})
} catch (err) {
// Log but do not crash the tracker - a failed pixel is not critical
console.warn('[tracker] Third-party pixel failed:', err)
}
} GDPR Compliance Patterns
If your site serves users in the EU, collecting impression and click data requires a lawful basis under GDPR. For most ad systems, that basis is consent. This means you must not record any personalised events until the user has explicitly opted in.
The implementation has three parts: a consent signal, a gating function, and data retention controls.
Consent Signal in the Ad Context
Extend the AdRequestContext from Part 3 to carry the user’s consent status:
// In src/lib/types/ads.ts - add to AdRequestContext:
export interface AdRequestContext {
// ... existing fields
/** Whether the user has consented to personalised ad tracking. */
hasTrackingConsent: boolean
} Read the consent signal from a cookie set by your CMP (Consent Management Platform):
// In src/lib/server/ads/context.ts - update extractAdContext:
export function extractAdContext(event: RequestEvent, placement: string): AdRequestContext {
const { cookies } = event
// Read consent cookie set by your CMP (OneTrust, Cookiebot, etc.)
// The specific cookie name and format depend on your CMP vendor.
const consentCookie = cookies.get('consent_preferences')
const hasTrackingConsent = consentCookie
? parseConsentCookie(consentCookie).analytics === true
: false
return {
// ... existing fields
hasTrackingConsent
}
}
function parseConsentCookie(value: string): { analytics: boolean } {
try {
const parsed = JSON.parse(decodeURIComponent(value))
return { analytics: Boolean(parsed?.analytics) }
} catch {
return { analytics: false }
}
} Gating Event Recording
In the tracker, skip personalised data if consent is absent:
// In tracker.ts - update trackEvent:
export async function trackEvent(
input: TrackEventInput & { hasConsent: boolean }
): Promise<AdEvent | null> {
// Without consent, record the event with anonymised data only.
// Do not store session ID or user ID - those are personal data under GDPR.
const event: AdEvent = {
id: crypto.randomUUID(),
type: input.type,
campaignId: input.campaignId,
adGroupId: input.adGroupId,
creativeId: input.creativeId,
placementId: input.placementId,
// Anonymise: null out identifiers if no consent.
// sessionId is set to the literal string 'anonymous' rather than null or
// a generated ID. This is a deliberate choice: a fixed string means all
// non-consenting sessions look the same in the database, preventing any
// cross-session analysis on anonymous data. The side effect is that
// frequency capping is effectively pooled for all anonymous users - they
// all share the same Redis key. In practice, without a stable session ID,
// per-user frequency capping is not possible anyway, so this is acceptable.
userId: input.hasConsent ? input.userId : null,
sessionId: input.hasConsent ? input.sessionId : 'anonymous',
timestamp: new Date(),
metadata: input.hasConsent ? input.metadata : {}
}
await db.insert(adEvents).values(event)
emitAdEvent(event)
return event
} Frequency capping without consent falls back to a degraded mode - you cannot track per-session impressions, so you either skip capping or cap at a very conservative threshold across all anonymous users:
// In resolver.ts - Stage 4:
if (!context.hasTrackingConsent) {
// No consent: skip frequency caps entirely (or apply a global cap)
// All candidates remain in contention; skip the Redis lookup
const underCap = withinBudget
// ... continue to Stage 5
} else {
// Consent given: normal frequency cap logic
} Data Retention
GDPR requires you to delete personal data when it is no longer needed for the purpose it was collected. For analytics, a typical retention window is 13 months. Add a scheduled task to the lifecycle runner:
// In src/lib/server/ads/lifecycle.ts - add:
import { lt } from 'drizzle-orm'
/**
* Delete ad events older than the retention window.
* Run once daily (midnight is ideal; the scheduler in Part 6 can call this).
*/
export async function pruneExpiredEvents(retentionDays: number = 395): Promise<number> {
const cutoff = new Date()
cutoff.setDate(cutoff.getDate() - retentionDays)
const result = await db
.delete(adEvents)
.where(lt(adEvents.timestamp, cutoff))
.returning({ id: adEvents.id })
const count = result.length
if (count > 0) {
console.log(`[lifecycle] Pruned ${count} events older than ${retentionDays} days`)
}
return count
} Add it to the scheduler in scheduler.ts:
// In tasks array - add:
{
name: 'prune-expired-events',
intervalMs: 24 * 3_600_000, // once per day
lastRun: 0,
fn: () => pruneExpiredEvents(395) // ~13 months
} GDPR Is Not a CheckboxThis section shows the technical implementation of consent gating and data retention. It is not legal advice. GDPR compliance also requires a privacy notice explaining what data you collect and why, a mechanism for users to request deletion of their data, and documentation of your data processing activities. Consult a legal professional before launching an ad system in markets with privacy regulation.
The Hardened System Architecture
Here is how the production-ready layers fit on top of the system we built across the series:
Series Complete
Across seven articles, this series has taken a dynamic ad campaign system from domain model to production deployment:
Part 1 built the data foundation: TypeScript types, Drizzle ORM schema, Valibot validation, seed data, and the mental model of how ad systems work.
Part 2 built the management layer: SvelteKit form actions, Valibot validation on the server, status machines, progressive enhancement, and the nested form patterns needed for complex targeting rules.
Part 3 built the resolution engine: targeting filters, budget checking, frequency caps, weighted random selection, the AdSlot component with IntersectionObserver impression tracking, and server-side caching.
Part 4 built the analytics layer: event tracking, in-memory pub/sub, SSE streaming, metrics aggregation, reactive charts, and a live event feed.
Part 5 built the experimentation layer: deterministic bucket assignment, per-variant metrics, chi-squared statistical significance, Wilson confidence intervals, and automated winner selection.
Part 6 built the operational layer: probabilistic budget pacing, daily budget resets, campaign lifecycle automation, server-side scheduling, and budget forecasting.
Part 7 closed the gap between working and production-ready: database indexes, Redis-backed frequency capping, rate limiting, HTML sanitisation, SSRF prevention, and GDPR compliance.
The architecture is extensible. Authentication and multi-tenant access control, viewability measurement beyond IntersectionObserver, advanced fraud detection, CDN-level ad serving, and real-time bidding integrations are natural next steps - each built on the same foundation.