Coming in Flink 2.4: Filter Pushdown for the Savepoint Table Connector

By Gabor Somogyi, Apache Flink PMC member and Ilya Soin

---

Apache Flink’s State Processor API lets you treat a savepoint like a database: read it, analyze it, rewrite it — all from a regular batch job. Since Flink 2.1, the API has been accessible via the Table API, meaning you can query savepoint state with plain SQL. That already opened a lot of doors. The upcoming Flink 2.4 makes it significantly faster in the cases that matter most.

The problem with full scans

When you read keyed state through the Table API, Flink distributes the work across key groups — the internal shards that hold your state. Before 2.4, every query scanned every key group, even when your SQL filter made it obvious that only a tiny fraction of keys were relevant.

Imagine a savepoint with millions of user records. You want to inspect the state of one user:

SELECT * FROM savepoint_source WHERE key = 'user-42'

Previously, Flink would still open and iterate all key groups, check each key, and discard the ones that didn’t match. The filter ran late — after reading.

What changes in 2.4

The savepoint connector now supports filter pushdown. When you add a WHERE clause on the key column, Flink translates it into a SavepointKeyFilter before the scan begins. Depending on the predicate type, one or both of these optimizations apply:

1. Split pruning — for equality and IN predicates, Flink computes exactly which key groups own the requested keys and skips all other input splits entirely, never opening them.
2. Key-level filtering — for all supported predicates, within each opened split only matching keys are iterated rather than every key in the split.

The filter translation supports the expressions you’d naturally write:

SQL predicate	Optimization applied
key = 'user-42'	Split pruning + key filtering: only the split owning this key is opened
key IN ('a', 'b', 'c')	Split pruning + key filtering: only splits owning these keys are opened
key BETWEEN 1000 AND 1999	Key filtering only: all splits are opened, non-matching keys skipped
key >= 100 AND key < 200	Key filtering only: all splits are opened, non-matching keys skipped

Expressions that can’t be pushed down (e.g. conditions on non-key columns) are handled by the remaining post-scan filter as before, so nothing breaks.

A concrete example

Say you have a fraud detection job that checkpoints per-user risk scores. After an incident, you want to pull the state for a specific cohort of flagged accounts:

SELECT key, risk_score, last_updated
FROM user_risk_state
WHERE key IN ('user-100', 'user-204', 'user-389')

Flink resolves the IN list into an exact key set, computes which key groups own those three keys, and opens only those splits. If your savepoint has 128 key groups, you might open 2 or 3 instead of all 128. For point lookups or small exact sets, the wall-clock time drops proportionally.

Range queries are also pushed down. If your keys are numeric customer IDs and you want a slice:

SELECT * FROM order_state WHERE customer_id BETWEEN 50000 AND 59999

Flink applies the bounds as a key-level filter during iteration: all splits are still opened, but non-matching keys are skipped without invoking the reader function. For large savepoints where only a fraction of keys fall in the range, this can still significantly reduce CPU and I/O on the hot path.

Who benefits

Filter pushdown is most impactful when:

• You’re doing incident response or forensics on a specific subset of keys
• You’re migrating or validating a slice of state rather than the whole savepoint
• Your savepoint is large but your query is narrow

If you’re already reading all state (no WHERE on the key column), behaviour is unchanged.

Availability

Filter pushdown for the savepoint connector is coming in Apache Flink 2.4. No configuration is needed — the optimizer picks it up automatically when your query includes a supported predicate on the key column. The existing Table API and DataStream API for savepoint reading remain fully compatible; this is a pure performance improvement on the read path.

If your team works with Flink savepoints today, this is worth watching. Targeted state queries that previously required reading the entire savepoint will have a direct, low-overhead path.