Pull to refresh
Logo
Amazon SageMaker adds catalog-wide data quality checks and DuckDB 2.0 notebooks

Amazon SageMaker adds catalog-wide data quality checks and DuckDB 2.0 notebooks

New Capabilities

AWS moves governance from single tables to whole catalogs, and lets project teams deploy approved resources without an administrator

Overview

Updated 41 minutes ago

Amazon Web Services added three things to SageMaker Unified Studio on November 20: blueprints that let a project team deploy approved resources such as Amazon EMR Serverless applications on its own, DuckDB 2.0 inside notebooks, and data quality checks that run across an entire catalog or database instead of one table at a time. The same day, AWS Glue Data Catalog began selling those catalog-wide checks as a standalone feature. Three days earlier, the studio's catalog gained centralized metadata management and the ability to hand an asset to whichever team knows it best.

The unit of governance is what moved. A data quality rule used to attach to a table, so every new table needed its own setup. One configuration now covers a database or a catalog, including tables added later, and triggers can fire on a schedule or whenever table metadata changes, down to each Apache Iceberg commit. A blueprint, meanwhile, is a template an administrator writes in infrastructure-as-code that defines which services a project may use. Teams provision from the approved list instead of filing a request.

Why it matters

Catalog-wide checks mean data quality stops depending on whether someone remembered to configure each new table.

Questions about this story

Free account needed to ask — your question is kept and asked for you right after sign-up. Answers are public.

No questions yet — be the first to ask.

Key Indicators

3
Quality runs before anomaly flags
After three runs on a table, AWS Glue Data Quality flags statistics that drift outside expected bounds.
1
Configuration per catalog or database
A single data quality configuration covers every table in scope, including tables added later.
24 hours
Federated catalog refresh
Tables in a newly created federated catalog land in the business catalog automatically every 24 hours.
2.0
DuckDB version in SageMaker notebooks
Notebooks now run DuckDB 2.0 with its extension library, including Excel, geospatial and full-text search functions.

Voices

Curated perspectives — historical figures and your fellow readers.

Sign up to generate historical perspectives on this story.

Organizations Involved

Timeline

May 2026 November 2026

4 events Latest: Today
Tap a bar to jump to that date
  1. Blueprints, DuckDB 2.0 and catalog-scale quality arrive

    Today Product release

    Projects can deploy approved resources such as Amazon EMR Serverless applications through resource blueprints. Notebooks run DuckDB 2.0 and its extensions. Teams can profile whole catalogs and databases.

  2. Glue Data Catalog adds catalog and database monitoring

    Today Product release

    One configuration profiles every table in a catalog or database, including tables added later, with triggers on a schedule or on metadata changes such as each Apache Iceberg commit.

  3. Catalog behaves the same in every domain type

    Product release

    Administrators gained one place to manage metadata forms, glossaries and custom asset types. Asset owners can transfer an AWS Glue asset to the project that knows the data best.

  4. Unified Studio gains data quality rules

    Product release

    SageMaker Unified Studio added rule authoring and evaluation for catalog tables and Visual ETL jobs, with on-demand or scheduled runs and per-rule pass/fail results.

Scenarios

1

Catalog-wide quality checks become the default for new projects

Possible Resolves by Q2 2027

Discussed by: AWS product documentation and release notes, which describe catalog and database scope as the new configuration level

If catalog-level profiling works at scale, AWS would likely turn it on by default when a domain or project is created, rather than leaving it as an opt-in from the catalog page. That would make a baseline quality score exist for every table the moment it is registered, with no per-table decision. The trigger to watch is whether new-project setup documentation starts listing quality configuration as part of provisioning.

2

DuckDB support spreads from notebooks into visual data pipelines

Possible Resolves by Q2 2027

Discussed by: AWS documentation and third-party SageMaker release trackers that already list DuckDB operators in serverless and visual workflows

Notebook support for DuckDB 2.0 is the narrowest form of the change. If AWS documents DuckDB as an engine inside Visual ETL jobs, analysts who never open a notebook get the same query speed over Amazon S3 files, and the notebook becomes one surface among several rather than the only one.

3

Asset ownership transfer reaches beyond AWS Glue tables

Uncertain Resolves by Q3 2027

Discussed by: AWS release notes, which currently limit ownership transfer to AWS Glue assets in the same AWS account

Delegation only helps if it covers the data a team actually uses. Today transfer applies to AWS Glue assets. Extending it to assets from other connected sources would let a project own and approve access to its own Redshift or S3-backed data instead of routing every request through the admin project.

Historical Context

3 moments from history that rhyme with this story — and how they unfolded.

May 2026

Per-table data quality rules in Unified Studio (2026)

SageMaker Unified Studio added data quality rule authoring and evaluation powered by AWS Glue Data Quality, covering data at rest in catalog tables and data moving through Visual ETL jobs. Users could run checks on demand or on a schedule and read per-rule pass or fail results.

Then

Quality checks had to be scoped to specific tables or jobs, so coverage depended on someone setting each one up.

Now

That per-table model became the baseline the November release widens to catalogs and databases.

Why this matters now

The November 20 change keeps the same engine but moves the configuration point up a level, from table to catalog.

2026

Custom blueprints turn project setup into infrastructure as code (2026)

AWS let administrators define SageMaker Unified Studio blueprints as AWS CloudFormation templates carrying their own IAM policies and internal standards. Instead of assembling project structures by hand, teams spin up environments that match company requirements from the start.

Then

Blueprints became the place where an organization encodes which services a project may touch.

Now

That template model made it possible to hand provisioning decisions to the project itself, within limits an administrator sets.

Why this matters now

The improved blueprints announced November 20 let a project deploy approved resources such as Amazon EMR Serverless applications without waiting for an administrator.

2018-2026

Open table formats move governance into metadata (2018-2026)

Apache Iceberg stores a table's state as metadata written alongside the data files, so every change is a recorded commit rather than an edit someone has to notice later. Analytics engines read that metadata to plan queries without scanning everything.

Then

Query engines could read a consistent table snapshot without locking or copying files.

Now

Governance tools gained a reliable event to hang work on, because a commit is a specific, timestamped change.

Why this matters now

AWS Glue's new monitoring can trigger an evaluation on every Apache Iceberg commit, which is only possible because the format records those commits.

Sources

(5)