AWS Glue 5.0 now supports fine-grained access control (FGAC) by integrating with AWS Lake Formation, enabling granular permissions at the table, column, and row levels on data lake resources. This enhanced control helps organizations meet governance and compliance requirements, especially when handling sensitive datasets.

Lake Formation allows for defining and enforcing permissions through familiar SQL-like GRANT and REVOKE commands, ensuring seamless policy enforcement across services like Amazon Athena, Amazon EMR (Spark), and Redshift Spectrum. With Glue 5.0, these rules are now honored during AWS Glue Spark jobs and interactive sessions, streamlining data security across the analytics stack.

How FGAC Works in AWS Glue 5.0

Glue 5.0 introduces a dual-profile mechanism:

• User Profile: Executes your custom Spark code and builds the query plan. It has no direct S3 or Glue catalog access.
  
  
• System Profile: Uses a privileged role to enforce Lake Formation permissions, retrieve metadata, and launch executors.

The process:

1. A user invokes StartJobRun on a Glue job enabled for Lake Formation.
   
   
2. The user driver prepares the job plan without executing code.
   
   
3. A secure TLS channel is established to the system driver, running Spark with full permissions.
   
   
4. The system driver enforces access control, accesses data, and orchestrates executors.
   
   
5. Executors run portions of your code or policy-enforced data reads according to FGAC rules.
   
   

This separation ensures user code cannot bypass access restrictions. Only the system profile can query tables with sensitive data.

Enabling FGAC in AWS Glue Jobs

To activate FGAC in AWS Glue 5.0 ETL jobs:

1. In the AWS Glue Console, select your ETL job and ensure the Glue Version is set to Glue 5.0 – Supports Spark 3.5, Scala 2, Python 3.

Under Job parameters, add:

 ini
CopyEdit
--enable-lakeformation-fine-grained-access = true

2. Save the configuration.

For interactive notebooks, use the configuration cell:

python

CopyEdit

%%configure

{"--enable-lakeformation-fine-grained-access":"true", "glue_version":"5.0"}

 

Example Use Case: CSV and Iceberg Tables

The blog demonstrates a typical scenario:

1. S3 bucket creation with CSV dataset.
2. Data Catalog tables defined for both CSV and Iceberg formats via AWS Glue and Athena CTAS.
3. Lake Formation permissions applied to enforce FGAC across both table types.
4. Glue PySpark jobs executed to read data with FGAC enforced, and write filtered results back to S3.

The sample dataset includes:

• product_id
• category
• product_name
• quantity_available
• last_update_time
• op (operation type: I/U/D)

After setting up Lake Formation’s row- and column-level filters, two Glue jobs demonstrate compliance with access policies using PySpark scripts.

Step-by-Step Implementation

Prerequisites

Before starting, ensure:

• AWS account with IAM roles to manage S3, Glue, Catalog, and Lake Formation
• Lake Formation is configured, and you're an administrator or have Data Lake admin permissions
• Working in a supported AWS region (example: eu-west-1)

Implementation Outline

1. S3 Bucket and Dataset
   
   
   • Upload input CSV data.
     
     
2. Data Catalog Tables
   
   
   • Register a standard table and an Iceberg table via Athena CTAS.
     
     
3. Lake Formation Policies
   
   
   • Define row and column filters for both CSV and Iceberg tables to enforce FGAC.
     
     
4. Glue Job Configuration
   
   
   • Enable the --enable-lakeformation-fine-grained-access
     
     
   • Run Glue PySpark scripts that respect Lake Formation rules.
     
     
5. Result Validation
   
   
   • Output from the jobs confirms only permitted data is read or written as per policies.

Conclusion

With AWS Glue 5.0, integrating Lake Formation's FGAC brings unified, secure data access to your Spark ETL pipelines. The mechanism ensures:

• Centralized policy enforcement across services
• Transparent, secure separation between user and system execution profiles
• Full FGAC support on table formats like CSV, Iceberg, and beyond

Adopting this integration enhances your data lake governance, ensures compliance, and minimizes risk when working with sensitive data.