Building better security and data governance without AWS Lake Formation

Building better security and data governance without AWS Lake Formation can still be effectively achieved by leveraging other AWS services and best practices. Here’s how you can ensure robust security and governance for your data lake in S3, while using Glue Data Catalog, Athena, and Redshift:

Key Security and Governance Practices:

1. S3 Security and Access Control

• S3 Bucket Policies: Implement fine-grained access control using bucket policies. These policies can restrict access based on IAM roles, users, or conditions (e.g., IP address, encryption).
  • Example

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/DataScientistRole"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

• IAM Policies: Use IAM roles and policies to control access to S3, Athena, Glue, and Redshift. Grant least privilege, ensuring users or applications can only access the data and services they need.
  • Example: An IAM role for a data analyst may only have permissions to run Athena queries but not modify Glue Data Catalog tables.
• S3 Object-Level Permissions: Control access at the object level within S3. This is useful when different datasets in the same bucket need different access levels.
• Server-Side Encryption: Enable S3 encryption for data at rest. You can use either SSE-S3 (Amazon-managed keys) or SSE-KMS (AWS Key Management Service with your own keys).
  • Using SSE-KMS allows for additional control over key rotation, auditing, and access permissions.
• Client-Side Encryption: If more security is needed, you can implement client-side encryption where data is encrypted before being uploaded to S3.

2. Athena Query Access Control

• Restrict Query Access via IAM: Define IAM policies that control which Athena queries users or roles can run. You can limit users to specific databases, tables, or even specific columns.
  • Example: A policy that allows access only to specific Athena tables

{
  "Effect": "Allow",
  "Action": "athena:StartQueryExecution",
  "Resource": "arn:aws:athena:region:account-id:workgroup/MyWorkGroup"
}

AWS Resource Access Manager (RAM): Use RAM to share Athena query results securely with other AWS accounts while maintaining control over who can access the query results.

Athena Workgroups: Configure workgroups to manage user access, set query limits, and monitor usage. Workgroups allow you to enforce different query settings (e.g., query timeout, data access restrictions).

Data Masking and Row-Level Security: Implement data masking or row-level security by controlling access to specific sensitive columns or rows via views or selective queries.

• Example: Create views that mask sensitive columns (e.g., masking customer credit card information).

CREATE VIEW customer_view AS
SELECT id, name, 'XXXX-XXXX-XXXX' AS masked_credit_card, email
FROM customers;

3. Glue Data Catalog Governance

• Restrict Glue Catalog Access: Control access to the Glue Data Catalog by restricting who can read, update, or delete metadata definitions. Use IAM policies to manage this access.
• Tagging: Use AWS Glue Catalog Tags to label datasets with metadata that can later be used for governance and auditing purposes. Tags can also help in setting up resource-based access controls.
  • Example: Tag sensitive datasets with PII=True to denote personally identifiable information.
• Data Catalog Encryption: Encrypt metadata stored in the Glue Data Catalog by enabling AWS KMS for encryption.
• Auditing Access to Glue Data Catalog: Enable AWS CloudTrail to track who accessed, modified, or deleted catalog objects. This provides an audit trail for governance and security.

4. Redshift Security

• Network Security (VPC): Place your Redshift cluster inside a VPC (Virtual Private Cloud) to control access to the cluster using VPC security groups and network access control lists (ACLs).
• Redshift User Access Control: Use Redshift user roles and groups to manage fine-grained access to tables, columns, and schemas. You can restrict certain users to specific schemas or tables using GRANT statements.
  • Example

GRANT SELECT ON TABLE orders TO analyst_role;

• Encryption: Use KMS to encrypt data stored in Redshift, both at rest and in transit. Redshift supports SSL encryption for data in transit and KMS encryption for data at rest.
• Data Auditing: Enable Redshift auditing by logging all connections, user activity, and queries. This can be useful for governance and detecting unauthorized access.

5. Auditing and Monitoring

• CloudTrail: Enable AWS CloudTrail to audit and monitor all API activity on services like S3, Athena, Glue, and Redshift. This will help track who accessed what data and when.
• AWS Config: Use AWS Config to track the configuration of your resources, ensuring compliance with security policies. You can set up rules to detect configuration drift, such as unencrypted S3 buckets or improper IAM roles.
• Amazon CloudWatch and GuardDuty: Use CloudWatch for monitoring and Amazon GuardDuty for threat detection. GuardDuty helps identify unusual or malicious activity in S3 or across your AWS environment.

6. Implement Fine-Grained Data Access Control

• Column-Level Access Control: You can create views that only expose certain columns to certain users. This ensures sensitive data is not exposed inadvertently.
• Row-Level Security: For more granular control, you can implement row-level security by using views or filters based on user roles or data ownership (tenant-based access control).
• Tag-Based Access Control: Implement tag-based access control on resources, using AWS IAM policies to control access based on the tags applied to your S3 objects, Glue tables, or Redshift schemas.

7. Best Practices for Data Governance

• Data Classification and Tagging: Classify your data based on sensitivity levels (e.g., confidential, public, PII) and tag it accordingly in Glue and Redshift. This classification helps in enforcing security and governance policies.
• Data Retention and Lifecycle Policies: Implement S3 lifecycle policies to manage the retention and archiving of data. Ensure that sensitive data is retained for only as long as necessary to comply with data protection regulations.
• Access Reviews: Regularly review IAM roles and policies to ensure that data access permissions are current and that no excessive permissions are granted to users.
• Data Lineage: Track data lineage using Glue to understand how data flows through your pipeline from S3 to Redshift. This is crucial for governance, compliance, and ensuring data accuracy.

Summary:

Even without AWS Lake Formation, you can implement robust security and governance for your data lake and data warehouse using these best practices:

• S3 bucket policies, IAM roles, and encryption for securing your data lake.
• Athena workgroups, row-level security, and data masking for securing query access.
• Glue Data Catalog policies, tagging, and encryption for managing metadata securely.
• Redshift user access controls, VPC security, and encryption for data warehousing security.
• Auditing and monitoring using CloudTrail, CloudWatch, and GuardDuty for compliance and governance.

These practices will ensure that your system is secure, compliant, and governed even without Lake Formation.