Amazon Enhances Data Discovery and Governance in SageMaker with Integrated Catalogs

Enterprises frequently encounter significant hurdles when data assets are created and managed outside of centralized data catalogs, leading to inefficiencies in data discovery and limiting collaborative efforts across teams. Amazon's Business Data Technologies (BDT) team has been actively working to overcome these challenges by integrating their enterprise data catalog, Andes, with Amazon SageMaker. This strategic move aims to provide a unified platform for discovering and governing a wide array of data assets. Historically, teams within Amazon often maintained separate catalogs for local datasets and non-tabular assets like dashboards and metrics, distinct from the central Andes catalog. This fragmentation made it difficult for users to gain a comprehensive view of available resources, forcing them to navigate multiple systems and spend considerable time identifying the correct catalog for their tasks. This process not only slowed down operations but also diverted valuable time from core business problem-solving. To address these issues, the BDT team identified four critical capabilities essential for an improved data experience. Firstly, a multimodal catalog was needed to allow data consumers to seamlessly blend enterprise data with local datasets and other asset types, such as metrics and business files, within a single, consolidated location. Secondly, uniform governance and enforcement were crucial to ensure consistent data protection practices across all compute engines, enabling users to request access once and have it enforced universally. Thirdly, extended observability, facilitated by Trusted Identity Propagation (TIP) through AWS IAM Identity Center, allows users to access data interactively using their corporate identities. Finally, the goal was to achieve single-pane asset discovery, making all datasets and data assets accessible through one unified view. By extending Andes with Amazon SageMaker, Amazon is enhancing the data discovery experience. SageMaker's native support for multimodal catalogs and its integration with enterprise identity management make it an ideal foundation for expanding Andes' governance model. This approach centralizes and synchronizes data assets within a single enterprise-wide domain, eliminating the need to broadcast assets across disparate systems. The integration ensures that governance of both enterprise-wide and local datasets is orchestrated through a single system, leading to a more streamlined and secure data environment.