Gen Studio
Created during a two-day hackathon, Gen Studio uses Microsoft AI to visually and creatively navigate art collections at the Metropolitan Museum of Art (The Met).
Explore Gen Studio
The need
The Met collaborated with Microsoft and MIT to explore how AI could connect people to art. The goal was to imagine new ways for global audiences to discover, learn, and create with one of the world's foremost art collections.
The idea
The team started with two questions: Can we leverage Generative Adversarial Networks (GAN) to recombine artwork in new, interactive ways? If so, can we combine this with visual search to allow everyone to explore the collection?
The solution
Gen Studio was created, allowing users to explore dreamlike images—created by a GAN—and generated by AI. Gen Studio allows us to not just create random works, but to interpolate between real artworks in the collection.
Immerse yourself in The Met collection
Gen Studio allows you to explore, search, and be immersed in The Met’s collection. Find an inspiring piece, then explore related works through immersive visual search—or recombine artwork into new experiences.
Technical details for Gen Studio
Our first visualization lets users explore a two-dimensional slice of the vast “latent” space of the GAN. Users can move throughout this space and see how the GAN’s dreams change as they bump into real pieces in The Met’s collection. Our second visualization gives the user precise control of how to blend different works together into a larger work. Gen Studio shows the inferred visual structure underlying The Met’s collection, allowing explorers to create and recombine artwork that draw from a variety of styles, materials, and forms.
To create this experience, we used a microservice architecture of deep networks, Azure services, and blob storage. We used Visual Studio Code to develop a Flask API to serve the GAN from an Azure Kubernetes Service (AKS) cluster powered by Nvidia GPUs. These services make it possible to generate new images in real-time. Azure Kubernetes Service streamlines the path to production, making it possible to quickly deploy, host, and scale the solution.
Our GAN generates images from an initial ‘seed’ or vector of 140 numbers. A core challenge we faced was how to map images from The Met to a seed that generates it. To overcome this, we used gradient-descent-based network inversion, to learn the seeds for each image. The key was instructing the network to not just match the pixels of the target image, but also its high-level characteristics and content.
We loaded the Open Access images into an Azure Databricks cluster. We used Microsoft Machine Learning for Apache Spark (MMLSpark), to enrich these images with annotations from the Azure Computer Vision API. We then built a fast visual-similarity search by featuring all images with ResNet50, and constructing a locally sensitive hash tree on these features for approximate nearest neighbor lookup. We deployed this model onto AKS, and then used MMLSpark to add these nearest neighbors to our search index. We wrote the data from our Spark cluster to the Azure Search Service. The front-end was built using React and hosted in Azure using our App Service.
Resources
JFK Files
JFK Files takes 34,000 complex files including photos, handwriting, government documents, and more, then extracts readable information. This knowledge is organized to enable new ways to explore the information.
Snow Leopard Trust
The Snow Leopard Trust uses AI to study snow leopards in the wild. Camera traps capture hundreds of thousands of photos of rare snow leopards; machine learning and cognitive services are used to accelerate photo analysis and to map areas of activity.
Clean Water AI
Clean Water AI uses deep learning to detect dangerous bacteria and particles in water. The device analyzes drinking water with real-time detection and contamination mapping.
Angel Eyes
Angel Eyes is an IOT device that monitors a baby’s sleeping position and environment. Caregivers can view a live stream from anywhere and receive notifications if the device detects any issues.
Explore the possibilities of AI
Jumpstart your own AI innovations with learning resources and development solutions from Microsoft AI.
Innovation Tech Hub
Explore insights and behind-the-scenes technology for breakthrough AI innovations. From Tech Minutes videos to Technology Deep Dives, learn about the engineering that powers the future of AI.
AI School
Learn to create your own AI experiences with learning paths in conversational AI, machine learning, AI for devices, cognitive services, autonomous systems, AI strategy, and more.
AI platform
Start building AI solutions with powerful tools and services. Microsoft AI is a robust framework for developing AI solutions in conversational AI, machine learning, data sciences, robotics, IoT, and more.
