Tag Archives: technology

The Magic of Generative AI

8 May

“The Magic of Generative AI” is still my favorite talk I’ve ever given, hands down. I loved collaborating with Google’s top AI minds on the story and the visuals, building demos that showed how Vertex AI helps marketers like me, and connecting with fellow AI enthusiasts in awesome places like LA and Rome.

But the best part was diving deep into how large language models (LLMs) actually work, reading those mind-bending research papers, and piecing together the “magic” they create. Preparing this talk was like living Google’s innovation mantra: stay curious, experiment, build something useful.

In this newsletter, I’m sharing my reflections on the magic of Gen AI and how Google’s unique innovation culture was key to making these incredible tools a reality.

Innovation💡 = Curiosity🧐 + ExperimentationđŸ§Ș + Application 🚀

Curiosity, experimentation, and application: This is the heart of how Google is driving the generative AI revolution. It’s the same formula behind some of our biggest breakthroughs, like Google Search, Translate, and Vertex AI.

Here’s how it works:

  • Curiosity: This is where it all starts – that burning question of “what if?” or “why not?” Curiosity is what drives us to explore the unknown and challenge the status quo.
  • Experimentation: Curiosity without action is just daydreaming. Experimentation is where we get our hands dirty, trying new things, making mistakes, and learning from them. It’s the messy but essential part of the process.
  • Application: The ultimate goal of innovation is to create something that makes a real difference in the world. Application takes those wild ideas and experiments and turns them into practical solutions that people can use and benefit from.

This isn’t just a theory; it’s the blueprint behind Google’s most groundbreaking AI tools.

Embeddings in Google Search: Grasp query intent beyond exact keywords

In 2013, Google researchers authored the seminal paper “Efficient Estimation of Word Representations in Vector Space“. This paper unveiled a revolutionary method for creating Word Embeddings, mathematical representations of words capturing both their meaning (semantics) and relationships (semantic similarity). Here’s the Google’s innovation formula in action:

  • Curiosity: Dissatisfied with existing word organizational methods, such as dictionaries ordering words by lexicographical order, researchers were curious if a better approach could capture word semantics and organize them by semantic meaning.
  • Experimentation: They explored various neural network types, training objectives and relationship representations. Through experimentation, they discovered how to automatically create a word embedding. A name to be remembered, an embedding is a mathematical representation for each word that captures their semantic meaning in the form of a vector of 768 numbers.
  • Application: Way before it was applied in gen AI, word embeddings found a magical application in semantic search, enabling Google Search 🔍 to grasp query intent beyond exact keywords. For example, a search for “cars that are good on gas” now returns results for fuel-efficient cars, even if the word “gas” doesn’t appear in the options returned.
Source: “The Magic of Generative AI” talk, Google Gen AI Live and Labs event series

Transformer in Google Translate: More accurate translations

In 2017, Google researchers presented “Attention is All You Need” introducing the Transformer architecture, built on decision-making and attention-span concepts. It empowers the language models to understand context and relationships within word sequences. Curiosity, experimentation and application were again vital:

  • Curiosity: In the search to improve the quality of language translation, researchers sought ways to model relationships among words in a sentence.
  • Experimentation: They experimented with various mechanisms, relationship representations and training methods, discovering that much could be extracted by simply paying attention to the relationship between each word and every other word in a sentence. They discovered that these interdependencies could be achieved through parallel computations, which accelerated time to result, and found that representations through embeddings could capture long-range dependencies between words in fluent, grammatically correct text. VoilĂ ! The Transformer architecture was born, introducing a huge breakthrough in science.
  • Application: The transformer revolutionized Google Translate 🌐. The Transformer’s attention mechanisms are excellent at understanding the relationships between words in a sentence, leading to more accurate translations.
Source: Transformers, FT,

Let’s see this in action by translating this sentence from English to Italian: “The cat didn’t cross the street because it was too wide.

Source: “The Magic of Generative AI” talk, Google Gen AI Live and Labs event series

Gen AI in Enterprise Search: New way of working

Fast forward to 2023, Google Cloud researchers set to simultaneously tackle two common challenges for many organizations:

  • How to organize enterprises information scattered across many internal systems
  • How to make this information accessible and useful for enterprises, and seamlessly available and actionable in applications such as customer service bots, document summarizations or as part of steps in automated workflows.

Not surprisingly, Google Cloud researchers followed the proven innovation framework:

  • Curiosity: While Google Search was designed to scale to organize the world’s information, researchers started exploring whether the technology could be descaled and made available to enterprises to organize their information in a way that could be easily accessible and useful to them, and only to them.
  • Experimentation: Intrigued by the potential to bring together several cutting-edge technologies, researchers used the ability to crawl web-sites to discover content on internal websites and structured content, and Optical Character Recognition (OCR) to discover content from all sorts of semi-structured and unstructured documents, creating a wealth of knowledge about the enterprise. The researchers then used embeddings to extract and organize the semantic meaning of all of this data. Once enterprise’s data has been semantically organized in embeddings, the full power of Generative AI can be applied to it and leveraged across the Vertex AI platform.
  • Application: First launched in March 2023, Google Cloud Vertex AI Search 🔍 quickly became “the killer enterprise app”. A killer application, often abbreviated as killer app, is a software application that is so necessary or desirable that it proves the core value of some larger technology, such as its video game console, software platform, or in this case of gen AI in the enterprise context. Killer apps are the pinnacle of innovation: well-designed, easy to use and solving a real problem for users. Enterprise Search is the killer enterprise app as it unlocks unprecedented levels of productivity and efficiency.

These transformative breakthroughs exemplify Google’s dedication to AI innovation, with continued explorations on the horizon.

Ready to experience the magic of Gen AI? Explore Gemini today: https://gemini.google.com/

#GoogleAI #GenerativeAI #Innovation #ArtificialIntelligence #SemanticSearch #NLP #AIInnovation #TransformerArchitecture #ApplicationsOfAI

Tags: , , , ,

Blending Science and Art: The Multimodal Craft of an Exceptional Gen AI Paper

5 Apr
With the entire text of Les Misérables in the prompt (1382 pages), Gemini 1.5 Pro locates a famous scene from a hand-drawn sketch

Technical writing is one of my favorite reads. It’s clear, succinct, and informative. DeepMind’s technical paper on Gemini 1.5 epitomizes all I love about technical writing. Read the abstract for a glimpse into the groundbreaking advancements encapsulated in Gemini 1.5 Pro; it’s a masterclass in effective communications. We learn how to deliver maximum insight with minimum word count.

In just 177 words, my DeepMind colleagues articulate:

  • #ProductCapabilities: “a highly compute-efficient multimodal* mixture-of-experts model** capable of recalling and reasoning*** over fine-grained information from millions of tokens of context”
  • #UniqueSellingPoint: “near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k)”
  • #UseCases: “surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content”
Gemini 1.5 Pro is able to translate from English to Kalamang with similar quality to a human

The science of writing succinctly

In a few words, the paper abstract communicates the model’s superior performance, its leap over existing benchmarks, and its novel capabilities. It sparks curiosity about the future potentials of large language models—a true testament of powerful, precise, impactful technical communication.

How did the Gemini 1.5 paper authors achieve this mastery? By following the guiding principles of Brevity (saying more with fewer words) that my friend and thought partner D G McCullough and I recently summarized as: “Trust, Commit, Distill”:

  • #Trust means believing in the power of your message without over-explaining nor adding unnecessary details. Trust empowers the communicator to eliminate redundancy, focusing on what’s truly important. The Gemini 1.5 paper authors trust their curious readers to look up terms that may be new to them. On first read, I had to look up “mixture-of-experts” but the context I’ve had from my 2 years of working with data and AI allowed me to “guesstimate” its meaning before getting the proper definition.
  • #Commit refers to sticking with the essentials of your message, understanding your message’s objective, and resisting tangents or unnecessary explanations diluting the message’s impact. (Which requires discipline!)
  • #Distill requires breaking down your message to full potency. Like distilling a liquid to increase its purity, we must strip away the non-essential until the most impactful, clear, and concise message remains. Every word and idea then serves a purpose–and voila! Your message becomes clearer, and more memorable.

The art of replacing 100s of words with a single image

The saying “A picture is worth a thousand words” truly shines in technical communication. A single, well-chosen image can articulate complex ideas with more efficiency and impact than verbose descriptions. The Gemini 1.5 paper’s authors skillfully weave in visual elements, showcasing a deep grasp of conciseness. This approach not only makes complex AI and machine learning concepts approachable and captivating but also boosts understanding and enhances the reader’s journey. It demonstrates that when it comes to sharing the latest scientific breakthroughs, visual simplicity can convey a wealth of information.

With the entire text of Les Misérables in the prompt (1382 pages), Gemini 1.5 Pro locates a famous scene from a hand-drawn sketch

Simplify complexity with brevity

In our rapid world, where attention is a rare commodity and people often skim rather than read, the skill of conveying ideas briefly and through visual storytelling stands out as a significant edge. Simplifying complex concepts into engaging visuals and concise explanations can mean the difference between being noticed or ignored.

Richard Feynman, the celebrated physicist, Nobel laureate, and cherished educator, famously stated, “If you can’t explain it simply, you don’t understand it well enough.”

Richard Feynman quotes

Feynman’s approach isn’t just about words; it involves using visuals and images to make intricate ideas more approachable. After all, the deepest insights are usually the easiest to understand when we apply brevity to break down complexity.

DeepMind’s Gemini 1.5 technical paper exemplifies this principle perfectly. It’s essential reading for anyone intrigued by general AI (especially with #GoogleCloud #NEXT24 on the horizon), and it’s an exemplary model for those dedicated to honing their communication skills.

#TechnicalWriting #Innovation #ArtificialIntelligence #LanguageModels #Brevity #BrevityRules #GoogleCloud #NEXT24 #DeepMind

Read the full abstract

“In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra’s state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro’s long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.” https://storage.googleapis.com/deepmindmedia/gemini/gemini_v1_5_report.pdf

Define the key terms used in the abstract

* #Multimodality: Gemini is natively multimodal.  Prior to Gemini, AI models were first trained on a single modality, such as text, or image, and then corresponding embeddings were concatenated. For example, the embedding of an image would be generated by an AI model trained on images, the embedding of the text describing the image would be generated by an AI model trained on texts, and then the two embeddings would be concatenated to represent the image and its transcript. Instead, the Gemini family of models was trained on content that is inherently multimodal such as text, images, videos, code, and audio. Imagine being able to ask a question about a picture, or generate a poem inspired by a song – that’s the power of Gemini.

** #Mixture-of-Experts Model: At the core of Gemini’s groundbreaking capabilities lies its innovative mixture-of-experts model architecture. Unlike traditional neural networks that route all inputs through a uniform set of parameters, the mixture-of-experts model consists of numerous specialized sub-networks, each adept at handling different types of information or tasks—these are the “experts.” Upon receiving an input, a gating mechanism intelligently directs the input to the most relevant experts. This selective routing allows the model to leverage specific expertise for different aspects of the input, akin to consulting specialized departments within a larger organization for their unique insights. For Gemini, this means an unparalleled ability to process and integrate a vast array of multimodal data—whether it’s textual, visual, auditory, or code-based—by dynamically engaging the most suitable experts for each modality. The result is a model that not only excels in its depth and breadth of understanding but also in computational efficiency, as it can focus its processing power where it matters most, without overburdening the system with irrelevant data processing. This approach revolutionizes how AI models handle complex, multimodal inputs, enabling more nuanced interpretations and creative outputs than ever before.

A Mixture of Experts (MoE) layer embedded within a recurrent language model https://openreview.net/pdf?id=B1ckMDqlg

*** #Reasoning: Gemini goes beyond simple pattern recognition. It utilizes a novel architecture called “uncertainty-routed chain-of-thought” to reason and understand complex relationships within and across modalities. This enables it to answer open-ended questions, solve problems, and generate creative outputs that are not just factually accurate but also logically coherent.

Tags: , , , ,

Newer Entries →