What is Google Gemini?
Google’s large language model (LLM) is called Gemini. More generally, it’s a family of multimodal AI models capable of handling multiple modalities or types of data like audio, images, software code, text and video.
Much like Anthropic’s Claude, named for the chatbot and the family of LLMs that power it, Gemini is the name used for Google’s generative AI (gen AI) chatbot (formerly Bard) as well. Both web and mobile Gemini apps serve as a chat-bot front-end to the models.
Google is slowly embedding the Gemini chatbot into the range of technologies it offers. For example, Google Assistant will be replaced by Gemini as the default AI assistant on the newest Google Pixel 9 and Pixel 9 Pro phones. The Gemini feature is integrated into Google Workspace in Documents to help with writing and editing, as well as in Gmail to draft emails, suggest responses, and search a user’s Gmail inbox.
Google’s other applications are also integrating the Gemini feature. Google Maps, for instance, is utilizing the capabilities of the Gemini model to provide descriptions of locations and areas.
How does Google Gemini work?
A large multilingual and multimodal dataset has been used to train Gemini. Uses a transformer model, which is an architecture that Google introduced in 2017.
Here’s a brief overview of how transformer models work:
Encoders convert input sequences into numerical representations called embeddings, which represent the semantics and position of the tokens in the input sequence.
The self-attention mechanism lets transformers “focus their attention” on the most critical tokens of the input sequence, irrespective of their position.
This self-attention, along with the embeddings produced by the encoders, is used to produce the most likely output sequence using the decoder.
Google Gemini can accept interleaved sequences of audio, image, text and video as input, and interleaved text and image as output; this is unlike the generative pretrained transformer (GPT) models, which take only text-based prompts, and diffusion models used for image generation, which take both text and image prompts.
Gemini AI model versions
There are a few different variants of the Gemini family of multimodal AI models. Each variant is optimized for specific devices and functions.
Gemini’s first generation is called 1.0, and is available in both Nano and Ultra. 1.5 is the next generation model and is available in the Pro and Flash.
gemini’s AI features and functions can be built upon and experimented with using the Gemini API in the Google AI Studio and Google Cloud Vertex AI development platforms. Currently, there is no other plan for other variants except for Gemini 1.5 Pro and Gemini 1.5 Flash.
Gemini 1.0 Nano
The smallest member of the 1.0 family is Gemini 1.0 Nano, which runs on mobile devices even when there’s no data network. It can perform on-device tasks such as describe images, suggest replies to chat messages, summarize text and transcribe speech.
Starting with the Pixel 8 Pro, Gemini Nano is now on Android devices. Google is rolling out Gemini Nano to its Chrome desktop browser, bringing the new AI to its platform beyond just the mobile app.
Gemini 1.0 Ultra
Gemini 1.0 Ultra is the biggest member of the Gemini 1.0 family with enhanced analytical features. It’s designed for very sophisticated tasks, like coding, mathematical reasoning and multimodal reasoning. The context window—the number of tokens that a model can process at once—of both Gemini Nano and Gemini Ultra is 32,000 tokens.2
Gemini 1.5 Pro
Gemini 1.5 Pro is a large, multimodal model with a large 2 million token context window. This extended context window allows Gemini Pro to analyze information on a much larger scale, whether that’s hours of audio or video, thousands of lines of code, or hundreds of pages of documents.3
Besides using a transformer architecture, Gemini 1.5 Pro also uses Mixture of Experts (MoE) architecture. MoEs decompose a single neural network into smaller “expert” networks, with specific focus on different domains or data types. The model learns to selectively activate only the most relevant experts, based on the type of the input. This gives increased performance and cuts down on the computational expenses.4
Gemini 1.5 Flash
Gemini 1.5 Flash is a lightweight version of the Gemini Pro AI. It was fine-tuned with a machine learning (ML) method known as knowledge distillation, where the knowledge from Gemini 1.5 Pro was distilled into the smaller Gemini 1.5 Flash. It also supports a context window of up to 1 million tokens which is very long but at the same time, it has a lower latency, i.e., it is faster and more efficient.3
A short history of Google Gemini.
Google has been a leader in the field of LLM architecture, and they have also benefited from their extensive research to create their own AI models.
- 2017: Google researchers present the transformer architecture, which underpins many of today’s LLMs.
- 2020: The firm launches the Meena chatbot, an AI-powered conversational bot with 2.6 billion parameters, which is a neural network-based conversational bot.
- Google introduces its conversational LLM: Google’s LaMDA (Language Model for Dialogue Applications) is introduced in 2021.
- 2022: PaLM (Pathways Language Model) is released, which has more advanced capabilities than LaMDA.
- 2023: Bard starts during the first quarter of the year, backed by a lightweight and optimized version of LaMDA.8 The second quarter sees PaLM 2 released—with enhanced coding, multilingual and reasoning skills—and adopted by Bard.9 Google announces Gemini 1.0 in the last quarter of the year.
Google changes Bard to Google Gemini, and revamps the multimodal AI models in Bard to 1.5.Google announces that it has renamed Bard as Google Gemini and upgraded its multimodal AI models to 1.5.
The word “Gemini” translates to “twins” in Latin and is also a constellation and a Zodiac sign. It seemed like a fitting name, as the Gemini model is the result of a collaboration between the DeepMind and Google Brain teams—who join forces to create Google DeepMind. The company also also got ideas from NASA’s Project Gemini, a two-person spacecraft that was integral to the success of the Apollo mission.
Gemini performance
Compared to the other models that are comparable, Gemini Ultra performs better in a number of LLM benchmarks. In benchmarks like MMLU, Claude 2 outperforms GPT-4, Claude 3 and Llama 2 in natural language understanding; in GSM8K it does better than mathematical reasoning; and in HumanEval it is better than code generation.2
Interestingly, Gemini Ultra outperformed even human experts in MMLU. GPT-4, however, is still more capable than Gemini Ultra in the HellaSwag test, which measures common sense reasoning and natural language inference.2
Google also tested the multimodal capabilities of Gemini Ultra. It got better results than other models in document understanding, image understanding and automatic speech recognition benchmarks. And despite beating LLMs in benchmarks for automatic speech translation, English video captioning, multimodal understanding and reasoning, and video question answering, Gemini Ultra’s performance in these areas leave room for improvement.2
Gemini 1.5 Pro’s context window expands, and it stays at a high level of performance.4
Gemini use cases
Although Google Gemini is still in its infancy, this powerful AI model can be used in countless applications:
- Advanced coding
- Understanding of images and text.
- Language translation
- Malware analysis
- Personalized AI experts
- Universal AI agents
- Voice assistants
Advanced coding
The Gemini AI model can work across programming languages such as C++, Java and Python to understand, explain and generate code. Google used fine-tuned versions of Gemini Pro as foundation models to develop AlphaCode2, a code generation system that can solve competitive programming problems with elements of theoretical computer science and complex math.
Understands images and text.
It can be used to extract text from images and caption images using Gemini. It can process visual information like charts, diagrams and figures, even without using optical character recognition (OCR) software to convert images to machine-readable text.
Language translation
Google’s AI models are multilingual, which enables them to translate among various languages. For example, users can enable translated captions to translate between languages in Meet video conferencing.
Malware analysis
Both Gemini 1.5 Pro and Gemini 1.5 Flash can be employed for malware analysis. Gemini Pro can accurately identify malicious files or malicious code snippets and can report on the content of that finding.12 Gemini Flash can perform a fast malware dissection of large files and reports.13
Personalized AI experts
Google recently launched a new feature called Gems, which enables users to train the Gemini chatbot to become an AI “expert” in any task or topic. Examples of premade Gems include a learning coach for helping to simplify a complex idea for understanding, a brainstorming partner to bring new ideas to the next video, a writing editor for grammar and structure.
Gems are available for Gemini Advanced users using the Gemini 1.5 Pro model.
Universal AI agents
Google’s Project Astra aims to develop a single AI agent capable of real-time processing, memorization and comprehension of multimodal information, incorporating its Gemini models. To enhance recall and efficiency, Project Astra makes use of caching, continuous encoding of video frames and ties speech and video input together into a sequential stream of events.14
In one of Google’s demos, the Gemini AI assistant was able to explain the parts of a speaker, recognize the neighborhood a person was in and remember where they put their glasses.14
Voice assistants
In Gemini Live, users can chat with the Gemini chatbot as if it were a real person, making the interaction more human-like and engaging. It provides more natural and user-friendly replies and can adapt to an individual’s conversational pattern.
Gemini risks
As with other LLMs, AI is a danger for Google Gemini. It is advised caution be taken, particularly with any person planning to use Gemini or with organisations wanting to use the model for commercial or as part of their workflows.
Bias: In February 2024, Google decided to pause the Gemini chatbot’s ability to create images of people owing to its inaccurate portrayal of historical figures, erasing a history of racial bias.15
Hallucinations: At this time, AI overview search results from Gemini are sometimes yielding information that is factually incorrect.
