The Gemini extension for AI2 interacts with Google Gemini-Pro & Gemini-Pro-Vision models (used by Bard) for text generation and streaming.
Features of the Gemini Extension for AI2:
- Gemini API Text Generation: Use various Gemini models.
- Image Generation: Create new images from text prompts.
- Image Editing: Modify existing images via text (path/Base64).
- Streaming Text Generation: Real-time, interactive responses.
- Vision Support: Text from images, video thumbnails, PDFs (URL).
- Code Execution (Optional): Run code within API responses.
- Structured JSON Output: Define schemas for predictable results.
- File Handling: Base64 encoding (optimized/standard), path/URI handling, MIME detection.
- PaLM API Integration: Basic text generation via PaLM.
- List Models: Fetch available Gemini model names.
- Stream Control: Start/Stop streaming.
- Error Handling: Events for API/file/JSON errors, stream completion/stop.
- Asynchronous Operations: Keeps UI responsive.
Benefits:
- Integrate advanced AI text/multimodal generation into AI2 apps easily.
- Enhance user engagement with streaming & interactive features.
- Generate diverse content (text, code, based on images/videos/PDFs).
- Reliably process/generate structured data using JSON Schema.
- Work seamlessly with local files for richer AI interactions.
- Flexibility: Choose between Gemini and PaLM APIs/models.
- Easy to use in App Inventor, extensible.
Use Cases:
- Chatbots: Understand text/images for context-aware conversations.
- Content Tools: Generate articles, posts, stories (with/without image prompts).
- Media Analysis: Analyze images/videos, generate descriptions.
- Doc Processing: Process PDFs from URLs for summaries/Q&A.
- Code Tools: Generate/assist code (optional execution).
- Data Structuring: Extract info into structured JSON via schemas.
- Edu/Creative: Interactive learning, story generation.
The potential is vast!
Blocks
Explanation
Generating Content (Non-Streaming)
Use
GenerateGeminiContent
block.
modelName
(String): Gemini model (e.g., “gemini-1.5-flash”). See docs.apiKey
(String): Your Google API key.contents
: List of conversation turn dictionaries (role
,parts
list withtext
).Blocks example:
RespondedToGemini
event handles the response.
apiResponse
: Raw API response.textParts
: List of generated text strings.role
: Response role.finishReason
: Why generation stopped.index
: Content index.safetyRatings
: List of safety rating dicts (category
,probability
).
Function: StreamGenerateGeminiContent
Stream content using Gemini API, with optional Code Execution.
Parameters:
contents
: List of conversation turn dictionaries.apiKey
(String): Your Google API key.modelName
(String): Gemini model (e.g., “gemini-1.5-flash”). See docs.enableCodeExecution
(boolean): Enable code execution.
Blocks example
Functionality: Streams responses via SSE. Triggers
GotGeminiStream
(text/code chunks),StreamFinished
(completion),ErrorOccurred
(errors).
Callbacks:GotGeminiStream(textValue)
,StreamFinished()
,ErrorOccurred(errorMessage, component)
.
Notes: For streaming text/code. Handles multi-turn/images. Use events for results/errors. Needs internet & API key.
GotGeminiStream` event.
Triggered with each chunk of streamed data.
text
: Generated text chunk (String).
StopStream
block manually stops.StoppedStream
event fires when stopped.
IsStreaming
block checks if a stream is active.
Function: GenerateGeminiThinkingContent
Generate content using Gemini 1.5 Flash model (non-streaming).
prompt
(String): Text prompt.apiKey
(String): Google API key.Triggers
GotGeminiStream
with full response orErrorOccurred
.
Function: StreamGenerateGeminiThinkingContent
Stream content using Gemini 1.5 Flash model. Retrieves in chunks.
prompt
(String): Text prompt.apiKey
(String): Google API key.Triggers
GotGeminiStream
,StreamFinished
, andErrorOccurred
.
Function: StreamGenerateContentFromPdfUrl
Stream content based on a PDF file from a URL.
pdfUrl
(String): URL of PDF.prompt
(String): Text prompt related to PDF.apiKey
(String): Google API key.modelName
(String): Gemini model (e.g., “gemini-pro-vision”).Handles download, upload, streaming. Triggers
GotGeminiStream
,StreamFinished
,ErrorOccurred
.
Function: StreamGenerateGeminiStructuredContent
-----------------------Stream structured JSON content matching a schema.
contents
(List): Conversation turns.apiKey
(String): Google API key.modelName
(String): Gemini model (e.g., “gemini-pro”).scheme
(String): JSON Schema string (useCreateJsonSchema
).Triggers
GotGeminiStream
(JSON chunks),StreamFinished
,ErrorOccurred
.
Function: CreateJsonSchema
Builds a JSON Schema string.
propertyNames
(List): Property names.propertyTypes
(List): Corresponding types (“string”, “number”, etc.).propertyDescriptions
(List): Property descriptions.requiredProperties
(List): Required property names.Returns formatted JSON Schema string or empty string on error (triggers
ErrorOccurred
).
Generating Content with Images (Streaming)
UseStreamGenerateGeminiVisionContent
.
contents
: List of dicts (can includetext
andinlineData
parts with image Base64).apiKey
: Google API key.Blocks example:
Results arrive via theGotGeminiStream
event.
StreamGenerateGeminiFileContentFromBase64Streams content based on Base64 files and text.
apiKey
(String): Google API key.modelName
(String): Gemini model. See docs.fileBase64List
(List): Base64 file strings.mimeTypeList
(List): Corresponding MIME types.additionalText
(String): Text prompt.
GenerateImageCreates image from text.
prompt
(Text): Description.apiKey
(Text): Google API Key.modelName
(Text): Image generation model. Check Google docs.
EditImage
Modifies image from Base64.
prompt
(Text): Instructions.inputImageBase64
(Text): Base64 image string.inputMimeType
(Text): Input MIME type.apiKey
(Text): Google API Key.modelName
(Text): Image editing model.
EditImageFromPath
Modifies image from file path.
prompt
(Text): Instructions.inputImagePath
(Text): Path to image file.apiKey
(Text): Google API Key.modelName
(Text): Image editing model.
EditMultipleImagesSimple
Advanced edit/gen using multiple images (URL/Path/Base64).
prompt
(Text): Instructions.imageSourceStrings
(List): Image sources.apiKey
(Text): Google API Key.modelName
(Text): Multi-image model.
DisplayBase64ImageHelper to display Base64 on Image component.
base64Data
(Text): Base64 data.mimeType
(Text): Image MIME type.imageComponent
(Component): Target Image component.
Event: GotImageResponse
Fires on successful image task completion.
imageBase64
(Text): Result image Base64.mimeType
(Text): Result MIME type.responseText
(Text): API text response.rawApiResponse
(Text): Full JSON response.imagePath
(Text): Path where result was saved (ASD).
Examples of generating and editing with Gemini
(upload://2O0F26UstT6K9AAcJOOUHWWdv87.jpeg)
- StreamGenerateContentFromLocalVideoPath
- Parameters:
- videoPath (String): The local file path to a video file.
- prompt (String): The text prompt related to the video content.
- apiKey (String): Your Google AI API Key.
- modelName (String): The Gemini model to use.
- systemInstructionsValue (String): Optional system instructions.
- jsonSchemaString (String): Optional JSON schema for structured output.
- Description: Uploads a local video file using the File API, polls until the file is processed (“ACTIVE”), and then starts a streaming request based on the video content and prompt. Optionally includes system instructions and/or requests structured output via a JSON schema. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.
- StreamGenerateContentFromLocalVideoPathWithInstructions
- Parameters:
- videoPath (String): The local file path to a video file.
- prompt (String): The text prompt related to the video content.
- apiKey (String): Your Google AI API Key.
- modelName (String): The Gemini model to use.
- systemInstructionsValue (String): Optional system instructions.
- Description: Similar to StreamGenerateContentFromLocalVideoPath, but only includes the option for system instructions (no structured output schema). Uploads the video, waits for processing, then starts the streaming request. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure. Uses standard Designer Properties for generation config.
- StreamGenerateContentFromYouTubeUrl (Overload 1 - Basic)
- Parameters:
- youtubeUrl (String): Public URL of a YouTube video (including Shorts).
- prompt (String): Text prompt relating to the video.
- apiKey (String): Your Google AI API Key.
- modelName (String): The Gemini model to use.
- Description: Starts a streaming analysis request using a YouTube URL and prompt. Uses default generation settings from Designer Properties. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.
- StreamGenerateStructuredContentFromYouTubeUrl (Overload 2 - Advanced)
- Parameters:
- youtubeUrl (String): Public URL of a YouTube video (including Shorts).
- prompt (String): Text prompt relating to the video.
- apiKey (String): Your Google AI API Key.
- modelName (String): The Gemini model to use.
- systemInstructionsValue (String): Optional system instructions.
- jsonSchemaString (String): Optional JSON schema for structured output.
- Description: Starts a streaming analysis request using a YouTube URL and prompt. Optionally includes system instructions and/or requests structured output via a JSON schema. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.
GetGeminiModelNamesRetrieves available Gemini model names.
apiKey
(String): Google API key.Events:
- GotGeminiModelNames(modelNames as List): Triggered on success with list of names.
- ErrorOccurred(message, component): Triggered on API request error.
Encoding Images to Base64
EncodeImageToBase64
block encodes image path to Base64 (no line breaks).
imagePath
: Path to image file.Returns Base64 string.
Error Handling
ErrorOccurred
event signals errors.
message
: Error description.component
: Component source of error.
Examples
Generate text (non-streaming):
Generate text (streaming):
Generate text with images (streaming):
Generate text with images in FreeForm Prompt (streaming):
Use TextFormater extension for layout.
Freeform preview:
Applications using this extension:
Videos preview:
Aix_file:
Compare PAID vs FREE versions
PAID_file
Price: $5.99
Purchase: PayPal Link or You can pay HERE using your credit card or You can pay HERE using your credit card
In both cases, you will be redirected to the download page after successful payment. Contact me for issues.
FREE_file
Gemini.aix (11.6 KB)
Have Inquiries?
Contact via PM on Telegram.
Note :
Get your Gemini API key from Google AI Studio.