[PAID] :google:emini Extension to interact with the Gemini-pro and PaLM 2 models from :google:oogle

Gemini

The Gemini extension for AI2 interacts with Google Gemini-Pro & Gemini-Pro-Vision models (used by Bard) for text generation and streaming.

Features of the Gemini Extension for AI2:

  • Gemini API Text Generation: Use various Gemini models.
  • Image Generation: Create new images from text prompts.
  • Image Editing: Modify existing images via text (path/Base64).
  • Streaming Text Generation: Real-time, interactive responses.
  • Vision Support: Text from images, video thumbnails, PDFs (URL).
  • Code Execution (Optional): Run code within API responses.
  • Structured JSON Output: Define schemas for predictable results.
  • File Handling: Base64 encoding (optimized/standard), path/URI handling, MIME detection.
  • PaLM API Integration: Basic text generation via PaLM.
  • List Models: Fetch available Gemini model names.
  • Stream Control: Start/Stop streaming.
  • Error Handling: Events for API/file/JSON errors, stream completion/stop.
  • Asynchronous Operations: Keeps UI responsive.

Benefits:

  • Integrate advanced AI text/multimodal generation into AI2 apps easily.
  • Enhance user engagement with streaming & interactive features.
  • Generate diverse content (text, code, based on images/videos/PDFs).
  • Reliably process/generate structured data using JSON Schema.
  • Work seamlessly with local files for richer AI interactions.
  • Flexibility: Choose between Gemini and PaLM APIs/models.
  • Easy to use in App Inventor, extensible.

Use Cases:

  • Chatbots: Understand text/images for context-aware conversations.
  • Content Tools: Generate articles, posts, stories (with/without image prompts).
  • Media Analysis: Analyze images/videos, generate descriptions.
  • Doc Processing: Process PDFs from URLs for summaries/Q&A.
  • Code Tools: Generate/assist code (optional execution).
  • Data Structuring: Extract info into structured JSON via schemas.
  • Edu/Creative: Interactive learning, story generation.
    The potential is vast!

Blocks









Explanation


Generating Content (Non-Streaming)

2|690x

Use GenerateGeminiContent block.

  • modelName (String): Gemini model (e.g., “gemini-1.5-flash”). See docs.
  • apiKey (String): Your Google API key.
  • contents: List of conversation turn dictionaries (role, parts list with text).

Blocks example:
Screenshot 2023-12-16 235605|690x




Screenshot 2023-12-16 235605|690x

RespondedToGemini event handles the response.

  • apiResponse: Raw API response.
  • textParts: List of generated text strings.
  • role: Response role.
  • finishReason: Why generation stopped.
  • index: Content index.
  • safetyRatings: List of safety rating dicts (category, probability).




Function: StreamGenerateGeminiContent

Stream content using Gemini API, with optional Code Execution.

Parameters:

Blocks example
Screenshot 2023-12-16 235605

Functionality: Streams responses via SSE. Triggers GotGeminiStream (text/code chunks), StreamFinished (completion), ErrorOccurred (errors).
Callbacks: GotGeminiStream(textValue), StreamFinished(), ErrorOccurred(errorMessage, component).
Notes: For streaming text/code. Handles multi-turn/images. Use events for results/errors. Needs internet & API key.





GotGeminiStream` event.

Screenshot 2023-12-16 235605|624x
Triggered with each chunk of streamed data.

  • text: Generated text chunk (String).

Screenshot 2023-12-16 235605|508x
StopStream block manually stops. StoppedStream event fires when stopped.

Screenshot 2023-12-16 235605|522x
IsStreaming block checks if a stream is active.





blocks(3)

Function: GenerateGeminiThinkingContent

Generate content using Gemini 1.5 Flash model (non-streaming).

  • prompt (String): Text prompt.
  • apiKey (String): Google API key.

Triggers GotGeminiStream with full response or ErrorOccurred.





blocks(2)

Function: StreamGenerateGeminiThinkingContent

Stream content using Gemini 1.5 Flash model. Retrieves in chunks.

  • prompt (String): Text prompt.
  • apiKey (String): Google API key.

Triggers GotGeminiStream, StreamFinished, and ErrorOccurred.




Function: StreamGenerateContentFromPdfUrl

Stream content based on a PDF file from a URL.

  • pdfUrl (String): URL of PDF.
  • prompt (String): Text prompt related to PDF.
  • apiKey (String): Google API key.
  • modelName (String): Gemini model (e.g., “gemini-pro-vision”).

Handles download, upload, streaming. Triggers GotGeminiStream, StreamFinished, ErrorOccurred.





Function: StreamGenerateGeminiStructuredContent

-----------------------

Stream structured JSON content matching a schema.

  • contents (List): Conversation turns.
  • apiKey (String): Google API key.
  • modelName (String): Gemini model (e.g., “gemini-pro”).
  • scheme (String): JSON Schema string (use CreateJsonSchema).

Triggers GotGeminiStream (JSON chunks), StreamFinished, ErrorOccurred.




Function: CreateJsonSchema

Builds a JSON Schema string.

  • propertyNames (List): Property names.
  • propertyTypes (List): Corresponding types (“string”, “number”, etc.).
  • propertyDescriptions (List): Property descriptions.
  • requiredProperties (List): Required property names.

Returns formatted JSON Schema string or empty string on error (triggers ErrorOccurred).




25588|690x
Generating Content with Images (Streaming)
Use StreamGenerateGeminiVisionContent.

  • contents: List of dicts (can include text and inlineData parts with image Base64).
  • apiKey: Google API key.

Blocks example:
Screenshot 2023-12-16 235605



Screenshot 2023-12-16 235605
Results arrive via the GotGeminiStream event.



4|690x
StreamGenerateGeminiFileContentFromBase64

Streams content based on Base64 files and text.

  • apiKey (String): Google API key.
  • modelName (String): Gemini model. See docs.
  • fileBase64List (List): Base64 file strings.
  • mimeTypeList (List): Corresponding MIME types.
  • additionalText (String): Text prompt.



GenerateImage

Creates image from text.

  • prompt (Text): Description.
  • apiKey (Text): Google API Key.
  • modelName (Text): Image generation model. Check Google docs.

EditImage

Modifies image from Base64.

  • prompt (Text): Instructions.
  • inputImageBase64 (Text): Base64 image string.
  • inputMimeType (Text): Input MIME type.
  • apiKey (Text): Google API Key.
  • modelName (Text): Image editing model.

EditImageFromPath

Modifies image from file path.

  • prompt (Text): Instructions.
  • inputImagePath (Text): Path to image file.
  • apiKey (Text): Google API Key.
  • modelName (Text): Image editing model.

EditMultipleImagesSimple

Advanced edit/gen using multiple images (URL/Path/Base64).

  • prompt (Text): Instructions.
  • imageSourceStrings (List): Image sources.
  • apiKey (Text): Google API Key.
  • modelName (Text): Multi-image model.


DisplayBase64Image

Helper to display Base64 on Image component.

  • base64Data (Text): Base64 data.
  • mimeType (Text): Image MIME type.
  • imageComponent (Component): Target Image component.

component_event

Event: GotImageResponse

Fires on successful image task completion.

  • imageBase64 (Text): Result image Base64.
  • mimeType (Text): Result MIME type.
  • responseText (Text): API text response.
  • rawApiResponse (Text): Full JSON response.
  • imagePath (Text): Path where result was saved (ASD).

Examples of generating and editing with Gemini


(upload://2O0F26UstT6K9AAcJOOUHWWdv87.jpeg)



  • StreamGenerateContentFromLocalVideoPath
    • Parameters:
      • videoPath (String): The local file path to a video file.
      • prompt (String): The text prompt related to the video content.
      • apiKey (String): Your Google AI API Key.
      • modelName (String): The Gemini model to use.
      • systemInstructionsValue (String): Optional system instructions.
      • jsonSchemaString (String): Optional JSON schema for structured output.
    • Description: Uploads a local video file using the File API, polls until the file is processed (“ACTIVE”), and then starts a streaming request based on the video content and prompt. Optionally includes system instructions and/or requests structured output via a JSON schema. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.

  • StreamGenerateContentFromLocalVideoPathWithInstructions
    • Parameters:
      • videoPath (String): The local file path to a video file.
      • prompt (String): The text prompt related to the video content.
      • apiKey (String): Your Google AI API Key.
      • modelName (String): The Gemini model to use.
      • systemInstructionsValue (String): Optional system instructions.
    • Description: Similar to StreamGenerateContentFromLocalVideoPath, but only includes the option for system instructions (no structured output schema). Uploads the video, waits for processing, then starts the streaming request. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure. Uses standard Designer Properties for generation config.

  • StreamGenerateContentFromYouTubeUrl (Overload 1 - Basic)
    • Parameters:
      • youtubeUrl (String): Public URL of a YouTube video (including Shorts).
      • prompt (String): Text prompt relating to the video.
      • apiKey (String): Your Google AI API Key.
      • modelName (String): The Gemini model to use.
    • Description: Starts a streaming analysis request using a YouTube URL and prompt. Uses default generation settings from Designer Properties. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.

  • StreamGenerateStructuredContentFromYouTubeUrl (Overload 2 - Advanced)
    • Parameters:
      • youtubeUrl (String): Public URL of a YouTube video (including Shorts).
      • prompt (String): Text prompt relating to the video.
      • apiKey (String): Your Google AI API Key.
      • modelName (String): The Gemini model to use.
      • systemInstructionsValue (String): Optional system instructions.
      • jsonSchemaString (String): Optional JSON schema for structured output.
    • Description: Starts a streaming analysis request using a YouTube URL and prompt. Optionally includes system instructions and/or requests structured output via a JSON schema. Response chunks arrive via GotGeminiStream. Triggers StreamFinished when done or ErrorOccurred on failure.



5|690x
GetGeminiModelNames

Retrieves available Gemini model names.

  • apiKey (String): Google API key.

Events:
6|690x0

  • GotGeminiModelNames(modelNames as List): Triggered on success with list of names.



7|568x

  • ErrorOccurred(message, component): Triggered on API request error.



Screenshot 2023-12-16 235605|690x
Encoding Images to Base64
EncodeImageToBase64 block encodes image path to Base64 (no line breaks).

  • imagePath: Path to image file.

Returns Base64 string.


Error Handling
Screenshot 2023-12-16 235605|568x0
ErrorOccurred event signals errors.

  • message: Error description.
  • component: Component source of error.

Examples

Generate text (non-streaming):
Screenshot 2023-12-16 235605|690x



Generate text (streaming):
Screenshot 2023-12-16 235605|690x

Generate text with images (streaming):
Screenshot 2023-12-16 235605|690x

Generate text with images in FreeForm Prompt (streaming):
Use TextFormater extension for layout.
Screenshot 2023-12-16 235605|690x

Freeform preview:


Applications using this extension:

Videos preview:

Aix_file:

Compare PAID vs FREE versions :point_down:

PAID_file
Price: $5.99
Purchase: PayPal Link or You can pay HERE using your credit card or You can pay HERE using your credit card
In both cases, you will be redirected to the download page after successful payment. Contact me for issues.

FREE_file
Gemini.aix (11.6 KB)

Have Inquiries?
Contact via PM on Telegram.

Note :

Get your Gemini API key from Google AI Studio.

4 Likes

Wow Nice Extension and nice work :two_hearts::sparkling_heart:

1 Like

Can you please get back to me on the features i asked for because i have a deadline and i don’t know if you’re working on it or not, kindly respond asap. Thanks

1 Like

Thank you very much :dizzy:

check your PM please

PaLM 2 NewBlocks added

1 Like

I purchased the Gemini extension for AI2 from the MIT APP Inventor community, but PayPal did not redirect me to download the file. The purchase was made on May 17th of this year. I have already left some messages in the MIT APP Inventor community. What is the procedure for me to download the file?

I think your problem solved right now
and thanks for your report.

New update for the Extension to meet the latest updates of the Gemini API .

Here’s a summary of the updates made to the Gemini.aix compared to the initial version.

1. Model Selection:

  • The GenerateGeminiContent, StreamGenerateGeminiContent, and functions now all accept a modelName parameter, allowing the user to specify which Gemini model to use for the request. This provides flexibility in choosing the appropriate model for different tasks.

2. StreamGenerateGeminiFileContentFromBase64 Function:

  • New Function: A new function called StreamGenerateGeminiFileContentFromBase64 has been added.



  • Base64 File Input: This function accepts a list of Base64 encoded files (fileBase64List) and a corresponding list of MIME types (mimeTypeList).

  • Generic File Handling: It handles various file types (not just images) by using the MIME type information.

  • Streaming Response: It uses streaming to receive the response from the Gemini API and triggers the GotGeminiStream event for each chunk of text received.

3. GetGeminiModelNames Function:

  • New Function: A new function called GetGeminiModelNames has been added.

5(1)



  • Retrieving Model Names: It retrieves a list of available Gemini model names from the API and triggers the GotGeminiModelNames event with the list.

6

4. GetFilePathFromDataURI Function:

  • New Function: A new function called GetFilePathFromDataURI has been added.

  • Data URI to File Path: It converts a Data URI (representing a file) to a local file path. It handles content://, file://, and data:// URI schemes.

5. getMimeType Function:

  • New Function: A new function called getMimeType has been added.

  • Get MIME Type: It takes a file path as input and returns the MIME type of the file using Files.probeContentType(path).

6. Code Cleanup and Improvements:

  • Removed Redundant Parameter: The contents parameter in the StreamGenerateGeminiVisionContentFromPathsAndText function was removed as it became unnecessary after adding separate parameters for images and text.

  • Error Handling: The code now includes more robust error handling, using try-catch blocks and triggering the ErrorOccurred event when necessary.

Overall, the updated code is more versatile, efficient, and user-friendly:

  • More Features: It provides functions to retrieve model names, handle various file types, and work with Data URIs.

  • Flexibility: Users can now choose specific Gemini models and send different file types to the API.

  • Efficiency: Streaming responses allow for better handling of large data.

  • Improved Usability: The code is more organized and includes better documentation and error handling.

These updates enhance the functionality and make the extension more useful for a wider range of applications within App Inventor.

2 Likes

Text response like this

Hi! \n\nHow can I help you today? \n

Why

Just enable that html text(something like that, can’t recall the exact thing) checkbox in properties. It’ll enable this and you’ll be able to see a new line instead of /n

As @anon92712962 said you can enable html

Same problem and

What is the problem
Do you mean in the code section
This is not from my extension code this Is from the API it self

When I use stream then I got, code and \n problem

It’s parfact with without streaming

I will check it and confirm what I got
And thanks for informing me

Problem solved now and thanks @AyanDeveloper for informing about this problem

1 Like

did you create this app with this UI design ?

1 Like

Yes i made using dynamic

congrats! This is a cool UI design that attracts my attention
what tools that you use to add code viewer in the dynamic component like in this image