------------------------------
Title: Generating Texts using Files Uploaded by Gemini 1.5 API
[content]
AbstractThe Gemini API allows the generating of text from uploaded files using Google Apps Script. It expands the potential of various scripting languages for diverse applications.IntroductionWith the release of the LLM model Gemini as an API on Vertex AI and Google AI Studio, a world of possibilities has opened up. Ref The Gemini API significantly expands the potential of various scripting languages and paves the way for diverse applications. Also, recently, Gemini 1.5 in AI Studio has been released. Ref In the near future, Gemini 1.5 API will be also released soon.Recently, the files got to be able to be uploaded with Gemini API. Ref1 and Ref2 When this is used, the text is generated using the uploaded files. This report introduces the sample scripts for uploading the files and generating the texts using Google Apps Script.UsageIn order to test this script, please do the following steps.1. Create an API keyPlease access https://makersuite.google.com/app/apikey and create your API key. At that time, please enable Generative Language API at the API console. This API key is used for this sample script.This official document can be also seen. Ref.2. Create a Google Apps Script projectIn this report, Google Apps Script is used. Of course, the method introducing this report can be also used in other languages.Please create a standalone Google Apps Script project. Of course, this script can be also used with the container-bound script.And, please open the script editor of the Google Apps Script project.3. Steps of scriptHere, it introduces the following 4 sample scripts.Upload a file with “Method: files.list”.Confirm the uploaded file with “Method: media.upload”.Generate content using the uploaded file with “Method: models.generateContent”.Delete the uploaded file with “Method: files.delete”.Limitation of the uploaded file RefCan only be used with model.generateContent or model.streamGenerateContent Automatic file deletion after 2 days Maximum 2GB per file, 20GB limit per project No downloads allowed4. ScriptUpload a fileYou can see the official document at Method: media.upload. The sample script of Google Apps Script is as follows.Please set your API key and the file ID of the image file. Here, PNG image is used.function sample1() {  const apiKey = "###"; // Please set your API key.  const fileId = "###"; // Please set the file ID of the image file. Here, PNG image is used.  const url = `https://generativelanguage.googleapis.com/upload/v1beta/files?uploadType=multipart&key=${apiKey}`;  const metadata = {    file: { displayName: DriveApp.getFileById(fileId).getName() },  };  const payload = {    metadata: Utilities.newBlob(JSON.stringify(metadata), "application/json"),    file: UrlFetchApp.fetch(      `https://drive.google.com/thumbnail?sz=w1000&id=${fileId}`,      { headers: { authorization: "Bearer " + ScriptApp.getOAuthToken() } }    ).getBlob(),  };  const options = {    method: "post",    payload: payload,    muteHttpExceptions: true,  };  const res = UrlFetchApp.fetch(url, options).getContentText();  console.log(res);}When this script is run, the following value is returned.{  "file": {    "name": "files/###",    "displayName": "###",    "mimeType": "image/jpeg",    "sizeBytes": "123456",    "createTime": "2024-03-30T01:23:00.000000Z",    "updateTime": "2024-03-30T01:23:00.000000Z",    "expirationTime": "2024-04-01T01:23:00.000000Z",    "sha256Hash": "###",    "uri": "https://generativelanguage.googleapis.com/v1beta/files/###"  }}The values of mimeType and uri are used with generateContent.In this sample, I used uploadType=multipart because of the small size of the image file. If you want to upload a large file, I think that resumable upload can be also used.Get the file listYou can see the official document at Method: files.list. The sample script of Google Apps Script is as follows.Please set your API key.function sample2() {  const apiKey = "###"; // Please set your API key.  const url = `https://generativelanguage.googleapis.com/v1beta/files?pageSize=100&key=${apiKey}`;  const res = UrlFetchApp.fetch(url);  console.log(res.getContentText());}When this script is run, the following value is returned.{  "files": [    {      "name": "files/###",      "displayName": "###",      "mimeType": "image/jpeg",      "sizeBytes": "123456",      "createTime": "2024-03-30T01:23:00.000000Z",      "updateTime": "2024-03-30T01:23:00.000000Z",      "expirationTime": "2024-04-01T01:23:00.000000Z",      "sha256Hash": "###",      "uri": "https://generativelanguage.googleapis.com/v1beta/files/###"    },    ,    ,    ,  ]}When the number of files is more than 100, please retrieve all files using pageToken.Generate content from the uploaded fileYou can see the official document at Method: models.generateContent. The sample script of Google Apps Script is as follows.Please set your API key, the URI of the uploaded file, and the mimeType of the file.function sample3() {  const apiKey = "###"; // Please set your API key.  const fileUri = "https://generativelanguage.googleapis.com/v1beta/files/###"; // Please set your file uri of the uploaded file.  const mimeType = "image/jpeg"; // Please set the mimeType of the uploaded file.  const q = "Describe the image and count apples in the image.";  const model = "models/gemini-1.5-pro-gf-fc";  const baseUrl = `https://generativelanguage.googleapis.com/v1beta/${model}`;  const payload = {    contents: [{ parts: [{ text: q }, { fileData: { fileUri, mimeType } }] }],  };  const options = {    payload: JSON.stringify(payload),    contentType: "application/json",    muteHttpExceptions: true,  };  const res = UrlFetchApp.fetch(    `${baseUrl}:generateContent?key=${apiKey}`,    options  );  console.log(res.getContentText());}In this sample, the following image created by Gemini was uploaded as a sample file and was used.There are 12 apples including 7 red apples, 4 green apples, and 1 yellow apple are shown in the image. This image was generated by Gemini.When this script is run, the following generated contents are returned.At the model models/gemini-1.0-pro-latest, Image input modality is not enabled for models/gemini-1.0-pro-latest was returned.At the model models/gemini-1.0-pro-vision-latest, There are 10 apples in the image. Four red, five green, and one yellow. was returned.At the model models/gemini-1.5-pro-latest, The image shows a group of apples on a wooden table. There are red, green, and yellow apples. There are 14 apples in total. was returned.At the model models/gemini-1.5-pro-gf-fc, The image shows a group of apples on a wooden table. There are 13 apples in total. The apples are of different colors, including red, green, and yellow. The apples are arranged in a random pattern on the table. The light is coming from the left side of the image, and it is casting shadows on the apples and the table. was returned.Delete the uploaded fileYou can see the official document at Method: files.delete. The sample script of Google Apps Script is as follows.Please set your API key and the name of the uploaded file.function sample4() {  const apiKey = "###"; // Please set your API key.  const name = "files/###"; // Please set the name of the uploaded file.  const url = `https://generativelanguage.googleapis.com/v1beta/${name}?key=${apiKey}`;  const res = UrlFetchApp.fetch(url, { method: "delete" });  console.log(res.getContentText()); // {}}In this case, an empty object like {} is returned.In the current stage, the expiration time of the uploaded file is 2 days. So, the uploaded file is automatically deleted 2 days later.SummaryIn this report, we present sample scripts for using the Gemini API’s generateContent function with uploaded files. Our findings are as follows:Uploading files, retrieving file lists, and deleting files all functioned smoothly using an API key. Also, I heard that at Google APIs, both “snake_case” and “camelCase” within the request body. This was confirmed through testing.For generating content from uploaded image files, Gemini 1.5 API models models/gemini-1.5-pro-latest and models/gemini-1.5-pro-gf-fc can be used for image analysis. However, accurately counting objects within the image might still be challenging.Currently, uploading text, CSV, and PDF files results in an error message like “Request contains an invalid argument.” It appears that only image and movie files are supported at this stage. We anticipate this limitation to be addressed in a future update.NoteThe top illustration was created by Gemini with giving the abstract.Generating Texts using Files Uploaded by Gemini 1.5 API was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
[/content]

Author: Kanshi Tanaike
PublishedDate: 2024-03-31
Category: Blogs
NewsPaper: Google Cloud - Medium
Tags: gemini, google-apps-script, generative-ai, google-cloud-platform, machine-learning