Editing article

Title

Summary

Content

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iUeKnOWdjYkVi3BY.jpg" /></figure>As I was working on tweaking the Vertex AI text embedding model in <a href="https://github.com/langchain4j">LangChain4j</a>, I wanted to better understand how the textembedding-gecko<a href="https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text-embeddings">model</a> tokenizes the text, in particular when we implement the <a href="https://arxiv.org/abs/2005.11401">Retrieval Augmented Generation</a> approach.The various PaLM-based models offer a computeTokens endpoint, which returns a list of tokens (encoded in Base 64) and their respective IDs.<blockquote>Note: At the time of this writing, there’s no equivalent endpoint for Gemini models.</blockquote>So I decided to create a <a href="https://tokens-lpj6s2duga-ew.a.run.app/">small application</a> that lets users:<ul><li>input some text,</li><li>select a model,</li><li>calculate the number of tokens,</li><li>and visualize them with some nice pastel colors.</li></ul>The available PaLM-based models are:<ul><li>textembedding-gecko</li><li>textembedding-gecko-multilingual</li><li>text-bison</li><li>text-unicorn</li><li>chat-bison</li><li>code-gecko</li><li>code-bison</li><li>codechat-bison</li></ul>You can <a href="https://tokens-lpj6s2duga-ew.a.run.app/">try the application</a> online.And also have a look at the <a href="https://github.com/glaforge/llm-text-tokenization">source code</a> on Github. It’s a <a href="https://micronaut.io/">Micronaut</a> application. I serve the static assets as explained in my recent <a href="https://glaforge.dev/posts/2024/01/21/serving-static-assets-with-micronaut/">article</a>. I deployed the application on <a href="https://cloud.run/">Google Cloud Run</a>, the easiest way to deploy a container, and let it auto-scale for you. I did a source based deployment, as explained at the bottom <a href="https://glaforge.dev/posts/2022/10/24/build-deploy-java-17-apps-on-cloud-run-with-cloud-native-buildpacks-on-temurin/">here</a>.And voilà I can visualize my LLM tokens!Originally published at <a href="https://glaforge.dev/posts/2024/02/05/visualize-palm-based-llm-tokens/">https://glaforge.dev</a> on February 5, 2024.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8760b3122c0f" width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/visualize-palm-based-llm-tokens-8760b3122c0f">Visualize PaLM-based LLM tokens</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author

Link

Published date

Image url

Feed url

Guid

Hidden blurb

--- !ruby/object:Feedjira::Parser::RSSEntry
title: Visualize PaLM-based LLM tokens
url: https://medium.com/google-cloud/visualize-palm-based-llm-tokens-8760b3122c0f?source=rss-431147437aeb------2
author: Guillaume Laforge
categories:
- llm
- google-cloud-platform
- gcp-app-dev
- vertex-ai
- generative-ai-tools
published: 2024-02-05 00:00:45.000000000 Z
entry_id: !ruby/object:Feedjira::Parser::GloballyUniqueIdentifier
 is_perma_link: 'false'
 guid: https://medium.com/p/8760b3122c0f
carlessian_info:
 news_filer_version: 2
 newspaper: Guillaume Laforge - Medium
 macro_region: Blogs
rss_fields:
- title
- url
- author
- categories
- published
- entry_id
- content
content: <figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iUeKnOWdjYkVi3BY.jpg"
 /></figure>As I was working on tweaking the Vertex AI text embedding model in
 <a href="https://github.com/langchain4j">LangChain4j</a>, I wanted to better understand
 how the textembedding-gecko<a href="https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text-embeddings">model</a>
 tokenizes the text, in particular when we implement the <a href="https://arxiv.org/abs/2005.11401">Retrieval
 Augmented Generation</a> approach.The various PaLM-based models offer a computeTokens
 endpoint, which returns a list of tokens (encoded in Base 64) and their respective
 IDs.<blockquote>Note: At the time of this writing,
 there’s no equivalent endpoint for Gemini models.</blockquote>So I decided
 to create a <a href="https://tokens-lpj6s2duga-ew.a.run.app/">small application</a>
 that lets users:<ul><li>input some text,</li><li>select a model,</li><li>calculate
 the number of tokens,</li><li>and visualize them with some nice pastel colors.</li></ul>The
 available PaLM-based models are:<ul><li>textembedding-gecko</li><li>textembedding-gecko-multilingual</li><li>text-bison</li><li>text-unicorn</li><li>chat-bison</li><li>code-gecko</li><li>code-bison</li><li>codechat-bison</li></ul>You
 can <a href="https://tokens-lpj6s2duga-ew.a.run.app/">try the application</a> online.And
 also have a look at the <a href="https://github.com/glaforge/llm-text-tokenization">source
 code</a> on Github. It’s a <a href="https://micronaut.io/">Micronaut</a> application.
 I serve the static assets as explained in my recent <a href="https://glaforge.dev/posts/2024/01/21/serving-static-assets-with-micronaut/">article</a>.
 I deployed the application on <a href="https://cloud.run/">Google Cloud Run</a>,
 the easiest way to deploy a container, and let it auto-scale for you. I did a source
 based deployment, as explained at the bottom <a href="https://glaforge.dev/posts/2022/10/24/build-deploy-java-17-apps-on-cloud-run-with-cloud-native-buildpacks-on-temurin/">here</a>.And
 voilà I can visualize my LLM tokens!Originally published at
 <a href="https://glaforge.dev/posts/2024/02/05/visualize-palm-based-llm-tokens/">https://glaforge.dev</a>
 on February 5, 2024.<img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8760b3122c0f"
 width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/visualize-palm-based-llm-tokens-8760b3122c0f">Visualize
 PaLM-based LLM tokens</a> was originally published in <a href="https://medium.com/google-cloud">Google
 Cloud - Community</a> on Medium, where people are continuing the conversation by
 highlighting and responding to this story.

Language

Active

Ricc internal notes

Ricc source

Show this article Back to articles