Integrating Conversational Query Interface on Your Company’s Data or Your Customer’s Transaction Data Using Gemini Function Calling.
Integrating a Conversational Query Interface (CQI) opens up exciting possibilities for developers. Gemini Function Calling can be a game-changer in this context, enabling more intuitive and natural language interactions with your company’s data or customer transaction data, whether stored on-premises or in the cloud. It can make querying and analyzing data much more accessible and user-friendly, leading to the development of sophisticated systems that enhance decision-making and user experience.
If you’re looking to explore this further, grab a cup of coffee and ride on here with me
Understanding function calling can be beneficial for interacting with Gemini in a more powerful way. Here’s why:
- Structured information: Function calls provide a way to structure information you provide to Gemini. This can be especially helpful when requesting data from external systems. By defining functions that map to specific actions, you can guide Gemini towards the information you need.
- Efficient interactions: Function calls can streamline your interactions with Gemini. Instead of needing to craft complex prompts, to nudge Gemini in the right direction, you can provide clear instructions through defined functions. This can save time and effort, especially for repetitive tasks.
- Access to real-time data: One of the limitations of large language models like Gemini is that their knowledge has a cutoff point. Function calling allows Gemini to connect with external systems and APIs that provide up-to-date information. This unlocks the ability to integrate real-time data into Gemini’s responses.
To follow through with the demo here, the following requirements are necessary:
- Python installed on your machine.
- A project on Google Cloud with billing set up.
- Visual Studio Code installed.
- Python running on your system.
- Running Mac OS.
Our demo will showcase how we can use Gemini Function Calling to answer questions relating to pharmacy product detail
I shall have more demo update to continue the with Gemini Function Calling so we can consolidate our understanding towards applying it in various domain.
Demo - Answering Questions Related to Pharmacy Product Details
Setting up our Service Account on Google Cloud for Authentication
- Assuming you already have a new project on Google Cloud and it has billing enabled. Open Google Cloud Console: Go to the Google Cloud Console at https://console.cloud.google.com/ and log in with your Google account.
- Navigate to IAM & Admin: In the Cloud Console, navigate to the “IAM & Admin” section. You can find it in the left-hand navigation menu.
- Select Service Accounts: In the IAM & Admin section, select “Service Accounts” from the sidebar menu.
- Create a New Service Account: Click on the “Create Service Account” button to create a new service account.
- Fill in Service Account Details: Enter a name for your service account and optionally provide a description. You can choose the role(s) you want to assign to this service account; I will recommend Vertex AI Administrator, Owner and Editor.
6. Create Key: After creating the service account, click on the three dots next to the service account you just created and select “Create Key” from the dropdown menu.
7. Choose Key Type and Download JSON: Choose the key type (JSON is recommended) and click on the “Create” button. This action will generate a JSON file containing the credentials for your service account.
8. Download JSON File: Once the key is created, a JSON file containing the credentials will be downloaded to your computer. Keep this file secure, as we will use it later in the demo project to access our Google Cloud resources.
Enabling a Google Cloud Resource for our Project
The Google Cloud resource we are going to be enabling is the Vertex AI through which we will access Gemini.
- Open Google Cloud Console: Go to the Google Cloud Console at https://console.cloud.google.com/ and log in with your Google account.
- Navigate to the Vertex AI Dashboard: In the Cloud Console, navigate to the Vertex AI dashboard. You can find it in the left-hand navigation menu under “AI & Machine Learning” > “Vertex AI.”
- Enable Vertex AI API: If you haven’t enabled the Vertex AI API yet, Click on the “Enable” or similar button to enable the API.
- Wait for Activation: It may take a few moments for the services to be activated and ready for use. Once activated, you can start using Vertex AI to build and deploy machine learning models, manage datasets, and perform other AI-related tasks.
Getting Gemini API Key
Steps
- Visit ai.gemini.dev
- Click on the Get API Key in Google AI Studio
- Click on Get API Key once again to generate an API Key
- Copy and save this Key as we will use it later on
Coding our app
- Create a new folder on your system, preferably on the desktop, and give it any name you desire, such as ‘med-ask’
- open the folder in Visual Studio Code and create a new file called medical-ask.py
- use the command shift+cmd+p and then select Python:create environment
- Select Venv to create a virtual environment in your current workspace
- Select the recent Python Interpreter you will be using for the project- I recommend Python 3.12.2.
- You are now set up to begin.
When you open your terminal, you should see that you will be entering commands from: ((.venv)macaddr@macid med-ask%
Introducing our JSON service account file from earlier and installing libraries
Copy the json file you downloaded when setting up service account on your google cloud project and paste on your med-ask folder.
Your visual studio should now be like so:
Saving our Gemini Api Key
In other to save our API Key we would be using a library called python-dotenv so that we dont use our key directly on our .py file but instead we retrieve from systems environment
- Enter the command pip install python-dotenv on your terminal
- All things being equal, after install successfully,
- create a file .env on the med-ask folder and save your Gemini API Key like so:
GEMINI_API_KEY = “your gemini_api_key here”
Retrieving our GEMINI_API_KEY on our .py file
Steps
open your medical-ask.py and then enter the following code
import os
from dotenv import load_dotenv
load_dotenv()
GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")Installing Google Generative AI Python library and other libraries needed for this demo
for more context visit — https://ai.google.dev/tutorials/python_quickstart
enter the command in your terminal
pip install -q -U google-generativeai
pip install requests #since we will be calling an endpoint
pip install google-auth #to be able to set up our service account via Credentials
pip install vertexaiThis should install google-generativeai library functions and the others onto your python environment and we should proceed now to the main business.
Let’s Begin- 👇
The Context of what we are trying to achieve — so say we are running a pharm company and we have an open portal where patients can ask questions relating to pharmaceutical products- for example; they want to know if a particular drug is in store and possibly get more info or description about the med and maybe price as well.
the query can be like-
What is the price of paracetamol you have in your store
It’s important to emphasize the need for insightful interactions and avoiding information overload, especially in sensitive areas like pharmaceuticals where clarity and ease of access to relevant information are crucial for patients.
After installing the python libraries, your medical-ask.py should now be like so:
we initialize our med-ask-service.json which we downloaded when we created our service account from Google Cloud, set up the credentials and initialize our vertex ai with our credentials
import vertexai
med_service = "med-ask-service.json"
credentials = Credentials.from_service_account_file(med_service,scopes=['https://www.googleapis.com/auth/cloud-platform'])
if(credentials.expired):
credentials.refresh(Request())
vertexai.init(
project="your_project_id_on_google_cloud",
location="us-central1",credentials=credentials
)Let’s import the necessary function we will be using from vertexai generative models and then initialize our model and start the chat
from vertexai.generative_models import(FunctionDeclaration,GenerativeModel,Part,Tool,)
model = GenerativeModel("gemini-1.0-pro",generation_config = {"temperature":0},)
chat = model.start_chat()I would like us to revisit the flow we have earlier;
User Enters Prompt In a Natural Language: so lets say a user who opens your pharm web portal decides to ask in a natural way some question about some medicine
Prompt is sent to Gemini AI; Wrapped with Tool(s) for resolving the prompt:
let’s follow some steps to achieve this:
- so we are going to first set up our tool; we can call it a function that should usually fetch details about medicines in your data store
def pharmacy_product_detail(parameters):
# lets use a fictional api; although you can use your companies api here
url = f"https://yourpharmapi.app/pharmacy_product_list"
response = requests.get(url,params=parameters)
if response.status_code == 200:
data = response.json()
pharm_products = data.get("pharmacy_products",[])
if pharm_products:
return pharm_products[0] #am returning first object in the array for test purpose
else:
return 'no pharm products available'
else:
return {"error": f"API error: {response.status_code}"}
"""
if your api requires authorization, you could add it on the get method for requests like so:
response = requests.get(url,params=parameters,headers={"enter_key_here":AUTH_KEY_VALUE})
make sure to test your endpoint in postman to know the format of the response
if your pharmacy_products is returning in this json format
{
pharmacy_products:[
{
_id:"product_id_1",
name:"paracetamol"
price:20,
description:"the description of paracetamol"
...
},
{
_id:"product_id_2",
name:"paracetamol2"
price:50,
description:"the description of paracetamol2"
...
}
]
then you can adjust your return as:
pharm_products = data.get("pharmacy_products",[])
if pharm_products:
return pharm_products
else:
return 'no pharm products available'
"""
2. Let’s use the Tool library which we imported from VertexAI Generative Model to wrap our function tool
# you can have your function tool schema defined using FunctionDeclaration;
# you can have more than one function tool
''' tool = Tool(function_declarations=[
FunctionDeclaration(), FunctionDeclaration(), ....
] '''
tool = Tool(function_declarations=[
FunctionDeclaration(
name="pharmacy_product_detail",
description= "return the detail of a pharmacy product based on the search word or return only first page list if no search word found",
parameters={
"type": "object",
"properties": {
"search_word": {. #here i am specifying the parameter the api actually requires, you can specify more based on your api parameter within the properties
"type": "string",
"description": "The search word for returning pharmacy product detail"
}
},
}
)
]
)
# here is now our function_handler
function_handler = {
"pharmacy_product_detail": pharmacy_product_detail
}3. Let’s reassign our model and start_chat() passing in our Tool
#re-assign our model with our tools setup
model = GenerativeModel("gemini-1.0-pro",generation_config={"temperature":0},tools=[tool])
chat = model.start_chat()4. Now we have our function tool, we can send the user’s prompt to Gemini
prompt = "what is the price of paracetamol"
response = chat.send_message(prompt)5. What happens next on our flow-
Gemini tries to understand our prompt and checks to see if there are relevant tool to process the prompt.
It searches the function_declarations and perharps thinks- “oh this user wants to get the price of paracetamol if its available on store. Hmm, I think i found a tool that perharps should help - the pharmacy_product_detail tool which can get pharmaceutical product information about a medicine. Hmm, I know paracetamol is a medicine yeah, that means i can treat it as a parameter for the tool”.
6. When the send message executes and the above is done by Gemini, we can get the actual function which will then be called together with the args passed to it- as understood by Gemini
function_call = response.candidates[0].content.parts[0].function_call
# you can decide to print the function_call to see what response came back from Gemini
print(function_call)Try to run this section of code for now entering python medical-ask.py on your terminal. The result should be something like this depending on the search word picked by Gemini
7. We are going to check if the function_call.name is part of our function_handler we defined earlier; if so we extract the argument; in this case the search_word and then recall Gemini to go ahead to call the function with the argument and return response in natural language
From our flow; if you remember, if tool is found we :
Gemini calls the function with relevant argument retrieved from the prompt and return result
if function_call.name in function_handlers:
function_name = function_call.name
#get the arg and pass it unto the tool_name
args = {key: value for key,value in function_call.args.items()}
function_response = function_handlers[function_name](args)
#use the tool and its argument to answer the question
response = chat.send_message(
Part.from_function_response(name=function_name,
response={
"content": function_response
})
)
chat_response = response.candidates[0].content.parts[0].text
print(chat_response)
else:
#if it didnt find any tool for the prompt, we return Gemini default
#response
print(response.text)The complete code should be like this- please adjust where necessary and good luck
import os
from dotenv import load_dotenv
load_dotenv()
API_KEY = os.getenv("GEMINI_API_KEY")
AUTH_KEY = os.getenv("API_AUTH_KEY")
import requests
import vertexai
import urllib
from google.auth.transport.requests import Request
from google.oauth2.service_account import Credentials
med_service = "med-ask-service.json"
credentials = Credentials.from_service_account_file(med_service,scopes=['https://www.googleapis.com/auth/cloud-platform'])
if(credentials.expired):
credentials.refresh(Request())
vertexai.init(
project="serverless-rest-4b558",
location="us-central1",credentials=credentials
)
from vertexai.generative_models import(FunctionDeclaration,GenerativeModel,Part,Tool,)
#1. model initialization
model = GenerativeModel("gemini-1.0-pro",generation_config={"temperature":0,})
chat = model.start_chat(response_validation=False)
#2. define the function for getting pharmacy product detail
def pharmacy_product_detail(parameters):
# lets use a fictional api; although you can use your companies api here
url = f"yourendpoint.app/pharmacy_product/page=1&minimum_price=10&maximum_price=200"
response = requests.get(url,params=parameters,headers={'x-api-key':API_AUTH_KEY})
if response.status_code == 200:
data = response.json()
pharmacy_products = data.get("pharmacy_products",[])
if pharmacy_products: # Check if the array is not empty
return pharmacy_products[0] # Return the first object in the array
else:
return {"error": "No pharmacy products found"}
else:
return {"error": f"API error: {response.status_code}"}
#3. define the function tool schema
tool = Tool(function_declarations=[
FunctionDeclaration(
name="pharmacy_product_detail",
description= "return the detail of a pharmacy product based on the search word or return only first page list if no search word found",
parameters={
"type": "object",
"properties": {
"search_word": {
"type": "string",
"description": "The search word for returning pharmacy product detail"
}
},
}
)
]
)
# here is now our function_handler
function_handler = {
"pharmacy_product_detail": pharmacy_product_detail
}
#re-assign our model with our tools setup and start the chat
model = GenerativeModel("gemini-1.0-pro",generation_config={"temperature":0,},tools=[tool])
chat = model.start_chat(response_validation=False)
prompt = "what is the price of new product 9"
response = chat.send_message(prompt)
function_call = response.candidates[0].content.parts[0].function_call
# you can decide to print the function_call to see what response came back from Gemini
print(function_call)
if function_call.name in function_handler:
function_name = function_call.name
#get the arg and pass it unto the function
args = {key: value for key,value in function_call.args.items()}
function_response = function_handler[function_name](args)
#use the tool and its argument to answer the question via Gemini
response = chat.send_message(
Part.from_function_response(name=function_name,
response={
"content": function_response
})
)
chat_response = response.candidates[0].content.parts[0].text
print(chat_response)
else:
#if it didnt find any tool for the prompt, we return Gemini default
#response
print(response.text)experiment with your company endpoint, run the program and lets know what you got back.
Challenge
Even though I can choose to ask natural language questions about pharmacy products, I still need to specify correctly which field from the API response I am requesting. For example, if your API response includes the ‘quantity_available’ field in a pharmacy product object, asking in this order: ‘What is the available quantity of paracetamol?’ does not yield a response; instead, it informs you that the request cannot be fulfilled. The available tools lack the desired functionality.
However, if you phrase the question as follows: ‘What is the quantity_available for paracetamol?’ then it provides an adequate response.
I believe there is still more work to be done in this area.”
