Can any gpt model sumarise the text and images contained in a pdf ?
#read images in pdf
vernal loom
drifting grailBOT
Hi @vernal loom. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Yes, it is possible to use OpenAI's GPT models to summarize the text and images contained in a PDF. However, this process involves several steps and the use of different models for different tasks.
Firstly, you would need to extract the text and images from the PDF. For text extraction, you can use any standard PDF parsing library. For images, you might need to convert the PDF pages into images. An example of how to do this can be found in the OpenAI Cookbook's guide on how to parse PDF docs for RAG (https://cookbook.openai.com/examples/parse_pdf_docs_for_rag#processing-all-documents).
Once you have the text and images, you can use the GPT-4 model to summarize the text. However, as per a forum post (https://community.openai.com/t/672380), each API call is treated separately and there is no "memory function". Therefore, you might need to summarize each document individually and then create an aggregate summary.
For images, you can use the GPT-4V model to analyze the content based on the images. You would need to convert the images to base64 encoded images in a data URI format to use with the ChatCompletions API. An example of how to do this can be found in the OpenAI Cookbook's guide on how to parse PDF docs for RAG (https://cookbook.openai.com/examples/parse_pdf_docs_for_rag#image-analysis-with-gpt-4v).
Here is an example of how to analyze an image using GPT-4V:
def analyze_image(img_url):
response = client.chat.completions.create(
model="gpt-4-vision-preview",
temperature=0,
messages=[
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": img_url,
},
],
}
],
max_tokens=300,
top_p=0.1
)
return response.choices[0].message.content
Please note that GPT-4V has some limitations, such as not being suitable for interpreting specialized medical images, struggling with tasks requiring precise spatial localization, and may generate incorrect descriptions or captions in certain scenarios. More details can be found in the OpenAI's guide on Vision (https://platform.openai.com/docs/guides/vision).