Anthropic PBC, the creator of the generative AI chatbot Claude, has announced the release of prompt caching. This new feature improves the response times of AI’s large language models by allowing developers to pass them longer, more detailed prompts.
A public beta of the company’s “Prompt Caching” feature for its Claude 3.5 Sonnet and Claude 3 Haiku models will be accessible on the Anthropic API starting on August 14.
According to the description, this function enables users to effectively retain and reuse particular contextual information, such as more detailed instructions and data, within prompts without incurring additional fees or experiencing delays.
For example, if a user requests that an LLM respond to enquiries regarding a lengthy document, that document must be included in the prompt for each ensuing discussion.
This means that everything must be completely reloaded into the AI between interactions, which might use many resources while everything is gradually absorbed again.
Developers can also save comprehensive instructions, sample answers, and pertinent data in cached prompts.
This enables them to quickly put up a method to generate a consistent response across various chatbot instances without constantly injecting them on top of the user prompt.
“As Claude continues to advance, prompt caching is one of the cutting-edge features we’re developing to enhance its capabilities,” an Anthropic spokesperson told PYMNTS.
“We’re exploring ways to optimize context retention for unique use cases, which aligns with our commitment to making Claude not just powerful but intuitive and indispensable for users across all technical levels.”
According to Anthropic, the new function can sometimes improve response times by up to two times and save costs by up to 90%.
Even while these numbers are significant, it’s crucial to remember that actual performance can change based on particular use cases and implementations.
We earlier reported Mike Krieger, the co-founder of Instagram and a former chief technology officer, has been announced by Anthropic as its first chief product officer.
