Kong has launched the most recent replace to Kong AI Gateway, an answer for securing, governing, and controlling LLM consumption from common third-party suppliers.
Kong AI Gateway 3.11 introduces a brand new plugin that reduces token prices, a number of new generative AI capabilities, and help for AWS Bedrock Guardrails.
The brand new immediate compression plugin that removes padding and redundant phrases or phrases. This method preserves 80% of the meant semantic which means of the immediate, however the removing of pointless phrases can result in as much as a 5x discount in price.
In line with Kong, the immediate compression plugin enhances different cost-saving measures, comparable to Semantic Caching to stop redundant LLM calls and AI Fee Limiting to handle utilization limits by software or group.
This replace additionally provides over 10 new generative AI capabilities, together with batch execution of a number of LLM calls, audio transcription and translation, picture era, stateful assistants, and enhanced response introspection.
Lastly, Kong AI Gateway 3.11 provides help for AWS Bedrock Guardrails, which might help defend AI purposes from malicious and unintended penalties, like hallucinations or inappropriate content material. Builders can monitor purposes and alter insurance policies in actual time while not having to vary code.
“We’re excited to introduce certainly one of our most important Kong AI Gateway releases to this point. With options like immediate compression, multimodal help and guardrails, model 3.11 provides groups the instruments they should construct extra succesful AI programs—quicker and with far much less operational overhead. It’s a serious step ahead for any group trying to scale AI reliably whereas conserving infrastructure prices beneath management,” mentioned Marco Palladino, CTO and co-founder of Kong.
