As part of Clarify, AWS offers enterprises a feature, dubbed FMEval, which is an open-source LLM evaluation library to help data scientists and ML engineers evaluate LLMs before deciding to use it for a specific use case.
“FMEval provides the ability to perform evaluations for both LLM model endpoints or the endpoint for a generative AI service as a whole. FMEval helps in measuring evaluation dimensions such as accuracy, robustness, bias, toxicity, and factual knowledge for any LLM,” the cloud service provider wrote in a blog post.
Enterprises can use EMEval to evaluate LLMs hosted on either AWS or third-party platforms, such as ChatGPT, HuggingFace, and LangChain, it added.