The site has previously ranked models from OpenAI, Meta, Mistral, Anthropic, Google, and others based on more than 20 technical specifications. Other model manufacturers are also being asked to request conformity assessments for their models.
Researchers from LatticeFlow, INSAIT, and ETH Zurich wrote in a technical paper that “shortcomings of existing models and benchmarks have been exposed, particularly in areas such as robustness, security, diversity, and fairness.” states. “Compl-AI shows for the first time the possibilities and difficulties of taking this legal obligation to a more concrete technical level.”
Most models struggle with diversity and non-discrimination
Under EU AI legislation, models and systems are labeled as having unacceptable, high, limited, or minimal risk. In particular, unacceptable labels prohibit model development and deployment. If model manufacturers are found to be non-compliant, they could be subject to hefty fines.
