New York State Department of Financial Services Adopts AI Guidance

On July 11, 2024, the New York State Department of Financial Services (“NYDFS”) issued a circular letter (the “AI Circular Letter”) addressing the use of external consumer data and information sources (“ECDIS”), and artificial intelligence systems (“AIS”) in insurance underwriting and pricing. The AI Circular Letter outlines the NYDFS’s expectations regarding the development and use of ECDIS, AIS and other predictive models by insurers, including:

• analysis of ECDIS and AIS for unfair and unlawful discrimination;
• demonstration of the actuarial validity of ECDIS and AIS;
• maintenance of a governance framework for oversight of the overall outcomes of the insurers’ use of ECDIS and AIS; and
• appropriate levels of transparency, risk management, and internal controls, including with respect to third-party vendors and consumer disclosures.

The NYDFS released a proposed draft of the AI Circular Letter (“Proposed Circular Letter”) on January 17, 2024, and requested comments from the public regarding the Proposed Circular Letter. The final AI Circular Letter reflects the NYDFS’s consideration of comments on the Proposed Circular Letter it received from insurers, trade associations, advisory firms, universities, and the broader public. 

Who’s Covered?

The AI Circular Letter applies to all insurers authorized to write insurance in New York, Article 43 Corporations, health maintenance organizations, licensed fraternal benefit societies, and the New York State Insurance Fund (collectively, “insurers”).

Changes from Proposed Circular Letter

The NYDFS adopted the AI Circular Letter with several changes from the Proposed Circular Letter, including:

• Proxy Discrimination: Rather than imposing an affirmative requirement that insurers demonstrate that ECDIS employed for underwriting and pricing do not serve as a proxy for unfair or unlawful discrimination against a protected class, the AI Circular Letter requires insurers to evaluate the extent to which ECDIS are correlated with status in any protected classes that may result in unfair or unlawful discrimination. If such correlation is identified (either using data available to the insurer or inferred from accepted statistical methodologies), insurers are asked to consider whether the use of such ECDIS is required by a legitimate business necessity.
• Data for Testing: The AI Circular Letter clarified that testing of whether the use of ECDIS or AIS produces disproportionate adverse effects in underwriting and/or pricing for similarly situated insureds or insureds of a protected class should be conducted using data that is available to the insurer or may be reasonably inferred using accepted statistical methodologies. The NYDFS has no expectations that insurers collect additional data from or about individuals for such testing.
• Ongoing Search for Less Discriminatory Alternatives: In the event that testing reveals a prima facie showing of a disproportionate adverse effect and a legitimate, lawful, and fair explanation or rationale can account for the differential effect, an insurer is required to conduct and appropriately document a search for and analysis of less discriminatory alternative variables or methodologies that would reasonably meet the insurer’s legitimate business needs. If no less discriminatory alternative exists at the outset, then the insurer is required to search for one at least annually.
• Annual Testing: The AI Circular Letter requires that insurers maintain comprehensive documentation for their use of all AIS, including all ECDIS relied upon for such AIS, which may include a description of testing conducted at least annually to assess the output of AIS models including drift that may result from the use of machine learning or other automated updates.
• Contractual Provisions for Third-Party Vendor Contracts: The AI Circular Letter includes expectations for terms that insurers should include in their contracts with third-party vendors as described in the section regarding Third-Party Vendors below.
• Clarification of Accelerated Underwriting Requirements: The AI Circular Letter expands the requirement to disclose threshold criteria for a process utilizing ECDIS and/or AIS for underwriting by requiring such disclosure in a clear and prominent manner in all relevant advertisements and marketing materials, and in disclosures provided to consumers during an application process. The AI Circular Letter also establishes a 15-day deadline for providing notice to an applicant describing the reason or reasons that the applicant cannot be underwritten for insurance using ECDIS and/or AIS.

The NYDFS declined to make changes to the requirements for board and senior management oversight, stating that the AI Circular Letter maintains the expectation that both senior management and the board have a responsibility for the overall outcome but not day-to-day implementation. The NYDFS described this expectation as consistent with its long-standing supervisory approach. The NYDFS also declined to make changes to the requirements for oversight of third-party vendors, again stating that the third-party oversight requirements in the AI Circular Letter were consistent with its long-standing supervisory approach.  

Fairness Principles

The AI Circular Letter sets out specific fairness principles that insurers must adhere to in the use of ECDIS or AIS for underwriting and/or pricing:

• The data source or model supporting the ECDIS or AIS must not use, and must not be based in any way on, any class protected pursuant to Article 26 of the New York Insurance Law;
• Such use must not result in or permit any unfair discrimination or otherwise violate the New York Insurance Law;
• Any ECDIS to be used must be supported by generally accepted actuarial standards of practice and based on actual or reasonably anticipated experience;
• Any ECDIS to be used must not be prohibited by the New York Insurance Law;
• Insurers must evaluate whether any ECDIS to be used serves as a proxy for any protected classes that may result in unfair or unlawful discrimination and, if such correlations are identified, must determine whether use of such ECDIS is required by a legitimate business necessity; and
• The ECDIS or AIS must not collect or use criteria that would constitute unfair or unlawful discrimination or an unfair trade practice.

Comprehensive Assessment for Discrimination

The AI Circular Letter requires an insurer using ECDIS or AIS in underwriting and/or pricing to conduct a comprehensive assessment to determine that such use would not be unfairly or unlawfully discriminatory in violation of the New York Insurance Law. The AI Circular Letter sets out a three-step process for such a comprehensive assessment:

• First, the insurer must assess whether the use of ECDIS or AIS would produce disproportionate adverse effects in underwriting and/or pricing on similarly situated insureds or insureds of a protected class.
• Second, if there is a prima facie showing of a disproportionate adverse effect, then a further assessment must be conducted to determine whether there is a legitimate, lawful, and fair explanation or rationale for such effect. If no such explanation or rationale can be determined, then the insurer is obligated to modify its planned use of ECDIS or AIS.
• Third, even if a legitimate, lawful, and fair explanation or rationale exists, the insurer must conduct and document a search for less discriminatory alternative variables or methodology that would still reasonably meet the insurer’s business needs and continue to conduct a search for less discriminatory alternatives at least annually.

To demonstrate compliance with the above requirements, the AI Circular Letter requires insurers to conduct this assessment before an AIS is launched and on a “regular cadence thereafter” and after material updates, although the requirement that insurers appropriately document the processes and reasoning behind their testing methodologies and analysis notes that such documentation may include a description of testing conducted at least annually to assess the output of AIS models, including drift that may result from the use of machine learning or other automated updates. The AI Circular Letter encourages insurers to use multiple statistical metrics in evaluating data and model outputs. These metrics may include: adverse impact ratio; denials odds ratios; marginal effects; standardized mean differences; Z-tests and T-tests; and drivers of disparity.

Governance and Risk Management

The AI Circular Letter notes that an existing NYDFS regulation (Insurance Regulation 215) requires an insurer to have a corporate governance framework that is appropriate for the nature, scale, and complexity of the insurer. It then goes on to describe key expectations for an insurer’s governance and risk management framework with respect to ECDIS and AIS:

• Such a framework should provide appropriate oversight of the insurer’s use of ECDIS and AIS, including at the board of directors and senior management levels.
• An insurer that uses ECDIS or AIS should have written policies and procedures that clearly define appropriate roles and responsibilities, outline monitoring and reporting requirements, provide for training of relevant personnel and set standards for the acquisition, use of, or reliance on ECDIS and AIS developed or deployed by third-party vendors.
• The insurer’s board of directors, or committees thereof, or senior management should review and approve such policies and procedures at least annually.
• The insurer’s internal audit function should be appropriately engaged with the insurer’s use of ECDIS and AIS, consistent with the financial, operational, and compliance risk.
• An insurer should maintain comprehensive documentation regarding its use of ECDIS or AIS.
• An insurer must establish a system for receiving and addressing consumer complaints and inquiries about the insurer’s use of ECDIS and/or AIS.

The AI Circular Letter provides that “[s]enior management is responsible for day-to-day implementation of the insurer’s development and management of ECDIS and AIS, consistent with the board’s or other governing body’s strategic vision and risk appetite.” Senior management can meet the foregoing requirement by:

• Establishing adequate policies and procedures, assigning competent staff, overseeing model risk management, ensuring effective challenge and independent risk assessment, reviewing internal audit findings, and taking prompt remedial action when necessary; and
• Ensuring that all relevant operational areas are appropriately engaged, such as through a cross-functional management committee with representatives from key function areas, including legal, compliance, risk management, product development, underwriting, actuarial, and data science, as appropriate.

Third-Party Vendors

The AI Circular Letter also provides that the ultimate responsibility for proper use of ECDIS or AIS rests with the insurers, even if such ECDIS or AIS was developed or deployed by third-party vendors. As noted above, insurers are required to maintain appropriate written policies and procedures regarding the use of third-party vendors in connection with the use of ECDIS and AIS developed or deployed by a third-party vendor. Insurers are also required to maintain procedures for reporting any incorrect information to third-party vendors for further investigation and update, as necessary, and to remediate and eliminate incorrect information from their AIS that the insurer has identified or has been reported to a third-party vendor. Finally, where appropriate and available, insurers should include terms in their contracts with third-party vendors that: (i) provide audit rights or entitle the insurer to receive audit reports by qualified auditing entities; and (ii) require the third-party vendor to cooperate with the insurer regarding regulatory inquiries and investigations related to the insurer’s use of the third-party vendor’s product or services.

Transparency

The AI Circular Letter also reminds insurers that the failure to adequately disclose to an insured or potential insured any specific reason or reasons for its refusal of coverage, limitation of coverage, or charging a different rate for coverage may be deemed an unfair trade practice. When an insurer is using ECDIS and/or AIS, the notice regarding an adverse underwriting or pricing decision is expected to include the following:

• The specific source of the information upon which the insurer based its decision;
• Whether the insurer uses AIS in its underwriting or pricing process;
• Whether the insurer uses ECDIS; and
• A description of the process for the insured or potential insured to request information about the specific data that resulted in the decision.

The AI Circular Letter warns that failure to disclose such information could constitute an unfair trade practice under Article 24 of the New York Insurance Law, and that an insurer may not rely on the proprietary nature of a third-party vendor’s algorithmic processes to justify the lack of specificity in such an adverse decision notice. Also, to the extent that an accelerated underwriting process is available only to certain persons, an insurer must disclose the objective criteria for using the accelerated process in writing in a clear and prominent manner in all relevant advertisements and marketing materials, and in disclosures provided to consumers during an application process. Further, if the accelerated process determines that an applicant will not be approved for insurance under the accelerated process, and can only obtain insurance by submitting to the traditional underwriting process, the reason for such a decision must be disclosed to the applicant within 15 days of such determination.

Conclusion

Adoption of the AI Circular Letter by the NYDFS is one of the most expansive efforts by a US insurance regulator to regulate the use of ECDIS and AIS by insurers in terms of the level of detailed requirements it imposes. Previously, on September 21, 2023, the Colorado Division of Insurance (“COI”) adopted a first-of-its-kind regulation in the US establishing governance and risk management requirements for life insurers that use ECDIS or algorithms or predictive models that use ECDIS, which regulation became effective on November 14, 2023;¹ the COI also released a draft proposed regulation on September 27, 2023, on Quantitative Testing of External Consumer Data and Information Sources, Algorithms, and Predictive Models Used for Life Insurance Underwriting for Unfairly Discriminatory Outcomes, but that draft remains subject to further consideration and has not been adopted yet.² Meanwhile, other states have proceeded with the adoption of the model bulletin regarding the “Use of Artificial Intelligence Systems in Insurance” adopted by the National Association of Insurance Commissioners in December 2023. As of June 30, 2024, 12 states and the District of Columbia have adopted the model bulletin. We will continue to monitor and report on states’ adoption of guidance regarding use of artificial intelligence in insurance.

¹ See Colorado Adopts Artificial Intelligence Regulation for Life Insurers | Insights | Mayer Brown. 

² See Colorado Releases Draft Regulation on AI Testing for Life Insurers | Insights | Mayer Brown