The Audit Log Your AI Vendor Should Be Giving You (and Probably Is Not)

The Audit Log Your AI Vendor Should Be Giving You (and Probably Is Not)

I was reviewing an AI vendor's SOC 2 report last month and noticed something that should bother more people than it does. The vendor logged that a query happened. They logged who made it. They logged a timestamp. And that was it. No record of what data was sent to the model, what the model returned, which version of the model was running, or whether any retrieval-augmented generation sources were consulted. If a regulator or opposing counsel asked us to reconstruct what the AI actually did with sensitive client data on a specific date, we would have had almost nothing.

This is the norm, not the exception. Most enterprise AI vendors treat audit logging as a checkbox item rather than a forensic capability. And for companies handling trade secrets, protected health information, or regulated financial data, the gap between what vendors log and what you actually need is a liability waiting to surface.

What "Defensible" Actually Means in an AI Audit Log

A defensible audit log is one you can hand to a regulator, a judge, or an internal investigation team and use to reconstruct exactly what happened during an AI interaction. The Defend Trade Secrets Act of 2016 (18 U.S.C. 1836) requires companies to demonstrate they took "reasonable measures" to protect trade secret information. If your AI system ingested proprietary formulas, client strategies, or source code, and you cannot show what happened to that data inside the system, your "reasonable measures" argument starts to collapse.

The FTC has been increasingly explicit about this. In its 2023 policy statement on AI and biometric information, the Commission emphasized that companies must maintain sufficient records to allow for meaningful oversight of automated systems. The EU AI Act, which entered into force in August 2024, goes further: Article 12 requires that high-risk AI systems be designed with logging capabilities that record events throughout the system's lifecycle, with enough granularity to enable post-market monitoring and incident investigation.

Even if your AI use case does not fall under "high-risk" classification, the regulatory direction is clear. Logging is becoming a baseline expectation, and the standard is moving well beyond "who clicked what."

The Seven Elements of a Complete AI Audit Log

Based on NIST AI RMF (AI 100-1), ISO/IEC 42001, and the practical requirements we see emerging from enforcement actions, a defensible AI audit log should capture at minimum:

• User identity and authorization level. Not just a username. The specific role, permission set, and authentication method used at the time of the interaction.
• Full input payload. The complete prompt or query, including any documents, data, or context uploaded or referenced. If your system uses RAG, the log should capture which documents were retrieved and fed to the model.
• Model version and configuration. Which model was running, what version, what temperature or parameter settings were active, and whether any fine-tuning or custom instructions were applied. Models change constantly. A response generated by GPT-4-0613 may differ meaningfully from GPT-4-turbo-2024-04-09, and you need to know which one produced a given output.
• Complete output. The full response the system generated, not a summary or truncation.
• Data classification tags. If the input contained data classified as confidential, PHI, PII, or trade secret material, the log should reflect that classification at the point of interaction.
• Guardrail and filter activity. Any content filtering, redaction, or policy enforcement that was triggered during the interaction, including cases where guardrails were evaluated but not triggered.
• Timestamp and session context. Precise timestamps (ideally UTC with millisecond precision), session identifiers, and any relevant workflow context linking the interaction to a broader business process.

If your vendor is only giving you items one and seven, you have a logging gap that matters.

The Trade Secret Problem Specifically

Trade secret litigation is where weak AI audit logs will cause the most pain in the near term. In Motorola Solutions v. Hytera Communications, which resulted in a $764.6 million judgment in 2020, the ability to trace exactly how proprietary information moved through systems was central to establishing misappropriation. Now add an AI layer to that chain of custody. If an employee feeds trade secret material into an AI tool, and the vendor cannot produce logs showing what happened to that data, you face two problems simultaneously: you may not be able to prove misappropriation occurred, and you may not be able to prove you took reasonable steps to prevent it.

The Uniform Trade Secrets Act, adopted in some form by 48 states, conditions protection on the owner having taken efforts "reasonable under the circumstances" to maintain secrecy. Courts have historically looked at access controls, NDAs, and physical security. They are now starting to look at digital data flows through AI systems. In Compulife Software Inc. v. Newman (11th Cir., 2020), the court examined in detail how data moved through automated systems to determine whether trade secret protections were maintained. The AI context makes this analysis more complex, not less.

If you are a general counsel thinking about this, the question is straightforward: could you reconstruct, six months from now, exactly what proprietary data entered your AI systems, what the system did with it, and where it went? If the answer is no, your trade secret protections have a gap that opposing counsel will eventually find.

Questions to Ask Your AI Vendor Today

These are concrete, and you should expect concrete answers:

• Do you log full input and output content for every interaction, or only metadata?
• What is your log retention period, and can it be configured to meet our regulatory requirements? (HIPAA requires six years. SEC Rule 17a-4 requires certain records for six years as well. Your vendor's default 30-day retention is probably insufficient.)
• Are logs stored in a tamper-evident format? Can you demonstrate chain of custody for log integrity?
• Can we export raw logs in a standard format (OCSF, CEF, or JSON) for ingestion into our own SIEM?
• When model versions change, is the version identifier captured in the log for each interaction?
• If our data is used in any form of model training or fine-tuning, is that event logged?
• Do your logs capture when content filters or guardrails activate, including near-misses?
• Where are logs stored, and are they subject to the same data residency and encryption requirements as our primary data?

A vendor that gets defensive or vague on these questions is telling you something important about their architecture.

Retention Is Its Own Problem

Even vendors that log comprehensively often retain logs for absurdly short periods. Thirty days is common. Some purge interaction content after 72 hours, keeping only metadata. For regulated industries, this is almost always insufficient. Litigation hold obligations under the Federal Rules of Civil Procedure can require preservation of electronically stored information for years. If your AI vendor has already purged the logs by the time you receive a discovery request, you have a spoliation problem on top of everything else.

Your vendor agreement should specify retention periods that match your longest applicable regulatory or litigation hold requirement, with the ability to extend retention on specific interactions or users when a hold is triggered.

How FirmAdapt Addresses This

FirmAdapt's architecture was designed with the assumption that every AI interaction may need to be reconstructed and defended later. The platform captures all seven log elements described above by default, including full input and output content, model version identifiers, RAG source attribution, guardrail activation events, and data classification tags applied at the point of interaction. Logs are stored in a tamper-evident, append-only format with configurable retention periods that can be set per regulatory framework or extended dynamically for litigation holds.

Logs are exportable in standard formats for SIEM integration and are subject to the same data residency, encryption, and access control policies as all other data within the platform. This is not an add-on module. It is how the system works at the infrastructure level, because building forensic-grade logging after the fact is significantly harder than building it in from the start.