Is AI Scraping Your PDFs? How to Protect Your Documents with TDMRep
AI systems can read, search, and reuse the content of your PDFs to train generative models. This practice is part of a broader activity known as Text and Data Mining (TDM). The good news: you have options for allowing, limiting, or preventing it.
The TDM Reservation Protocol (TDMRep) lets you embed machine-readable rights information directly in a PDF. This article explains how to use it to:
- Reserve your rights: make it clear to TDM users that rights to the PDF's content are reserved.
- Point to a licensing path: tell TDM users how to legally access and use the content, including rights clearance and compensation where required.
- Opt out entirely: signal that machines should not mine the PDF's content at all.
Watch a Q&A with Support Engineer Alec Goldstein, Is AI Consuming and Scraping Your PDFs?
What Is the TDM Reservation Protocol?
TDMRep defines an "opt-out" mechanism for text and data mining. It answers two questions:
- Is this content permitted to be searched or mined for TDM purposes?
- How can the rightsholder be contacted to obtain a license for that use, if one is required?
You can read the latest version of the TDMRep specification on the W3C site.
The Two Properties
TDMRep's rights reservation model uses two properties:
| Property | Type | Purpose |
|---|---|---|
tdm-reservation | Boolean | Indicates whether mining rights are reserved. |
tdm-policy | URL | Links to the publisher's contact information and the conditions for obtaining authorization to mine the content. |
Note: Technically,
tdm-policyis optional. However, leaving it out may render the effort ineffective, because TDM users won't know how to request permission.
An author or publisher can add these properties to a PDF's XMP metadata, either proactively when the document is created or retroactively after it has been published. Together, they signal a reservation of rights and give TDM users a pathway to obtain a license.
Image: W3C example of TDMRep properties in PDF XMP metadata. Source: W3C TDMRep Example 8
What Does a TDM Policy Look Like?
A TDM policy is written in JSON-LD (JSON for Linked Data) and uses standard terminology from ODRL (Open Digital Rights Language), a proposed language for standardizing digital rights management.
Required Properties
| Property | Description |
|---|---|
@context | An array of two values: http://www.w3.org/ns/odrl.jsonld and http://www.w3.org/ns/tdmrep.jsonld. |
uid | An identifier for the policy, expressed as a URI. It doesn't have to point to a publicly accessible resource, but that is encouraged. W3C recommends linking to your website's human-readable Terms of Use. |
@type | Must be "Offer". Per the ODRL model, Offers are "proposals from Rights Holders for specific Rights over their Assets." |
profile | Must be http://www.w3.org/ns/tdmrep. |
assigner | A set of child vCard properties for contacting the rightsholder. By our best interpretation of the spec, you don't need to use every vCard property, but any you do use must come from the list in section 7.1.4 of the TDMRep spec. |
permission | An array of permissions. By our best interpretation of the spec, only one element is required: "action": "tdm:mine". This expresses permission to "analyse, via automated analytical technique, text and data in digital form in order to generate information which includes but is not limited to patterns, trends and correlations." |
A Recommended Addition: duty
The permission array can also include a duty element, which expresses the TDM actor's duty to obtain verifiable consent. The spec doesn't explicitly say this element is required, but it's important enough that we strongly recommend including it.
The spec also lists other JSON properties as "recommended" or "optional." The list above covers strictly required properties only.
Example Policies
- Simple example: Insert Datalogics example policy here
- W3C example: Example 19 shows a structure for defining access for scientific research, which can vary greatly by locale.
Using Adobe PDF Library to Express TDMRep in a PDF
For this first example, we'll use the .NET interface of Adobe PDF Library (APDFL).
To express TDMRep, we need to set two XMP metadata properties, reservation and policy, under the namespace http://www.w3.org/ns/tdmrep/. APDFL's Document class provides a method for exactly this: SetXMPMetadataProperty().
doc.SetXMPMetadataProperty(tdmNS, "tdm", "reservation", "1");
doc.SetXMPMetadataProperty(tdmNS, "tdm", "policy", "http://www.test.com/some_tdm_policy");
In this code:
docis aDocumentobject.tdmNSis shorthand for the W3C namespace above."tdm"is the property prefix in XML.
The result is a PDF with XMP metadata properties that look like the W3C example shown earlier in this article.
Download the Datalogics .NET sample application: Insert link
Java and C++ Equivalents
APDFL's other interfaces offer equivalent functions:
| Interface | Function |
|---|---|
| Java | Document.setXMPMetadataProperty() |
| C++ | PDDocSetXAPMetadataProperty() |
Reading TDMRep Metadata
Datalogics also provides a sample application that extracts text from a PDF and prints its TDMRep metadata to the console. It checks the PDF's XMP metadata for the relevant properties, then uses a JSON parser to return the rightsholder's contact information. View the TextExtractTDMAware sample on GitHub.
Rights metadata is one layer of protection; where your PDFs are processed is another. Read our guide, On-Premises vs. Cloud PDF Processing: What Enterprise Teams Need to Know.
Protect Your PDFs from AI Scraping
Ready to address AI scraping challenges with your PDFs? Download a free trial of Adobe PDF Library and get started quickly. Prefer to discuss your project first? Contact Us. For for more technical questions, schedule a meeting with a software engineer.