Set Up AI for Playbooks and Workflows

This article will walk you through how to set up AI for AI Playbooks and workflows. This includes how to validate AI Clauses and use AI during content reviews.

Part 1: Initial Setup of AI Clauses for AI Playbooks

The first step in setting up AI for your playbooks is to configure AI Clauses for each play. Follow these steps for each play you want to set up:

1. Add a description and define positions (preferred, fallback, non-standard language) for each playbook clause.
2. Associate an AI Clause:
   • If an existing AI Clause fits, add it to the play.
   • If not, create a new Custom Clause for that play in Company Settings > AI Clauses > “Create an AI Clause”.

For each AI Clause, you’ll need to add representative examples. We’ve found 5 to 10 examples are generally enough to handle most scenarios but the goal is not quantity, rather diversity. Include all versions of the clause you expect to encounter, as this helps prevent non-detection issues. We also recommend setting the Detection Settings to “Highly Similar”.

Specifically, add:

• Your current clause positions.
• Redlined versions of these positions that you've seen in past negotiations.
• Versions of the clause you've accepted from third-party contracts (if applicable).

Then, add a Presence Flag to the playbook clause ("Clause is required in documents and will need approval to be excluded") to ensure the clause is detected even if the AI misses it.

To reduce potential misclassifications in future contracts, we recommend the following to train the AI on what not to detect, reducing errors during contract reviews.

1. After setting up the initial AI workflow, create a copy of the contract associated with the workflow.
2. Remove AI Clauses: In the copied document, remove any sections related to AI Clauses/Plays. If you’re working with multiple documents, repeat this process for each and combine them into a single document titled "No Classify."
3. Set Up a Test Workflow: Create a duplicate of the original workflow and name it “Test.” Upload the “No Classify” document into the "Test" workflow.
4. Launch the “Test” workflow, then open the AI Playbooks.
5. When a misclassification occurs (AI incorrectly tags text as the wrong clause):
   • Add the misclassified text to a new “No-Classify” Custom Clause. You can add up to 20 examples per custom clause, and create additional "No-Classify" clauses if necessary.
6. After completing the above steps, re-upload the same contract to the “Test” workflow to trigger the AI again and catch any remaining misclassifications. Keep doing this until you no longer have any language as being flagged as non-standard.

We recommend this option since it reduces the number of overall custom AI clauses to manage and maintain.

Part 2: Validating AI Clauses for AI Playbooks

Once the initial setup is done, validate how well the AI performs by testing it within your workflows.

1. Test 5–10 documents (mix of first-party and third-party papers if applicable) to replicate your external contracting process (Start workflow, exporting document, apply redlines, upload new document & repeat)
2. When a misclassification occurs (AI incorrectly tags text as the wrong clause):
   1. Add the misclassified text to the “No-Classify” Custom Clause. If on Option 2, add the text to the correct AI Clause or Custom Clause. 
3. When non-detection occurs after redlines (AI no longer tags text as the clause):
   1. Add the text that should have been flagged as an example to the corresponding AI Clause for that play.

Track how often non-detections occur for each AI Clause:

• If they are frequent, try lowering the detection threshold to “Somewhat Similar.”
• Review past contracts and add more clause variations as examples to improve accuracy.

Part 3: Process during Contract Review 

AI is an evolving tool, and it’s crucial to provide feedback as new contracts come in to improve accuracy over time.

This step should be completed by an admin in collaboration with legal reviews. If you have more than 10 reviewers, we recommend limiting participation to 5-10 key reviewers.

1. Set up a tracking system (e.g., an Excel spreadsheet) for your entire legal team or select members, with the following columns:
   1. Clause Text
   2. Clause Type
   3. Classification Issue (Misclassification or Nondetection)
2. During contract reviews, if a clause is:
   1. Misclassified, log the example under “Misclassification.”
   2. Not detected, log the example with its correct clause type under “Nondetection.”
3. Monthly review: Once a month, review the spreadsheet and add the logged examples to the AI model to continually refine its performance.
   1. Add the misclassified text to the “No-Classify” Custom Clause. If on Option 2, add the text to the correct AI Clause or Custom Clause. 
   2. Add the nondetection text as an example to the corresponding AI Clause.

We recommend decoupling the process of finding misclassifications and non-detections from the monthly review to ensure you can verify the logged examples are correct and that you aren’t training AI with any erroneous mistakes.