AI Decisions and Overrides

How AI Makes Screening Decisions

Scholara's AI uses large language models trained on millions of scientific articles to understand and evaluate studies against your protocol. The decision process is transparent and explainable.

The Decision-Making Process

  1. 1AI reads the abstract or full-text content
  2. 2Parses your protocol including PICO/PCC, inclusion/exclusion criteria, and study designs
  3. 3Extracts key information: population, intervention, outcomes, design, sample size
  4. 4Compares extracted information against each criterion
  5. 5Generates a decision (Include/Exclude/Uncertain)
  6. 6Assigns a confidence score (0-100%)
  7. 7Provides detailed reasoning explaining the decision

Understanding Confidence Scores

High Confidence (90-100%)

The AI is very certain about the decision. Common scenarios:

  • Clear match with all inclusion criteria
  • Explicit statement violating an exclusion criterion
  • Wrong study design explicitly stated
  • Population characteristics clearly described and matching/not matching
  • These decisions are usually correct but still benefit from spot-checking

Medium Confidence (70-89%)

The AI believes the decision is likely correct but has some uncertainty:

  • Most criteria are clearly met/not met, but one is ambiguous
  • Terminology is slightly different from protocol but likely equivalent
  • Some details are missing but available information suggests include/exclude
  • These warrant closer human review

Low Confidence (<70%)

Significant uncertainty in the decision:

  • Abstract lacks detail on critical criteria
  • Contradictory information in the text
  • Unclear whether intervention matches protocol
  • Borderline cases (e.g., age range partially overlaps)
  • Always requires human review

Uncertain (Flagged for Review)

AI explicitly marks for human decision when:

  • Insufficient information to make any determination
  • Complex study design requiring expert judgment
  • Edge case where criteria interpretation is needed
  • Conflicting indicators (some criteria met, others not clearly)
  • You must review and make a human decision on these

Reading AI Explanations

Each decision includes a structured explanation. For example:

Decision: INCLUDE (Confidence: 92%)

  • Population: Adults with type 2 diabetes (matches protocol ✓)
  • Intervention: 12-week exercise program (matches duration requirement ✓)
  • Comparison: Standard care control group (appropriate ✓)
  • Outcome: HbA1c measured at 12 weeks (primary outcome ✓)
  • Study Design: Randomized controlled trial (included design ✓)
  • Recommendation: Include for full-text review

Decision: EXCLUDE (Confidence: 88%)

  • Population: Children aged 8-12 (does not match adult criteria ✗)
  • Primary exclusion reason: Wrong population (age)
  • Note: Intervention and outcomes otherwise match protocol
  • Recommendation: Exclude

When to Override AI Decisions

Override to Include

Consider overriding an 'Exclude' decision when:

  • AI missed relevant information in the abstract
  • Technical terminology was misinterpreted
  • Your expert knowledge suggests the study is relevant despite apparent mismatch
  • Abstract is poorly written but study appears eligible
  • AI was overly strict in interpreting a criterion
  • Example: AI excluded for 'adolescent' population, but abstract defines this as 18-21 which matches your 18+ criterion

Override to Exclude

Consider overriding an 'Include' decision when:

  • You notice a disqualifying factor AI missed
  • Abstract was misleading but you have additional knowledge
  • Study is duplicate or secondary analysis of known excluded study
  • Intervention detail in abstract doesn't actually match your protocol
  • Example: AI included based on 'cognitive therapy' but you know from author that this is not CBT as required

How to Override

There are two ways to override an AI decision:

  • From the screening drawer — click the decision pill on the row to flip it. The change is recorded immediately and sent to the assistant as screening context on your next chat message, so the AI is aware of the override when answering questions or running subsequent steps.
  • From the paper review sheet — open a study, click the Override dropdown at the top of the assessment panel, and pick Include / Exclude / Uncertain. The decision banner switches to a Human Override badge.

You can also override from the chat directly — for example, "include the Chen 2019 trial — the abstract doesn't mention randomisation but the registered protocol confirms it." The assistant applies the override and updates the table. This is the right path when the override needs explanation or when you want to apply a rule across many studies ("exclude all studies with fewer than 50 participants").

Notes you add via the paper review sheet are persisted with the study and visible to anyone you share the review with. Overrides without notes are still recorded and visible in the audit trail (AI Decision vs Human Override badge).

Confidence percentages are visible in the paper review sheet during fulltext screening. During abstract screening you'll see the AI's reasoning but not a numeric confidence — the AI marks borderline cases as Uncertain so you can prioritise them.

Best Practices for AI-Assisted Screening

Trust but Verify

  • AI is a powerful assistant but not infallible
  • Review all 'Uncertain' and low-confidence decisions personally
  • Spot-check a random sample of high-confidence decisions (10-20%)
  • Always review all 'Include' decisions more carefully than excludes
  • Use AI to save time, not to replace expert judgment

Calibrate Your Protocol

If you notice AI consistently making errors:

  1. 1Review your protocol language for ambiguity
  2. 2Add clarifying details to inclusion/exclusion criteria
  3. 3Provide examples of borderline cases in protocol notes
  4. 4Re-run AI screening on affected studies after protocol updates
  5. 5AI performance improves with clearer, more specific protocols

Document Your Overrides

When overriding, always add detailed notes:

  • What AI got wrong or missed
  • What information led you to a different decision
  • Any special knowledge or context applied
  • This creates an audit trail for transparency
  • Helps co-reviewers understand your reasoning
  • Useful when writing methods section of systematic review

AI screening can make incorrect decisions, including on apparently clear cases. Review every Uncertain and low-confidence decision, spot-check both inclusions and exclusions, and use the chat to investigate reasoning that does not match your protocol.