Evaluators

Demo • Introduction • Quickstart

Evaluators help you to measure the attributes of LLM-generated text through the lens of learning science:

• Optimize features: Sharpen AI-generated content so it aligns with pedagogy and your goals.
• Maintain performance: Use evaluators as product analytics to confirm LLM output stays on target.
• Select models: Compare candidate models by testing their output against the evaluators.

Repository contents

Folder	Contents
evals	Evaluators (with prompts, input/output schemas, and Python notebooks), grouped by family: Text Complexity, Feedback, Academic Standards Alignment, and Durable Skills
sdks	Python and TypeScript
demos	Sample apps that demonstrate SDK use
datasets	Expert-annotated datasets used to create and validate evaluators
scripts	Repo check harness — run python3 scripts/check.py --fix before committing (see scripts/README)
LICENSE	Open source license details

Quickstart

Access method	When to use	How to use
Evaluators playground	For a quick demo of how evaluators work	1. Open the Evaluators playground 2. Choose an evaluator family and provide your inputs 3. Click Evaluate, then expand each result for its score and reasoning
SDK	To integrate into your TypeScript or Python project	1. Install: npm install @learning-commons/evaluators or pip install learning-commons-evaluators 2. Import an evaluator, pass your API key, and call evaluate 3. See the TypeScript and Python SDK READMEs for details
Python notebook	For quick prototyping	1. Start Jupyter Lab: jupyter lab (opens at http://localhost:8888) 2. Browse into evals/ and open the evaluator you want to try 3. Copy your text into the last code cell and run it You can also use an IDE with Python and Jupyter support, such as VS Code with Microsoft's Python and Jupyter extensions.
Local	To evaluate inputs locally on your machine	Clone this repository and follow the directions for your platform: - Setup on Mac/Linux - Setup on Windows

Available evaluators

Evaluator family	Evaluator
Text Complexity - Docs	Grade Level Appropriateness - Docs
	Background Knowledge Demands - Docs
	Vocabulary Complexity - Docs
	Sentence Structure - Docs
	Meaning Directness - Docs
	Purpose Clarity - Docs
	Organizational Structure - Docs
	Reference Knowledge Demands - Docs
Feedback - Docs	Strength Acknowledgment - Docs
	Revision Actionability - Docs
	Student Response Specificity - Docs
	Revision Accuracy - Docs
	Revision Manageability - Docs
	Withholding Answers - Docs
	Tone Appropriateness - Docs
Academic Standards - Docs	Math Standards Alignment - Docs
Durable Skills - Docs	Critical Thinking - Docs

Support & feedback

If you have questions or feedback, open an issue or reach out to us at support@learningcommons.org. To report a security issue, please disclose responsibly by contacting us at security@learningcommons.org.

Sign up for a Learning Commons Platform account to stay up-to-date about the latest updates and releases.

Disclaimer

Use of the resources provided in this repository is subject to our Terms of Use.