Overview
All submitted agents undergo a comprehensive quality assurance review to ensure they meet our standards for functionality, safety, and user experience. Our QA process evaluates three main areas: flow structure, best practices compliance, and overall quality.
This document outlines the specific criteria we evaluate during the QA process to help you design agents that pass review on the first submission.
Detailed Agent Flow Review
Your agent submission will be analyzed to verify the following components are clearly defined:
Step-by-step flow documentation: Each step in your agent's workflow must be identifiable and logical.
Model configuration: The AI model used at each step, including temperature settings.
Tool availability: All tools accessible to the agent at each step.
Prompt design: A clear executive summary of the prompting strategy for each agent step.
Best Practices Compliance
Your agent will be evaluated against the following criteria to ensure it adheres to industry standards and our platform requirements:
1) Temperature Settings
Temperature settings must be appropriate for the task at each step. Consider the balance between creativity and consistency based on your agent's prompt and available tools. Typically you want a lower temperature to guide the agent in accuracy rather than hallucination.
2) Tool Selection
The tools assigned to each step must be reasonable and sufficient to accomplish both the individual step's objectives and the overall agent flow goals.
3) API Tool Usage
All API tools must be properly configured and aligned with the stated objectives of your agent.
4) External Tool Usage
We provide an approved set of tools that agents should rely on whenever possible. The reviewer flags any use of external tools such as Serp or Google and assesses whether their inclusion is justified or requires revision.
5) Fail-Safe Mechanisms
Your agent must include safeguards to protect client databases from corruption. Specifically, the agent should not insert data into the database when uncertain about the appropriate action. Implementation of validation checks and confirmation steps where database modifications occur are required.
6) Missing Data Handling
Your agent must gracefully handle incomplete datasets. The design should not be overfitted to training data and must include logic to manage scenarios where expected data fields are missing or incomplete.
7) Data Volume Management
Your agent must be designed to handle reasonable amounts of data without overwhelming the context window. Tool calls should specify:
Pagination parameters where applicable
Field-specific returns to limit data volume
Strategies to prevent context window overflow
8) Overall Craftsmanship
An expert review will assess whether your agent flow is useful, helpful, and well-crafted. This includes evaluating logical flow, efficiency, and adherence to Flowise AI agent design principles.
9) Communication Quality
If your agent communicates with users via emails, activity feed posts, or other channels, the communication must be professional, clear, and appropriate for the context.
10) Action Documentation
Your agent should document its actions appropriately through one or more of the following methods:
Email notifications
Activity feed posts
Record updates or notifications
Other appropriate logging mechanisms
Users should be able to understand what actions the agent has taken, particularly when data has been modified.
11) Moderation and Guard Rails
Your agent must include moderation mechanisms, limits, or guard rails to prevent unexpected AI behavior. This includes handling edge cases, preventing infinite loops, and managing unusual or malformed inputs.
Final Quality Score
After evaluating all the above criteria, your agent will receive a numeric quality score. This score reflects the overall design quality, safety, and adherence to best practices. Agents must meet minimum quality thresholds to be approved for publication in the agent store.
Tips for Successful Submission
Test thoroughly: Validate your agent with diverse datasets, including incomplete or edge-case data.
Document clearly: Ensure each step's purpose and logic is evident in your flow design.
Prioritize safety: Always implement fail-safes when your agent interacts with databases or makes automated decisions.
Communicate transparently: Design your agent to inform users about actions taken.
Use appropriate tools: Leverage Yardi tools when available for optimal integration with our platform.
If you have questions about any of these criteria or need clarification before submitting your agent, please contact our support team.