Table of Contents
- Introduction
- Personality Types
- Personality Structure
- Scripted Personalities
4.1. Processor Class
4.2. Key Methods
4.3. Workflow Execution
4.4. User Interaction
4.5. AI Querying
4.6. Asset Management
4.7. Text Processing
4.8. Code Handling - Advanced Features
5.1. Text-to-Image Generation
5.2. Settings System - Best Practices
- Conclusion
1. Introduction
LoLLMs (Lord of Large Language Multimodal Systems) is a powerful framework for creating AI personalities with advanced capabilities. This documentation focuses on developing scripted personalities, which offer more complex and interactive functionalities compared to standard personalities.
2. Personality Types
LoLLMs supports two types of personalities:
- Standard Personalities: Controlled primarily by a system prompt.
- Scripted Personalities: Utilize both a system prompt and a Python script for complex interactions and workflows.
3. Personality Structure
Personalities in LoLLMs are organized within the personalities zoo. Each personality resides in its own folder within a category folder. The structure is as follows:
personalities_zoo/
├── category_1/
│ ├── personality_1/
│ │ ├── config.yaml
│ │ ├── assets/
│ │ │ └── logo.png
│ │ ├── files/ (optional)
│ │ ├── audio/ (optional)
│ │ └── scripts/
│ │ └── processor.py
│ └── personality_2/
└── category_2/
└── ...
Key components:
config.yaml
: Contains metadata about the personality (name, author, description, etc.)assets/logo.png
: The personality’s logofiles/
: Optional folder for augmented personalities (used by the inner RAG system)audio/
: Optional folder for prerecorded audioscripts/processor.py
: Main script for scripted personalities
4. Scripted Personalities
Scripted personalities offer advanced control and functionality through Python code. The main component is the Processor
class in processor.py
.
4.1 Processor Class
The Processor
class inherits from APScript
and defines the behavior of the personality:
class Processor(APScript):
def __init__(self, personality: AIPersonality, callback: Callable = None) -> None:
# Initialize configuration and states
personality_config_template = ConfigTemplate([
# Configuration entries
])
personality_config_vals = BaseConfig.from_template(personality_config_template)
personality_config = TypedConfig(personality_config_template, personality_config_vals)
super().__init__(
personality,
personality_config,
states_list=[
{
"name": "idle",
"commands": {
"help": self.help,
},
"default": None
},
],
callback=callback
)
4.2 Key Methods
mounted()
: Triggered when the personality is mountedselected()
: Triggered when the personality is selectedinstall()
: Sets up necessary dependencieshelp(prompt="", full_context="")
: Provides help informationrun_workflow(prompt, previous_discussion_text, callback, context_details, client)
: Main method for executing the personality’s workflow
4.3 Workflow Execution
The run_workflow
method is the core of a scripted personality. It handles user input and orchestrates the personality’s response:
def run_workflow(self, prompt:str, previous_discussion_text:str="", callback: Callable[[str | list | None, MSG_OPERATION_TYPE, str, AIPersonality| None], bool]=None, context_details:dict=None, client:Client=None):
self.callback = callback
full_prompt = self.build_prompt_from_context_details(context_details)
out = self.fast_gen(full_prompt)
self.set_message_content(out)
4.4 User Interaction
Scripted personalities can interact with users through various methods:
set_message_content(text)
: Sets the current AI message outputadd_chunk_to_message_content(text)
: Adds a new chunk to the current messagestep_start(step_text)
andstep_end(step_text)
: Show workflow execution stepsjson(dict)
: Display JSON dataui(ui_string)
: Send complete UI with HTML/CSS/JavaScript
Example:
self.step_start("Processing user input")
result = self.process_input(prompt)
self.step_end("Input processed successfully")
self.json({"result": result})
4.5 AI Querying
Scripted personalities can query the AI for decision-making:
yes_no(question, context="", max_answer_length=50, conditioning="")
: Ask yes/no questionsmultichoice_question(question, possible_answers, context="", max_answer_length=50, conditioning="")
: Ask multiple-choice questions
Example:
if self.yes_no("Does the user want to generate an image?", context=prompt):
self.generate_image()
4.6 Asset Management
Access personality and discussion assets:
- Personality assets:
self.personality.text_files
self.personality.image_files
self.personality.audio_files
- Discussion assets:
client.discussion.text_files
client.discussion.image_files
client.discussion.audio_files
4.7 Text Processing
Use the summarization feature for processing large texts:
summary = self.summarize_text(
text,
summary_instruction="Summarize the key points",
max_summary_size=512,
summary_mode=SUMMARY_MODE.SUMMARY_MODE_HIERARCHICAL
)
4.8 Code Handling
Extract and process code blocks from AI responses:
code_blocks = self.extract_code_blocks(ai_response)
for block in code_blocks:
print(f"File: {block['file_name']}, Type: {block['type']}")
print(block['content'])
5. Advanced Features
5.1 Text-to-Image Generation
Generate images using the built-in text-to-image service:
file, infos = self.tti.paint(
positive_prompt,
negative_prompt,
self.personality_config.sampler_name,
self.personality_config.seed,
self.personality_config.scale,
self.personality_config.steps,
self.personality_config.img2img_denoising_strength,
width=self.personality_config.width,
height=self.personality_config.height,
output_path=client.discussion.discussion_folder
)
escaped_url = discussion_path_to_url(file)
self.set_message_content(f"Generated image: {escaped_url}")
5.2 Settings System
Implement customizable settings for your personality:
personality_config_template = ConfigTemplate([
{"name": "image_size", "type": "int", "value": 512, "help": "Size of generated images"},
{"name": "style", "type": "string", "options": ["realistic", "cartoon", "abstract"], "value": "realistic", "help": "Image generation style"}
])
Access settings in your code:
image_size = self.personality_config.image_size
style = self.personality_config.style
6. Best Practices
- Use the step system to provide clear progress indicators to users.
- Implement error handling and provide informative error messages.
- Use the settings system to make your personality customizable.
- Leverage the AI querying methods for dynamic decision-making.
- Utilize assets and discussion files for context-aware responses.
- Implement the
help
method thoroughly to guide users.
7. Conclusion
Scripted personalities in LoLLMs offer powerful capabilities for creating advanced AI interactions. By leveraging the provided methods and features, developers can create rich, interactive, and context-aware AI personalities that can perform complex tasks and provide engaging user experiences.
Remember to test your personality thoroughly and consider edge cases in user interactions. The flexibility of the scripted approach allows for continuous improvement and expansion of your AI personality’s capabilities.