LoLLMs

LoLLMs Personality Development Documentation

23 August 2024 Non classé

Table of Contents

  1. Introduction
  2. Personality Types
  3. Personality Structure
  4. Scripted Personalities
    4.1. Processor Class
    4.2. Key Methods
    4.3. Workflow Execution
    4.4. User Interaction
    4.5. AI Querying
    4.6. Asset Management
    4.7. Text Processing
    4.8. Code Handling
  5. Advanced Features
    5.1. Text-to-Image Generation
    5.2. Settings System
  6. Best Practices
  7. Conclusion

1. Introduction

LoLLMs (Lord of Large Language Multimodal Systems) is a powerful framework for creating AI personalities with advanced capabilities. This documentation focuses on developing scripted personalities, which offer more complex and interactive functionalities compared to standard personalities.

2. Personality Types

LoLLMs supports two types of personalities:

  1. Standard Personalities: Controlled primarily by a system prompt.
  2. Scripted Personalities: Utilize both a system prompt and a Python script for complex interactions and workflows.

3. Personality Structure

Personalities in LoLLMs are organized within the personalities zoo. Each personality resides in its own folder within a category folder. The structure is as follows:

personalities_zoo/
├── category_1/
│   ├── personality_1/
│   │   ├── config.yaml
│   │   ├── assets/
│   │   │   └── logo.png
│   │   ├── files/ (optional)
│   │   ├── audio/ (optional)
│   │   └── scripts/
│   │       └── processor.py
│   └── personality_2/
└── category_2/
    └── ...

Key components:

  • config.yaml: Contains metadata about the personality (name, author, description, etc.)
  • assets/logo.png: The personality’s logo
  • files/: Optional folder for augmented personalities (used by the inner RAG system)
  • audio/: Optional folder for prerecorded audio
  • scripts/processor.py: Main script for scripted personalities

4. Scripted Personalities

Scripted personalities offer advanced control and functionality through Python code. The main component is the Processor class in processor.py.

4.1 Processor Class

The Processor class inherits from APScript and defines the behavior of the personality:

class Processor(APScript):
    def __init__(self, personality: AIPersonality, callback: Callable = None) -> None:
        # Initialize configuration and states
        personality_config_template = ConfigTemplate([
            # Configuration entries
        ])
        personality_config_vals = BaseConfig.from_template(personality_config_template)
        personality_config = TypedConfig(personality_config_template, personality_config_vals)

        super().__init__(
            personality,
            personality_config,
            states_list=[
                {
                    "name": "idle",
                    "commands": {
                        "help": self.help,
                    },
                    "default": None
                },
            ],
            callback=callback
        )

4.2 Key Methods

  • mounted(): Triggered when the personality is mounted
  • selected(): Triggered when the personality is selected
  • install(): Sets up necessary dependencies
  • help(prompt="", full_context=""): Provides help information
  • run_workflow(prompt, previous_discussion_text, callback, context_details, client): Main method for executing the personality’s workflow

4.3 Workflow Execution

The run_workflow method is the core of a scripted personality. It handles user input and orchestrates the personality’s response:

def run_workflow(self, prompt:str, previous_discussion_text:str="", callback: Callable[[str | list | None, MSG_OPERATION_TYPE, str, AIPersonality| None], bool]=None, context_details:dict=None, client:Client=None):
    self.callback = callback
    full_prompt = self.build_prompt_from_context_details(context_details)
    out = self.fast_gen(full_prompt)
    self.set_message_content(out)

4.4 User Interaction

Scripted personalities can interact with users through various methods:

  • set_message_content(text): Sets the current AI message output
  • add_chunk_to_message_content(text): Adds a new chunk to the current message
  • step_start(step_text) and step_end(step_text): Show workflow execution steps
  • json(dict): Display JSON data
  • ui(ui_string): Send complete UI with HTML/CSS/JavaScript

Example:

self.step_start("Processing user input")
result = self.process_input(prompt)
self.step_end("Input processed successfully")

self.json({"result": result})

4.5 AI Querying

Scripted personalities can query the AI for decision-making:

  • yes_no(question, context="", max_answer_length=50, conditioning=""): Ask yes/no questions
  • multichoice_question(question, possible_answers, context="", max_answer_length=50, conditioning=""): Ask multiple-choice questions

Example:

if self.yes_no("Does the user want to generate an image?", context=prompt):
    self.generate_image()

4.6 Asset Management

Access personality and discussion assets:

  • Personality assets:
  • self.personality.text_files
  • self.personality.image_files
  • self.personality.audio_files
  • Discussion assets:
  • client.discussion.text_files
  • client.discussion.image_files
  • client.discussion.audio_files

4.7 Text Processing

Use the summarization feature for processing large texts:

summary = self.summarize_text(
    text,
    summary_instruction="Summarize the key points",
    max_summary_size=512,
    summary_mode=SUMMARY_MODE.SUMMARY_MODE_HIERARCHICAL
)

4.8 Code Handling

Extract and process code blocks from AI responses:

code_blocks = self.extract_code_blocks(ai_response)
for block in code_blocks:
    print(f"File: {block['file_name']}, Type: {block['type']}")
    print(block['content'])

5. Advanced Features

5.1 Text-to-Image Generation

Generate images using the built-in text-to-image service:

file, infos = self.tti.paint(
    positive_prompt,
    negative_prompt,
    self.personality_config.sampler_name,
    self.personality_config.seed,
    self.personality_config.scale,
    self.personality_config.steps,
    self.personality_config.img2img_denoising_strength,
    width=self.personality_config.width,
    height=self.personality_config.height,
    output_path=client.discussion.discussion_folder
)

escaped_url = discussion_path_to_url(file)
self.set_message_content(f"Generated image: {escaped_url}")

5.2 Settings System

Implement customizable settings for your personality:

personality_config_template = ConfigTemplate([
    {"name": "image_size", "type": "int", "value": 512, "help": "Size of generated images"},
    {"name": "style", "type": "string", "options": ["realistic", "cartoon", "abstract"], "value": "realistic", "help": "Image generation style"}
])

Access settings in your code:

image_size = self.personality_config.image_size
style = self.personality_config.style

6. Best Practices

  1. Use the step system to provide clear progress indicators to users.
  2. Implement error handling and provide informative error messages.
  3. Use the settings system to make your personality customizable.
  4. Leverage the AI querying methods for dynamic decision-making.
  5. Utilize assets and discussion files for context-aware responses.
  6. Implement the help method thoroughly to guide users.

7. Conclusion

Scripted personalities in LoLLMs offer powerful capabilities for creating advanced AI interactions. By leveraging the provided methods and features, developers can create rich, interactive, and context-aware AI personalities that can perform complex tasks and provide engaging user experiences.

Remember to test your personality thoroughly and consider edge cases in user interactions. The flexibility of the scripted approach allows for continuous improvement and expansion of your AI personality’s capabilities.