PromptJ is a specialized platform for organizing a centralized repository of trained language models (LLM), prompts (queries for interacting with models) and metadata. This server not only provides functions for storing and managing models, but also offers advanced capabilities for remote access, ratings, evaluating the effectiveness of prompts and maintaining prompt registries.
Main server functions:
- Storing LLM models and prompts:
- Centralized storage for various versions of LLM models (e.g. Mistral, LLaMA, etc.).
- Support for storing and versioning prompts (change history for each prompt).
- Structuring prompts by tasks, categories or topics to ensure convenient navigation and quick access to the necessary queries.
- Remote access and RESTful API:
- RESTful API and WebSocket support for remote interaction with the model and prompt storage.
- Support for user authorization and authentication via tokens (OAuth2, JWT), user roles (administrator, user, guest).
- Ability to upload and download LLM models, prompts and related data from remote locations via API.
- Integration with external systems via SDK and libraries in Java.
- Ratings and evaluation of prompt performance:
- A rating system for evaluating prompts and models by users, based on the efficiency of task execution.
- Built-in metrics for assessing the quality of LLM work (accuracy, completeness, semantic relevance).
- The function of evaluating prompts by various criteria: execution speed, level of understanding of the model, applicability in different contexts.
- Comparative tables of model performance by prompts and tasks.
- Registry functions:
- Registry of LLM models with a description of their characteristics: number of parameters, architecture, training date, domain areas of application.
- Registry of prompts with metadata, including usage history, associated tasks and models with which they were used.
- Registry of versions of models and prompts with detailed descriptions of changes between versions (e.g. new layers in models, optimization of prompts, etc.).
- Moderation and access management:
- Ability to create public and private registries of models and prompts.
- Role-based access to models and prompts, including delimitation of rights to create, edit, delete and evaluate.
- Suggestion system for improving existing prompts or models based on user feedback.
- Interactive analytics and monitoring:
- Visualization of the use of models and prompts in real time, including statistics on queries, models and types of tasks.
- Ability to download analytical reports on server resource usage (CPU, memory) and LLM performance.
- Logging of all operations with prompts and models for auditing and debugging.
- Collaboration module:
- Ability to share prompts and models in teams with comments, tagging and edit histories.
- Functionality for suggestions for improving models and prompts based on analysis of previous interactions and data.
- Integration with training and experimental systems:
- Support for launching experiments on fine-tuning models through integration with training frameworks (e.g. PyTorch, TensorFlow).
- Support for automatic learning based on selected prompts and loaded data, with saving and versioning of results.
- Support for computing clusters for distributed model training and task execution.
PromptJ Architecture:
PromptJ is based on modern open-source solutions — the customizable e-commerce platform Shopizer and the open source chatbot TavernAI. We integrated both tools and connected them with the modules of the LLM ArtenaTech ecosystem, which allowed us to create flexible functionality for processing requests and working with ecosystem users. PromptJ supports extensive customization, allowing you to add new modules, so this solution can be adapted to the specific needs of customers, creating unique and convenient solutions
for your business.
- Frontend:
- Web interface for management and monitoring, including admin and analytics panels, end-user access and API documentation.
- Backend:
- Microservice architecture for scalability and support for various models, with a database for storing models, prompts and associated metadata.
- Database:
- Using relational (PostgreSQL) and non-relational databases (MongoDB) to store structured and unstructured data.
- Data Storage:
- Integration with object storage for storing large models (e.g. AWS S3, Google Cloud Storage).
- Security:
- Data encryption, authentication and authorization, operation logging and unauthorized access prevention systems.
- Deployment:
- AWS
- docker & k8s