Execution & Activity

Learn how to monitor, debug, and optimize tool execution in Prisme.ai agents using Activity tracking and event management

Understanding how tools execute and monitoring their performance is critical for developing reliable tool-using agents. Prisme.ai provides powerful Activity tracking capabilities that give you visibility into every aspect of tool execution, from request processing to result delivery.

The Value of Activity Tracking

Activity tracking in Prisme.ai provides:

Visibility

Complete transparency into how tools execute

Diagnostics

Powerful debugging for error resolution

Analytics

Performance metrics and usage patterns

Logging

Comprehensive audit trails of all operations

Monitoring

Real-time observation of system behavior

Optimization

Data-driven insights for performance improvements

Understanding the Event System

At the core of Prisme.ai’s Activity tracking is its event-driven architecture:

Key Components

Events

Structured messages representing actions, status changes, or notifications

Key attributes:

• Event type
• Timestamp
• Source identifier
• Payload data
• Request context

Publishers

Components that emit events when actions occur

Examples:

• Tool execution requests
• Automation processes
• User interactions
• System status changes

Subscribers

Components that listen for and process specific events

Examples:

• Automation workflows
• Activity monitors
• Logging systems
• Analytics processors

Event Bus

The central message broker that routes events

Responsibilities:

• Reliable message delivery
• Event persistence
• Publisher/subscriber coordination
• Load management

The Execution Lifecycle

Tool execution follows a well-defined lifecycle that generates events at each stage:

1. Request Initiation

   The process begins when an agent decides to use a tool.

   Events generated:

   • tool.request.initiated
   • tool.parameter.validation.started

   Key information:

   • Tool name and parameters
   • Request source (conversation ID, user ID)
   • Timestamp and request ID
   • Context information
2. Parameter Validation

   The system validates that all required parameters are present and properly formatted.

   Events generated:

   • tool.parameter.validation.completed
   • tool.parameter.validation.failed (if issues occur)

   Key information:

   • Validation status
   • Error details (if any)
   • Parameter transformations
   • Schema conformance
3. Execution

   The actual tool code runs, performing its designated function.

   Events generated:

   • tool.execution.started
   • tool.execution.step.completed (for multi-step tools)
   • tool.execution.completed

   Key information:

   • Execution duration
   • Resource utilization
   • Intermediate state
   • External system interactions
4. Result Processing

   The system processes the tool’s output for agent consumption.

   Events generated:

   • tool.result.processing.started
   • tool.result.processing.completed

   Key information:

   • Raw output data
   • Processed results
   • Formatting operations
   • Output size and structure
5. Response Delivery

   The formatted results are returned to the agent.

   Events generated:

   • tool.response.delivery.started
   • tool.response.delivery.completed

   Key information:

   • Final response format
   • Delivery time
   • End-to-end latency
   • Response size

Accessing Activity in Prisme.ai

You can monitor and analyze tool execution through several interfaces:

Activity View

The primary interface for monitoring and debugging tool execution.

Key features:

• Chronological event listing
• Filtering by event type, status, or date range
• Detailed event inspection
• Real-time monitoring
• Search capabilities

Access:

• Navigate to the “Activity” tab in your AI Builder workspace
• Use filtering to focus on relevant events
• Click on events to see detailed information
• Toggle real-time updates for live monitoring

Event API

Programmatic access to event data for custom monitoring and analysis.

Key capabilities:

• Query historical events
• Filter by type, source, or timestamp
• Subscribe to real-time event streams
• Integrate with external monitoring tools

Example usage:

// Fetch recent events from a specific workspace
  const options = {
    method: 'GET',
    headers: {
      'X-Prismeai-API-Key': '<api-key>'
    }
  };
  
  fetch('https://api.eda.prisme.ai/v2/workspaces/{workspaceId}/events', options)
    .then(response => response.json())
    .then(response => {
      console.log('Event data:', response);
      // Process event data
    })
    .catch(err => console.error('Error fetching events:', err));

Dashboard Visualizations

All activity is stored in your own OpenSearch/Elasticsearch. Build dashboards for performance monitoring and trend analysis.

Key metrics:

• Tool usage frequency
• Execution latency
• Error rates
• Resource utilization
• Success/failure trends

Features:

• Customizable dashboards
• Real-time updates
• Historical comparisons
• Threshold alerting
• Export capabilities

Common Event Types

Prisme.ai’s event system uses a hierarchical naming convention for clarity and filtering:

Tool Request Events

Events related to the initial tool request and setup.

Key event types:

• tool.request.initiated: Initial request for tool execution
• tool.request.validated: Request passed preliminary validation
• tool.request.rejected: Request failed preliminary checks

Example event data:

{
  "type": "tool.request.initiated",
  "timestamp": "2025-03-15T14:32:56.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "parameters": {
      "symbol": "AAPL",
      "dateRange": "5d"
    },
    "requester": {
      "conversationId": "conv-789012",
      "userId": "user-345678"
    }
  }
}

Execution Events

Events related to the actual tool execution process.

Key event types:

• tool.execution.started: Tool execution has begun
• tool.execution.step.completed: A step in multi-step execution completed
• tool.execution.completed: Tool execution finished successfully
• tool.execution.failed: Tool execution encountered an error

Example event data:

{
  "type": "tool.execution.completed",
  "timestamp": "2025-03-15T14:33:01.456Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "duration": 4667,
    "resourceUsage": {
      "cpuTime": 1250,
      "memoryPeak": 256384
    },
    "externalCalls": [
      {
        "service": "finance-api",
        "endpoint": "/stock/price",
        "duration": 3890
      }
    ]
  }
}

Result Events

Events related to the processing and delivery of tool results.

Key event types:

• tool.result.generated: Raw results from tool execution
• tool.result.processed: Results after formatting and preparation
• tool.result.delivered: Final delivery to the requester

Example event data:

{
  "type": "tool.result.processed",
  "timestamp": "2025-03-15T14:33:02.123Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "rawResultSize": 8712,
    "processedResultSize": 2340,
    "resultSummary": {
      "symbol": "AAPL",
      "currentPrice": 187.43,
      "priceChange": 2.31,
      "percentChange": 1.25,
      "dataPoints": 5
    }
  }
}

Error Events

Events related to errors during the tool execution lifecycle.

Key event types:

• tool.error.validation: Error during parameter validation
• tool.error.execution: Error during tool execution
• tool.error.timeout: Execution exceeded allocated time
• tool.error.resource: Resource limit exceeded

Example event data:

{
  "type": "tool.error.execution",
  "timestamp": "2025-03-15T14:33:00.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "errorCode": "API_UNAVAILABLE",
    "errorMessage": "External finance API returned status 503",
    "errorDetails": {
      "httpStatus": 503,
      "responseBody": "Service temporarily unavailable",
      "endpoint": "https://finance-api.example.com/stock/price"
    },
    "executionStage": "external_api_call"
  }
}

Debugging with Activity

Activity tracking provides powerful capabilities for diagnosing and resolving issues:

1. Identify the Issue

   Use filtering to locate relevant events.

   Effective filtering strategies:

   • Search for specific request IDs
   • Filter by error event types
   • Focus on a specific time range
   • Filter by tool name or source
   • Search for specific error codes
2. Trace Execution Flow

   Follow the sequence of events to understand the execution path.

   Key analysis techniques:

   • Map the complete event sequence
   • Identify missing or unexpected events
   • Analyze timing between events
   • Examine state changes across events
   • Look for execution branches
3. Examine Event Details

   Inspect detailed event data for diagnostic information.

   Focus areas:

   • Parameter values and transformations
   • Error details and context
   • Performance metrics
   • External system interactions
   • Resource utilization
4. Compare with Successful Executions

   Analyze differences between failed and successful runs.

   Comparison strategies:

   • Identify parameter differences
   • Compare timing patterns
   • Analyze resource usage
   • Examine external factors
   • Look for environmental changes
5. Implement and Verify Fixes

   Make changes and confirm resolution through activity monitoring.

   Verification approach:

   • Test the same parameters
   • Monitor in real-time
   • Verify complete event sequence
   • Check performance metrics
   • Test edge cases

Monitoring Performance with Activity

Activity data is invaluable for performance optimization:

Latency Analysis

Identify and address performance bottlenecks.

Key metrics to monitor:

• End-to-end latency: Total time from request to response
• Execution time: Time spent in actual tool execution
• External call latency: Time spent waiting for external services
• Processing overhead: Time spent in non-execution activities

Implementation approach:

// Calculate average latencies by tool
function analyzeToolLatencies(events, timeRange) {
  const executions = {};
  
  // Group events by request ID
  events.forEach(event => {
    if (!executions[event.data.requestId]) {
      executions[event.data.requestId] = {
        tool: event.data.toolName,
        events: []
      };
    }
    executions[event.data.requestId].events.push(event);
  });
  
  // Calculate metrics for each execution
  const toolMetrics = {};
  Object.values(executions).forEach(execution => {
    const startEvent = execution.events.find(e => e.type === 'tool.request.initiated');
    const endEvent = execution.events.find(e => e.type === 'tool.result.delivered');
    const execStartEvent = execution.events.find(e => e.type === 'tool.execution.started');
    const execEndEvent = execution.events.find(e => e.type === 'tool.execution.completed');
    
    if (startEvent && endEvent && execStartEvent && execEndEvent) {
      const tool = execution.tool;
      if (!toolMetrics[tool]) {
        toolMetrics[tool] = {
          count: 0,
          totalLatency: 0,
          totalExecTime: 0,
          totalOverhead: 0
        };
      }
      
      const latency = endEvent.timestamp - startEvent.timestamp;
      const execTime = execEndEvent.timestamp - execStartEvent.timestamp;
      const overhead = latency - execTime;
      
      toolMetrics[tool].count++;
      toolMetrics[tool].totalLatency += latency;
      toolMetrics[tool].totalExecTime += execTime;
      toolMetrics[tool].totalOverhead += overhead;
    }
  });
  
  // Calculate averages
  Object.keys(toolMetrics).forEach(tool => {
    const metrics = toolMetrics[tool];
    metrics.avgLatency = metrics.totalLatency / metrics.count;
    metrics.avgExecTime = metrics.totalExecTime / metrics.count;
    metrics.avgOverhead = metrics.totalOverhead / metrics.count;
    metrics.overheadPercentage = (metrics.avgOverhead / metrics.avgLatency) * 100;
  });
  
  return toolMetrics;
}

Error Rate Tracking

Monitor and minimize tool failures.

Key metrics to track:

• Overall error rate: Percentage of tool executions that fail
• Error breakdown by type: Distribution of different error categories
• Error trends: Changes in error rates over time
• Correlation analysis: Factors associated with higher error rates

Implementation approach:

// Calculate error rates by tool and type
function analyzeErrorRates(events, timeRange) {
  const toolStats = {};
  
  // Count total executions and errors
  events.forEach(event => {
    const tool = event.data.toolName;
    
    if (!toolStats[tool]) {
      toolStats[tool] = {
        totalExecutions: 0,
        errors: {
          total: 0,
          byType: {}
        }
      };
    }
    
    if (event.type === 'tool.execution.started') {
      toolStats[tool].totalExecutions++;
    }
    
    if (event.type.startsWith('tool.error.')) {
      toolStats[tool].errors.total++;
      
      const errorType = event.type.replace('tool.error.', '');
      if (!toolStats[tool].errors.byType[errorType]) {
        toolStats[tool].errors.byType[errorType] = 0;
      }
      toolStats[tool].errors.byType[errorType]++;
    }
  });
  
  // Calculate rates
  Object.keys(toolStats).forEach(tool => {
    const stats = toolStats[tool];
    stats.errorRate = stats.errors.total / stats.totalExecutions;
    
    stats.errors.typeBreakdown = {};
    Object.keys(stats.errors.byType).forEach(errorType => {
      stats.errors.typeBreakdown[errorType] = 
        stats.errors.byType[errorType] / stats.errors.total;
    });
  });
  
  return toolStats;
}

Usage Pattern Analysis

Understand how tools are being used.

Key metrics to analyze:

• Usage frequency: How often each tool is called
• Parameter distributions: Common values and patterns
• Usage contexts: When and why tools are invoked
• User segmentation: Differences in tool usage across users

Implementation approach:

// Analyze tool usage patterns
function analyzeUsagePatterns(events, timeRange) {
  const usageStats = {
    byTool: {},
    byHour: Array(24).fill(0),
    byUser: {},
    parameterDistributions: {}
  };
  
  events.forEach(event => {
    if (event.type === 'tool.request.initiated') {
      const tool = event.data.toolName;
      const timestamp = new Date(event.timestamp);
      const hour = timestamp.getHours();
      const userId = event.data.requester?.userId || 'unknown';
      const parameters = event.data.parameters || {};
      
      // Count by tool
      if (!usageStats.byTool[tool]) {
        usageStats.byTool[tool] = 0;
      }
      usageStats.byTool[tool]++;
      
      // Count by hour
      usageStats.byHour[hour]++;
      
      // Count by user
      if (!usageStats.byUser[userId]) {
        usageStats.byUser[userId] = {
          total: 0,
          byTool: {}
        };
      }
      usageStats.byUser[userId].total++;
      if (!usageStats.byUser[userId].byTool[tool]) {
        usageStats.byUser[userId].byTool[tool] = 0;
      }
      usageStats.byUser[userId].byTool[tool]++;
      
      // Analyze parameters
      if (!usageStats.parameterDistributions[tool]) {
        usageStats.parameterDistributions[tool] = {};
      }
      Object.keys(parameters).forEach(param => {
        if (!usageStats.parameterDistributions[tool][param]) {
          usageStats.parameterDistributions[tool][param] = {};
        }
        
        const value = String(parameters[param]);
        if (!usageStats.parameterDistributions[tool][param][value]) {
          usageStats.parameterDistributions[tool][param][value] = 0;
        }
        usageStats.parameterDistributions[tool][param][value]++;
      });
    }
  });
  
  return usageStats;
}

Optimizing Tool Execution

Use Activity data to drive execution optimizations:

Parameter Optimization

Refine parameter handling based on usage patterns

Strategies:

• Add intelligent defaults for common values
• Implement parameter validation improvements
• Optimize parameter extraction from user inputs
• Add preprocessing for frequently used parameter formats

Execution Efficiency

Improve tool execution performance

Strategies:

• Identify and optimize slow execution steps
• Implement caching for frequent operations
• Reduce external API latency
• Optimize resource utilization

Error Reduction

Minimize tool execution failures

Strategies:

• Add robust error handling for common failures
• Implement automatic retries for transient issues
• Improve validation to catch problems early
• Create fallback mechanisms for critical functions

Result Processing

Enhance result formatting and delivery

Strategies:

• Optimize result formatting for LLM consumption
• Reduce result size for better context utilization
• Implement progressive result delivery for long operations
• Add enrichment for more valuable outputs

Custom Event Emission

You can emit custom events to enhance visibility into your tools:

Custom Event Types

Define your own event types for specialized monitoring.

Naming conventions:

custom.[workspace].[category].[action]

Examples:

• custom.finance-assistant.calculation.started
• custom.support-agent.ticket.created
• custom.inventory-tool.stock.checked

This allows for clear organization and filtering of your custom events.

Emitting Custom Events

Add custom event emission to your tools.

Example implementation:

slug: product-recommendation-tool
do:
  # Emit custom event to track recommendation request
  - emit:
      event: custom.product-tool.recommendation.requested
      data:
        userId: '{{event.data.userId}}'
        productCategory: '{{event.data.parameters.category}}'
        priceRange: '{{event.data.parameters.priceRange}}'
        timestamp: '{{now()}}'
  
  # Execute recommendation logic
  - ProductCatalog.search:
      category: '{{event.data.parameters.category}}'
      priceMin: '{{event.data.parameters.priceRange.min}}'
      priceMax: '{{event.data.parameters.priceRange.max}}'
      output: searchResults
  
  # Emit event for search completion
  - emit:
      event: custom.product-tool.search.completed
      data:
        resultsCount: '{{length(searchResults.products)}}'
        executionTime: '{{searchResults.executionTime}}'
  
  # Apply recommendation algorithm
  - RecommendationEngine.process:
      products: '{{searchResults.products}}'
      userPreferences: '{{event.data.parameters.preferences}}'
      output: recommendations
  
  # Emit final recommendation event
  - emit:
      event: custom.product-tool.recommendation.generated
      data:
        recommendationCount: '{{length(recommendations)}}'
        topRecommendationId: '{{recommendations[0].id}}'
        algorithmsApplied: '{{recommendations.algorithmsApplied}}'
  
  # Return results
  - set:
      name: output
      value:
        recommendations: '{{recommendations}}'

This approach:

• Provides detailed visibility into tool execution stages
• Captures business-specific metrics
• Enables more nuanced performance analysis
• Creates richer debugging context

Event Payload Best Practices

Design effective event payloads for maximum utility.

Recommendations:

• Include identifiers: Always add request IDs, user IDs, and other correlation identifiers
• Add timestamps: Include timing information for performance analysis
• Provide context: Add relevant business context to make events meaningful
• Size appropriately: Include useful data without making events too large
• Structure consistently: Use consistent naming and structure across related events

Example event structure:

{
  "type": "custom.order-tool.validation.completed",
  "timestamp": "2025-03-15T15:45:12.678Z",
  "source": "workspace:commerce-assistant",
  "data": {
    "requestId": "req-abc123",
    "orderId": "ORD-98765",
    "validationResults": {
      "inventory": {
        "status": "passed",
        "duration": 124
      },
      "pricing": {
        "status": "passed",
        "duration": 86
      },
      "shipping": {
        "status": "warning",
        "message": "Delivery may be delayed",
        "duration": 231
      }
    },
    "totalDuration": 441,
    "validationPassed": true
  }
}

Real-Time Monitoring

For critical tools, implement real-time monitoring using Activity:

1. Set Up Real-Time Dashboards

   Create specialized dashboards for key metrics.

   Include dashboards for:

   • Error rates and types
   • Execution latency
   • Usage volume
   • Resource utilization
   • Business-specific metrics
2. Configure Alerts

   Define thresholds and conditions for notifications.

   Alert types to consider:

   • Error rate spikes
   • Latency threshold violations
   • Abnormal usage patterns
   • Resource exhaustion warnings
   • Custom business metric anomalies
3. Implement Event Subscribers

   Create subscribers for real-time processing.

   Example implementation:

   slug: error-rate-monitor
   when:
     event: tool.error.*
   do:
     # Increment error counter
     - cache.increment:
         key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
         value: 1
         expire: 86400  # 24 hours
     
     # Get current hour's execution count
     - cache.get:
         key: 'executions:{{date(now(), "YYYY-MM-DD:HH")}}'
         output: executionCount
     
     # Get current hour's error count
     - cache.get:
         key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
         output: errorCount
     
     # Calculate error rate
     - set:
         name: errorRate
         value: '{{errorCount / executionCount}}'
     
     # Check if threshold exceeded
     - conditions:
         '{{errorRate > 0.05 && executionCount > 10}}':  # 5% error rate with at least 10 executions
           - notification.send:
               channel: 'slack'
               message: 'Alert: Tool error rate at {{errorRate * 100}}% ({{errorCount}} errors out of {{executionCount}} executions) in the past hour'
   
4. Monitor Service Health

   Track overall system performance and health.

   Focus areas:

   • API response times
   • Service availability
   • Resource utilization
   • Queue lengths
   • Error distributions

Activity for Compliance and Audit

Beyond operational uses, Activity provides essential compliance capabilities:

Audit Trails

Maintain comprehensive records of all tool interactions

Key aspects:

• Complete chronological record of events
• User and system actions
• Parameter and result tracking
• Timing information
• Access evidence

Compliance Reporting

Generate reports for regulatory requirements

Capabilities:

• Data access audit reports
• Processing evidence
• Security monitoring
• Policy enforcement verification
• Chain of custody documentation

Forensic Analysis

Detailed investigation of specific incidents

Use cases:

• Security incident investigation
• Error root cause analysis
• User behavior auditing
• System interaction reconstruction
• Performance issue diagnostics

Records Retention

Maintain activity data according to policies

Considerations:

• Retention period configuration
• Data archiving strategies
• Privacy compliance
• Data minimization
• Access controls

Best Practices for Activity Utilization

Develop an Event Strategy

Plan your event ecosystem for maximum value.

Recommendations:

• Define an event taxonomy: Create a consistent naming and categorization system
• Identify key lifecycle events: Ensure all important state transitions emit events
• Balance detail and volume: Emit enough events for visibility without overwhelming storage
• Consider event consumers: Design with reporting, monitoring, and analysis needs in mind
• Document event types: Maintain clear documentation of all events and their meanings

Implementation approach:

Event Naming Structure:

[domain].[entity].[action]

Examples:
- tool.request.initiated
- order.payment.processed
- user.profile.updated
- system.resource.allocated

Implement Effective Filtering

Create filtering strategies for efficient data access.

Recommendations:

• Use consistent metadata: Add standard fields like request IDs to enable correlation
• Include categorization data: Add tags, types, and categories for efficient filtering
• Consider time-based access: Optimize for common time-range queries
• Balance filter granularity: Too granular filters can be as problematic as too broad ones
• Test filter performance: Ensure filtering is efficient at production scale

Example filtering approach:

// Effective event filtering
const events = await activity.queryEvents({
  // Time range
  timeRange: {
    start: new Date(Date.now() - (7 * 24 * 60 * 60 * 1000)), // 7 days ago
    end: new Date()
  },
  // Event types (using hierarchical pattern)
  eventTypes: ['tool.execution.*', 'tool.error.*'],
  // Source filter
  sources: ['workspace:customer-service'],
  // Metadata filters
  filters: [
    { field: 'data.requestId', operator: 'eq', value: 'req-123456' },
    { field: 'data.toolName', operator: 'in', value: ['CustomerLookup', 'OrderStatus'] }
  ],
  // Sorting
  sort: { field: 'timestamp', direction: 'desc' },
  // Pagination
  limit: 100,
  offset: 0
});

Design for Scalability

Ensure your activity management scales with usage.

Recommendations:

• Implement data lifecycle management: Archive older events to control storage growth
• Consider sampling for high-volume events: For very frequent events, consider sampling
• Optimize query patterns: Design queries to use available indexes
• Leverage aggregation: Use pre-aggregated metrics for frequent analyses
• Plan for growth: Design with future scale in mind

Example lifecycle implementation:

slug: activity-lifecycle-management
when:
  schedule: '0 0 * * *'  # Daily at midnight
do:
  # Archive events older than 30 days
  - activity.archiveEvents:
      olderThan: '{{date.add(now(), -30, "days")}}'
      destination: 'cold-storage'
  
  # Delete archived events older than 365 days
  - activity.deleteArchivedEvents:
      olderThan: '{{date.add(now(), -365, "days")}}'
  
  # Aggregate metrics for retention
  - activity.aggregateMetrics:
      timeRange:
        start: '{{date.add(now(), -2, "days")}}'
        end: '{{date.add(now(), -1, "days")}}'
      dimensions: ['toolName', 'hour', 'errorType']
      metrics: ['count', 'avgDuration', 'errorRate']
  
  # Emit completion event
  - emit:
      event: system.activity.lifecycle.completed
      data:
        timestamp: '{{now()}}'

Activity in AI Knowledge

For AI Knowledge users, Activity information is accessible through built-in interfaces:

1. Access the Activity Log

   Find the Activity section in AI Knowledge.

   Key features:

   • Chronological event listing
   • Filtering by event type and date
   • Search capabilities
   • Drill-down for details
2. Monitor Tool Usage

   Track how agents are using tools.

   Available information:

   • Tool usage frequency
   • Success and error rates
   • Performance metrics
   • Parameter patterns
3. Analyze Conversation Flows

   Examine how tools integrate into conversations.

   Analysis capabilities:

   • Tool usage in conversation context
   • User reactions to tool results
   • Multi-turn tool interactions
   • Conversation paths and patterns
4. Export Activity Data

   Extract data for external analysis.

   Export options:

   • CSV for spreadsheet analysis
   • JSON for programmatic processing
   • Filtered or complete data sets
   • Custom date ranges

Next Steps

Ready to implement effective execution monitoring and activity tracking? Continue with these resources:

Error Handling

Learn how to implement robust error management

AI Builder Event System

Explore the event system in greater depth