QueryCostMetricRecorder Class

reference

cosmos-db

observability

metrics

A comprehensive reference for the QueryCostMetricRecorder class that captures and records CosmosDB query costs as OpenTelemetry metrics, tracking Request Units (RU) consumption across database operations.

Author

Diginsight Team

Published

September 8, 2025

QueryCostMetricRecorder Class

The QueryCostMetricRecorder captures and records CosmosDB query costs as the diginsight.query_cost OpenTelemetry metric.

QueryCostMetricRecorder is part of the Observable extensions for CosmosDB that provide observability into database that are part of Diginsight.Components.Azure.

QueryCostMetricRecorder tracks Request Units (RU) consumption across your application’s database operations.

📋 Overview
- Key Features
- Metric Structure
🔍 Additional Details
⚙️ Configuration
🔧 Troubleshooting
📚 Reference
📖 Appendices
- Appendix A: Query Normalization Logic

📋 Overview

The QueryCostMetricRecorder works by listening to OpenTelemetry activities and automatically extracting query cost information from CosmosDB operations. When a database query completes, the recorder:

Detects CosmosDB Operations: Monitors activities for query_cost tags that indicate CosmosDB operations
Extracts Metrics: Captures the Request Units (RU) consumed by each query
Enriches with Context: Adds meaningful tags like method names, callers, database, and container information
Records Histogram: Stores the data as an OpenTelemetry histogram metric named diginsight.query_cost

Metric Structure

The diginsight.query_cost metric is recorded as a histogram with the following characteristics:

Name: diginsight.query_cost
Unit: RU (Request Units)
Type: Histogram
Description: “CosmosDB query cost in Request Units”

Key Features

Automatic Detection: No manual instrumentation required - works with existing Diginsight telemetry
Query Normalization: Optionally normalizes queries to reduce metric cardinality by replacing GUIDs, timestamps, and other high-cardinality values
Caller Tracking: Traces back through the call stack to identify the business logic methods that triggered queries
Configurable Tags: Flexible configuration for adding normalized queries and caller information to metrics
Error Resilience: Handles exceptions gracefully without impacting application performance

🔍 Additional Details

How Query Cost Detection Works

The recorder implements the IActivityListenerLogic interface to monitor OpenTelemetry activities. When an activity stops, it:

Checks for the presence of a query_cost tag
Validates that the cost is a positive number
Extracts contextual information from the activity and its parent chain
Applies configured enrichment and filtering
Records the metric with appropriate tags

Query Normalization

When AddNormalizedQueryTag is enabled, the recorder normalizes SQL queries to prevent metric cardinality explosion by replacing high-cardinality values with semantic placeholders:

// Original query
"SELECT * FROM c WHERE c.id = '123e4567-e89b-12d3-a456-426614174000' AND c.timestamp > '2023-01-01T10:30:00Z'"

// Normalized query
"SELECT * FROM c WHERE c.id = '{GUID}' AND c.timestamp > '{DATETIME}'"

The normalization process preserves query structure and intent while reducing cardinality. For detailed information about normalization patterns and implementation, see Appendix A: Query Normalization Logic.

Caller Chain Analysis

The recorder analyzes the activity parent chain to identify:

Entry Method: The top-level method that initiated the operation
Business Callers: Non-framework methods in the call chain (excludes “diginsight” internal calls)
Immediate Method: The direct method that executed the query

When AddQueryCallers is configured and IgnoreQueryCallers is specified, the recorder can exclude specific caller methods based on patterns to surface more meaningful business context:

Purpose of IgnoreQueryCallers

IgnoreQueryCallers is designed to skip over generic repository methods or infrastructure code that don’t provide meaningful business context, allowing the metric to capture the actual business operations that triggered the queries.

Example Scenario:

Call Chain: UserController.GetUserProfile() 
            → UserRepository.GetUserProfile() 
            → BaseCosmosDBRepository.GetItems() 
            → [CosmosDB Query Execution]

Without IgnoreQueryCallers:

caller1 = “BaseCosmosDBRepository.GetItems” (not very informative)
caller2 = “UserRepository.GetUserProfile” (more informative)

With IgnoreQueryCallers = [“BaseCosmosDBRepository*”]:

caller1 = “UserRepository.GetUserProfile” (business-relevant)
caller2 = “UserController.GetUserProfile” (even more context)

This configuration ensures metrics focus on business operations rather than infrastructure implementation details.

Pattern Matching Features

Exact Match: Method names that exactly match (case-insensitive) are excluded
Wildcard Patterns: Patterns containing * are supported for flexible matching (e.g., "*Controller*" excludes all controllers)
Performance Optimized: Uses compiled regex caching for efficient pattern matching

Tag Enrichment

Standard tags automatically added to metrics:

method: The immediate method that executed the query
entrymethod: The top-level entry point method
application: The application name (from entry assembly)
container: CosmosDB container name (if available)
database: CosmosDB database name (if available)

Optional tags (configurable):

query: Normalized query text
caller1, caller2, etc.: Business logic methods in the call chain

⚙️ Configuration

Configuration in appsettings.json

{
  "QueryCostMetricRecorderOptions": {
    "AddNormalizedQueryTag": false,
    "AddQueryCallers": 2,
    "IgnoreQueryCallers": [
      "BaseCosmosDBRepository*",
      "CosmosDbExtensions.*",
      "*Repository.GetItems",
      "*Repository.QueryAsync"
    ],
    "NormalizedQueryMaxLen": 500
  }
}

Configuration into the startup sequence

// In Program.cs or Startup.cs
services.AddCosmosDbQueryCostMetricRecorder();

Configure recorder options using the options pattern:

services.Configure<QueryCostMetricRecorderOptions>(options =>
{
    // Add normalized query text as a tag (default: false)
    // WARNING: This can increase metric cardinality
    options.AddNormalizedQueryTag = true;
    
    // Add caller method names as tags (default: 0)
    // Values: 0-5 representing number of caller levels to include
    options.AddQueryCallers = 2;
    
    // Exclude specific caller methods from metrics (default: empty array)
    // Supports exact matches and wildcard patterns with *
    options.IgnoreQueryCallers = new[] 
    { 
        "*Controller*",      // Exclude all controller methods
        "HealthCheckHandler", // Exclude specific method
        "*Middleware*",      // Exclude all middleware methods
        "Background*"        // Exclude methods starting with Background
    };
    
    // Configure query normalization length limit (default: 500)
    options.NormalizedQueryMaxLen = 300;
});

OpenTelemetry Integration

Ensure your OpenTelemetry configuration includes the Diginsight meter:

services.AddOpenTelemetry()
    .WithMetrics(builder =>
    {
        builder.AddMeter("Diginsight.Components.Azure");
        // Add other meters as needed
    });

Custom Registration

For advanced scenarios, you can create custom registration logic:

public class CustomQueryCostMetricRecorderRegistration : QueryCostMetricRecorderRegistration
{
    public CustomQueryCostMetricRecorderRegistration(QueryCostMetricRecorder recorder) 
        : base(recorder) { }

    public override bool ShouldListenTo(ActivitySource activitySource)
    {
        // Custom logic for which activity sources to monitor
        return activitySource.Name.Contains("MyApp.Data");
    }
}

// Register the custom implementation
services.AddCosmosDbQueryCostMetricRecorder<CustomQueryCostMetricRecorderRegistration>();

🔧 Troubleshooting

Common Issues

1. No Metrics Being Recorded

Check that:

CosmosDB operations are properly instrumented with Diginsight telemetry
Activities contain query_cost tags
The metric recorder is registered in the service collection
OpenTelemetry is configured to export the Diginsight.Components.Azure meter

2. High Metric Cardinality

If you experience high cardinality:

Disable AddNormalizedQueryTag if enabled
Reduce AddQueryCallers to 0 or 1
Use IgnoreQueryCallers to exclude generic repository methods and surface business operations instead
Review query normalization patterns
Implement custom IMetricRecordingFilter to exclude certain operations

3. Missing Context Information

Ensure that:

Diginsight telemetry is properly configured
Activities have appropriate parent-child relationships
Container and database information is being tagged by the CosmosDB instrumentation

4. Caller Filtering Not Working

If IgnoreQueryCallers patterns aren’t working as expected:

Check that the patterns match the actual operation names in your telemetry
Use exact method names for precise matching
Use wildcard patterns (*) for flexible matching
Enable debug logging to see which callers are being processed

Debugging

Enable detailed logging to troubleshoot issues:

services.Configure<LoggerFilterOptions>(options =>
{
    options.AddFilter("Diginsight.Components.Azure.Metrics.QueryCostMetricRecorder", LogLevel.Debug);
});

Performance Considerations

Query normalization has minimal performance impact but can be disabled if needed
Caller chain analysis processes only the activity hierarchy, not actual stack traces
Metric recording is asynchronous and won’t block database operations
Consider metric retention policies in your observability platform

Custom Filtering and Enrichment

Implement custom logic for filtering or enriching metrics:

// Custom filter to exclude certain operations
public class CustomMetricFilter : IMetricRecordingFilter
{
    public bool ShouldRecord(Activity activity)
    {
        // Skip recording for health check queries
        return !activity.OperationName.Contains("HealthCheck");
    }
}

// Custom enricher to add additional tags
public class CustomMetricEnricher : IMetricRecordingEnricher
{
    public IEnumerable<KeyValuePair<string, object?>> ExtractTags(Activity activity)
    {
        yield return new KeyValuePair<string, object?>("custom_tag", "custom_value");
    }
}

// Register custom implementations
services.AddSingleton<IMetricRecordingFilter, CustomMetricFilter>();
services.AddSingleton<IMetricRecordingEnricher, CustomMetricEnricher>();

📚 Reference

Classes and Interfaces

QueryCostMetricRecorder: Main recorder class that implements IActivityListenerLogic
QueryCostMetricRecorderOptions: Configuration options for the recorder
QueryCostMetricRecorderRegistration: Default registration that determines which activities to monitor
QueryMetrics: Static class containing the metric definitions
IMetricRecordingFilter: Interface for custom filtering logic
IMetricRecordingEnricher: Interface for custom tag enrichment

Extension Methods

AddCosmosDbQueryCostMetricRecorder(): Registers the recorder with default configuration
AddCosmosDbQueryCostMetricRecorder<TRegistration>(): Registers with custom registration logic

Metric Tags

Tag Name	Description	Always Present
`method`	Immediate method executing the query	✓
`entrymethod`	Top-level entry point method	✓
`application`	Application name	✓
`container`	CosmosDB container name	If available
`database`	CosmosDB database name	If available
`query`	Normalized query text	If configured
`caller1`, `caller2`, etc.	Business logic callers	If configured

Configuration Properties

Property	Type	Default	Description
`AddNormalizedQueryTag`	`bool`	`false`	Include normalized query text as tag
`AddQueryCallers`	`int`	`0`	Number of caller methods to include (0-5)
`IgnoreQueryCallers`	`string[]`	`[]`	Patterns to exclude specific caller methods
`NormalizedQueryMaxLen`	`int`	`500`	Maximum length for normalized queries (-1 for no limit)

IgnoreQueryCallers Pattern Examples

The IgnoreQueryCallers configuration supports flexible pattern matching to skip over uninformative infrastructure methods and surface meaningful business context:

Common Repository Patterns

// Skip generic repository methods to surface business operations
options.IgnoreQueryCallers = new[]
{
    "BaseCosmosDBRepository*",        // Skip base repository methods
    "*Repository.GetItems",           // Skip generic GetItems methods  
    "*Repository.QueryAsync",         // Skip generic query methods
    "*Repository.ExecuteAsync",       // Skip generic execute methods
    "CosmosDbExtensions.*",           // Skip extension method helpers
};

Before filtering: caller1 = “BaseCosmosDBRepository.GetItems” After filtering: caller1 = “UserRepository.GetUserProfile” (more meaningful)

Framework and Infrastructure Patterns

// Skip framework and infrastructure code
options.IgnoreQueryCallers = new[]
{
    "*Controller*",                   // Skip all controllers (focus on services)
    "*Middleware*",                   // Skip middleware components
    "HealthCheck*",                   // Skip health check methods
    "*Background*",                   // Skip background services
    "System.*",                       // Skip system methods
    "Microsoft.*",                    // Skip Microsoft framework methods
    "EntityFramework*",               // Skip EF infrastructure
    "*DbContext*"                     // Skip DbContext methods
};

Business-Focused Configuration

// Configuration to surface meaningful business operations
options.IgnoreQueryCallers = new[]
{
    // Repository layer (skip to get to service layer)
    "BaseRepository*",
    "*Repository.Get*",
    "*Repository.Query*", 
    "*Repository.Execute*",
    
    // Data access extensions (skip to get to business logic)
    "*Extensions.Query*",
    "*Extensions.Execute*",
    "CosmosDbExtensions.*",
    
    // Framework noise (skip completely)
    "Microsoft.*",
    "System.*"
};

Result: Metrics will show business service methods like:

UserService.GetUserProfile
OrderService.ProcessOrder
InventoryService.UpdateStock

Instead of generic repository methods like:

BaseRepository.GetItems
CosmosDbExtensions.QueryAsync

Pattern Matching Rules

Exact Match: String without * characters matches exactly (case-insensitive)
Wildcard Match: String with * characters is converted to regex pattern
Performance: Compiled regex patterns are cached for optimal performance
Error Handling: Invalid patterns are logged and ignored (won’t cause failures)
Caller Priority: When a caller is ignored, the next caller in the chain takes its place

📖 Appendices

Appendix A: Query Normalization Logic

The QueryCostMetricRecorder implements sophisticated query normalization to reduce metric cardinality while preserving query semantics and structure. This appendix provides detailed technical information about the normalization process.

Normalization Overview

Query normalization works by:

Extracting Queries: Handles both plain text and JSON-encoded query data
Pattern Matching: Applies regex patterns to identify and replace high-cardinality values
Structure Preservation: Maintains logical query structure for meaningful analysis
Length Management: Applies configurable length limits to prevent oversized metrics

Normalization Patterns

Basic Value Replacements

GUIDs: Replaced with {GUID}
- Pattern: \b[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{4}-?[0-9a-fA-F]{12}\b
- Example: '123e4567-e89b-12d3-a456-426614174000' → '{GUID}'
Large Numbers: Replaced with {NUMBER} (4+ digits)
- Pattern: \b\d{4,}\b
- Example: c.id = 12345678 → c.id = {NUMBER}
Long Strings: Replaced with {STRING} (6+ characters in quotes)
- Pattern: '[^']{6,}'
- Example: 'very-long-string-value' → '{STRING}'
DateTime Values: Replaced with {DATETIME}
- Pattern: \b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?(?:Z|[+-]\d{2}:\d{2})?\b
- Example: '2023-01-01T10:30:00Z' → '{DATETIME}'

CosmosDB-Specific Patterns

IN Clauses: Multiple values normalized
- Pattern: IN\s*\([^)]+\)
- Example: IN ('value1', 'value2', 'value3') → IN ({ITEMS})
BETWEEN Clauses: Range values normalized
- Pattern: BETWEEN\s+([""'][^""']*[""']|\{[A-Z]+\}|\d+)\s+AND\s+([""'][^""']*[""']|\{[A-Z]+\}|\d+)
- Example: BETWEEN '2023-01-01' AND '2023-12-31' → BETWEEN {VALUE} AND {VALUE}
ARRAY_CONTAINS Functions: Array values normalized
- Pattern: ARRAY_CONTAINS\s*\([^,]+,\s*([""'][^""']*[""']|\{[A-Z]+\})\)
- Example: ARRAY_CONTAINS(c.tags, 'specific-tag') → ARRAY_CONTAINS(c.tags, {VALUE})
ORDER BY Clauses: Field ordering preserved, values normalized
- Pattern: ORDER\s+BY\s+[^()]+?(ASC|DESC)?(?:\s*,\s*[^()]+?(ASC|DESC)?)*
- Example: ORDER BY c.timestamp DESC, c.id ASC → ORDER BY {FIELDS}

WHERE Clause Normalization

Complex WHERE clauses are normalized while preserving logical structure:

// Original
"WHERE (c.Type = 'User' AND c.Status = 'Active' AND c.CreatedDate > '2023-01-01')"

// Normalized (conditions sorted for consistency)
"WHERE (c.Type = 'User' AND c.CreatedDate > '{DATETIME}' AND c.Status = 'Active')"

Normalization Rules:

Type conditions are prioritized first
Remaining conditions are sorted alphabetically
Logical operators (AND/OR) are preserved
Parentheses grouping is maintained

JSON Query Extraction

The normalizer handles JSON-encoded queries commonly used in CosmosDB operations:

// Input: JSON-encoded query
"{\"query\":\"SELECT VALUE root FROM root WHERE root.Type = 'User'\"}"

// Extracted query for normalization
"SELECT VALUE root FROM root WHERE root.Type = 'User'"

// Final normalized result
"SELECT VALUE root FROM root WHERE root.Type = '{STRING}'"

Extraction Process:

Detect JSON structure using pattern matching
Parse JSON and extract query property
Apply normalization patterns to extracted query text
Handle extraction failures gracefully

Error Handling and Fallbacks

When normalization fails, the system provides meaningful fallbacks:

Query Prefix Extraction

// If normalization fails, extract meaningful prefix
"SELECT * FROM users WHERE complex_condition... (query normalization failed)"

// Preference for FROM clause extraction
"SELECT VALUE root FROM root ... (query normalization failed)"

Failure Scenarios Handled

JSON Parsing Errors: Falls back to treating input as plain text
Regex Pattern Failures: Logs warnings and continues with partial normalization
Length Limit Exceeded: Truncates with clear indication
Complete Normalization Failure: Returns {QUERY_NORMALIZATION_FAILED}

Performance Optimizations

Compiled Regex Patterns

All regex patterns are compiled with RegexOptions.Compiled for optimal performance:

private static readonly Regex GuidPattern = new Regex(@"pattern", RegexOptions.Compiled | RegexOptions.IgnoreCase);

Processing Order

Patterns are applied in specific order for efficiency:

Most specific patterns first (GUIDs, DateTime)
General patterns second (Numbers, Strings)
Structural patterns last (IN, BETWEEN, ORDER BY)

Whitespace Normalization

Consistent whitespace handling improves pattern matching:

Multiple spaces collapsed to single spaces
Leading/trailing whitespace trimmed
Consistent spacing around operators

Configuration Options

NormalizedQueryMaxLen

Controls the maximum length of normalized queries:

options.NormalizedQueryMaxLen = 300;  // Truncate at 300 characters
options.NormalizedQueryMaxLen = -1;   // No length limit
options.NormalizedQueryMaxLen = 0;    // Disable query tag completely

Truncation Behavior:

Truncation occurs after normalization (preserves more semantic content)
Truncated queries end with ... to indicate truncation
Length measurement excludes the ellipsis characters

AddNormalizedQueryTag

Master switch for query normalization feature:

options.AddNormalizedQueryTag = true;   // Enable normalization and query tags
options.AddNormalizedQueryTag = false;  // Disable completely (better performance)

Cardinality Considerations

Query normalization significantly reduces metric cardinality:

Without Normalization:

Each unique GUID creates a separate metric series
Timestamps create high cardinality
User IDs and other identifiers multiply metric series

With Normalization:

Similar query patterns are grouped together
Metric cardinality is based on query structure, not data values
Business query patterns become visible in metrics

Best Practices:

Enable normalization for query pattern analysis
Disable if only interested in aggregate query costs
Use IgnoreQueryCallers in combination to focus on business operations
Monitor metric cardinality in your observability platform

QueryCostMetricRecorder Class

Table of Contents

📋 Overview

Metric Structure

Key Features

🔍 Additional Details

How Query Cost Detection Works

Query Normalization

Caller Chain Analysis

Purpose of IgnoreQueryCallers

Pattern Matching Features

Tag Enrichment

⚙️ Configuration

Configuration in appsettings.json

Configuration into the startup sequence

OpenTelemetry Integration

Custom Registration

🔧 Troubleshooting

Common Issues

Debugging

Performance Considerations

Custom Filtering and Enrichment

📚 Reference

Classes and Interfaces

Extension Methods

Metric Tags

Configuration Properties

IgnoreQueryCallers Pattern Examples

Common Repository Patterns

Framework and Infrastructure Patterns

Business-Focused Configuration

Pattern Matching Rules

📖 Appendices

Appendix A: Query Normalization Logic

Normalization Overview

Normalization Patterns

Basic Value Replacements

CosmosDB-Specific Patterns

WHERE Clause Normalization

JSON Query Extraction

Error Handling and Fallbacks

Query Prefix Extraction

Failure Scenarios Handled

Performance Optimizations

Compiled Regex Patterns

Processing Order

Whitespace Normalization

Configuration Options

NormalizedQueryMaxLen

AddNormalizedQueryTag

Cardinality Considerations