OpenTelemetry integration with Dynatrace

Sonic Boom — Tue, 07 Jan 2025 01:52:14 +0000

Credit Decision Application Flow (Highest Priority)

Track complete decision flow from application submission to final decision
Measure processing time at each decision checkpoint
Monitor credit bureau API response times
Track decision rules execution time
Key metrics to collect:
Time from submission to final decision
Processing time per decision rule
Number of applications in each state
Failure rates at each checkpoint

Cross-Service Transaction Monitoring

Since your services are synchronous, track the entire request chain
Monitor service dependencies and their health
Identify service communication bottlenecks
Key areas to instrument:
Inter-service REST calls
Service response times
Error rates between services
Request payload sizes

MongoDB Performance Optimization

Track query patterns across all services
Monitor document access patterns
Key metrics to collect:
Query execution times
Frequently accessed collections
Slow queries (> 100ms)
Index usage statistics
Connection pool utilization

Span: CreditApplication
|- Attributes:
   |- application_id
   |- customer_id
   |- current_stage
   |- decision_outcome
|- Child Spans:
   |- DocumentValidation
   |- CreditCheck
   |- DecisionEngine
   |- CardIssuance

<properties>
    <opentelemetry.version>1.32.0</opentelemetry.version>
</properties>

<dependencies>
    <!-- OpenTelemetry API -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>${opentelemetry.version}</version>
    </dependency>

    <!-- OpenTelemetry SDK -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
        <version>${opentelemetry.version}</version>
    </dependency>

    <!-- OpenTelemetry OTLP Exporter -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
        <version>${opentelemetry.version}</dependency>

    <!-- Spring Boot Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
</dependencies>

@Configuration
public class OpenTelemetryConfig {

    @Value("${dynatrace.endpoint}")
    private String dynatraceEndpoint;

    @Bean
    public OpenTelemetry openTelemetry() {
        Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(
                ResourceAttributes.SERVICE_NAME, "credit-check-service",
                ResourceAttributes.SERVICE_VERSION, "1.0"
            )));

        SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
            .setResource(resource)
            .addSpanProcessor(BatchSpanProcessor.builder(
                OtlpGrpcSpanExporter.builder()
                    .setEndpoint(dynatraceEndpoint)
                    .build())
                .build())
            .build();

        return OpenTelemetrySdk.builder()
            .setTracerProvider(sdkTracerProvider)
            .buildAndRegisterGlobal();
    }
}

@Service
@Slf4j
public class CreditCheckService {

    private final Tracer tracer;
    private final CreditBureauClient creditBureauClient;

    public CreditCheckService(OpenTelemetry openTelemetry, CreditBureauClient creditBureauClient) {
        this.tracer = openTelemetry.getTracer("credit-check-service");
        this.creditBureauClient = creditBureauClient;
    }

    public CreditCheckResult performCreditCheck(String applicationId, String customerId) {
        Span span = tracer.spanBuilder("credit.check")
            .setAttribute("application.id", applicationId)
            .setAttribute("customer.id", customerId)
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            long startTime = System.currentTimeMillis();

            // Create child span for bureau call
            Span bureauCallSpan = tracer.spanBuilder("credit.bureau.call")
                .setAttribute("bureau.name", "ExampleBureau")
                .startSpan();

            CreditBureauResponse bureauResponse;
            try {
                bureauResponse = creditBureauClient.getCreditScore(customerId);
                bureauCallSpan.setAttribute("credit.score", bureauResponse.getCreditScore());
                bureauCallSpan.setStatus(StatusCode.OK);
            } catch (Exception e) {
                bureauCallSpan.setStatus(StatusCode.ERROR);
                bureauCallSpan.recordException(e);
                throw e;
            } finally {
                bureauCallSpan.end();
            }

            // Add important metrics to main span
            span.setAttribute("credit.score", bureauResponse.getCreditScore());
            span.setAttribute("processing.time_ms", System.currentTimeMillis() - startTime);
            span.setAttribute("check.status", "COMPLETED");

            return new CreditCheckResult(bureauResponse);
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

dynatrace.endpoint=https://your-environment-id.live.dynatrace.com/api/v2/otlp/v1/traces

Parent Span (Application Process):

Basic Info: application.id, customer.id, product.type
Application Details: application.type, channel, priority
Customer Info: customer.segment
Timing: timestamp.received, process.start.time, process.duration_ms
Outcome: application.status, decision.outcome
Error Tracking: error.occurred, error.reason, error.code

Child Span (Credit Check):

Basic Info: application.id, customer.id, check.type
Bureau Info: bureau.name, bureau.request.id
Timing: check.start.time, check.duration_ms
Results: credit.score, credit.score.band
Decision: decision.outcome, decision.factors.count
Performance: check.duration_ms
Additional Info: additional.checks.required

Nested Child Span (Bureau Call):

Request Info: bureau.request.id, bureau.request.type
Response: credit.score, bureau.response.status
Performance: bureau.response.time_ms
Error Handling: error.type, error.message

These attributes provide:

Complete traceability of the application process
Performance metrics at each stage
Business metrics (credit scores, decisions)
Error tracking and debugging information
Operational metrics for monitoring and alerting

DEV Community: Sonic Boom

OpenTelemetry integration with Dynatrace

Credit Decision Application Flow (Highest Priority)

Cross-Service Transaction Monitoring

MongoDB Performance Optimization