Refactor benchmarks by fdcastel · Pull Request #1203 · FirebirdSQL/NETProvider

fdcastel · 2024-10-26T20:36:11Z

Based on #1200 for now. I will rebase it with latest master in the future.

Rename project Perf to FirebirdSql.Data.FirebirdClient.Benchmarks.
Update project to use .NET8.
Upgrade BenchmarkDotNet to version 0.14.0.
Update baseline nuget package to v10.3.1.
Add /BenchmarkDotNet.Artifacts folder to .gitignore.
Pass command-line arguments to BenchmarkDotNet engine.
Apply SQL Formatting. Use raw strings.
Add script run-benchmark.ps1.

src/FirebirdSql.Data.FirebirdClient.Benchmarks/CommandBenchmark.cs

fdcastel · 2025-05-18T02:20:27Z

Rebase with master.

StartActivity() returns null when no listeners are interested in the activity. The code accessed activity.IsAllDataRequested without a null check, causing a NullReferenceException. Use the Npgsql-style pattern 'activity is not { IsAllDataRequested: true }' for early return.

Stopwatch ticks are not TimeSpan ticks when Stopwatch.IsHighResolution is true (most systems). TimeSpan.FromTicks(stopwatchTicks) produces incorrect durations. Use Stopwatch.GetElapsedTime() which correctly handles the frequency conversion.

The semantic conventions renamed db.system to db.system.name. The well-known value for Firebird is 'firebirdsql', not 'firebird'. Updated in both FbActivitySource (spans) and FbConnection (metrics).

The semantic conventions mark db.namespace as Conditionally Required when available. The connection's Database property is used as the value.

The semantic conventions mark error.type as Conditionally Required on failure. For FbException, use the SQLSTATE code; for other exceptions, use the full exception type name.

Per OTel semantic conventions, db.query.text and db.query.parameter.* are Opt-In level attributes that may expose sensitive data. They should not be collected by default. Added IsQueryTextTracingEnabled flag and reuse IsParameterLoggingEnabled for parameter tracing.

Replace hardcoded '1.0.0' version with the assembly's informational version, keeping the telemetry version in sync with the NuGet package.

…ecording TraceCommandException now records metrics immediately and resets _startedAtTicks so the later TraceCommandStop (called during Release) does not record a second, inflated duration that includes cleanup time.

- Remove db.transaction_id custom tag (not in OTel spec, causes server round-trip per traced command) - Remove commented-out db.snapshot_id code - Remove deprecated exception.escaped attribute - Remove NormalizeDbNull helper, inline DBNull check

- Set server.port on spans when using a non-default port - Set db.stored_procedure.name for StoredProcedure command types - Fix server.address in MetricsConnectionAttributes to not concatenate the port (server.port is now a separate attribute) - Clean up leftover placeholder comments

Set db.query.summary as the low-cardinality activity name on all spans. For Text commands, extract the first SQL verb (SELECT, INSERT, etc.) and use it as db.operation.name and in the activity name.

Debug.Assert checks that TraceCommandStart was called before the stop or exception path, helping catch lifecycle bugs during development.

Prevents potential NullReferenceException if metrics are recorded before the connection string is set.

Change GetMetrics() to return IEnumerable of tuples via Select instead of allocating a Dictionary via ToDictionary on every collection cycle. Also simplify GetConnectionMax to use Select instead of SelectMany.

Suggest default histogram bucket boundaries for operation duration and connection create time histograms, improving out-of-the-box resolution. Guarded by NET9_0_OR_GREATER since InstrumentAdvice is a .NET 9 API.

Expose the telemetry source names as public constants so consumers can easily subscribe: builder.AddSource(FbTelemetry.ActivitySourceName) and builder.AddMeter(FbTelemetry.MeterName). Internal code updated to use the constants.

Document the ActivitySource and Meter names, available span attributes, opt-in sensitive attributes, and available metrics with their types and semantic convention references.

- Rename project 'Perf' to 'FirebirdSql.Data.FirebirdClient.Benchmarks'. - Update project to use .net8. - Upgrade BenchmarkDotNet to version 0.14.0. - Update baseline nuget package to v10.3.1. - Add /BenchmarkDotNet.Artifacts to .gitignore. - Pass command-line arguments to BenchmarkDotNet engine. - Apply SQL Formatting. Use raw strings. - Add script run-benchmark.ps1.

fdcastel · 2026-03-23T03:37:55Z

Following up on the initial refactoring, these additional changes were made on the branch:

Infrastructure

Extracted a shared BenchmarkConfig : ManualConfig class to eliminate the duplicated inner Config class that existed independently in each benchmark class. Job configuration is now defined in a single place.
Extracted a BenchmarkBase abstract class holding the ConnectionString, CreateDatabase(), and GlobalCleanup() members that were duplicated across benchmark classes.
The connection string can now be overridden via the FIREBIRD_BENCHMARK_CS environment variable (falls back to the existing localhost default), enabling use in environments with a non-default Firebird setup.

Configuration

Updated the benchmark toolchain from .NET 8 to .NET 10, matching the project's TargetFramework.
Removed WithPlatform(Platform.X64) and WithJit(Jit.RyuJit) — both are no-ops on .NET 8+ (x64 + RyuJIT is the only supported configuration) and were just noise.
Added MarkdownExporter.GitHub so each run automatically writes a GitHub-flavored Markdown table to BenchmarkDotNet.Artifacts/, ready to paste into issues or PRs.
Added DefaultOrderer(SummaryOrderPolicy.FastestToSlowest) to make the results table easier to scan.

Benchmark methods

Single-value [Params(100)] and [Params(100_000)] replaced with plain constants — a single-value [Params] adds a column to the results table without varying anything.
Fetch methods now return the last read value instead of discarding it, to make dead-code elimination by the JIT impossible.
Added async variants (ExecuteAsync, FetchAsync, OpenCloseAsync) for all sync benchmarks. Async ADO.NET paths have different allocation profiles (state machines, Task wrappers) and are the primary path in web workloads.

New benchmarks

ConnectionBenchmark — measures connection open/close throughput (sync and async), which exercises connection pool logic.
LargeFetchBenchmark — added to the committed tree (was previously an untracked file).

Script

run-benchmark.ps1 now accepts -Benchmark ConnectionBenchmark in addition to the existing values, and two new optional switches: -Disasm (enables --disasm) and -Profile (enables ETW profiling on Windows).

Documentation

Added docs/benchmark.md describing prerequisites, how to run, connection string configuration, available benchmarks, and where results are written.

fdcastel · 2026-03-23T05:28:08Z

Future improvements:

async benchmarks
connection-pool benchmarks
configurable connection string
Remove Duplicated config (use a shared base)

This was referenced Oct 26, 2024

Adds .NET distributed tracing instrumentation & metrics. #1200

Open

Update benchmarks cincuranet/FirebirdSql.Data.FirebirdClient#129

Closed

willibrandon reviewed Oct 26, 2024

View reviewed changes

src/FirebirdSql.Data.FirebirdClient.Benchmarks/CommandBenchmark.cs Outdated Show resolved Hide resolved

fdcastel force-pushed the refactor-benchmarks branch from bd2e3a0 to e5e4075 Compare May 18, 2025 01:19

fdcastel added 25 commits March 22, 2026 21:43

Adds FbTransactionInfo.GetTransactionId().

256ea2f

Adds .NET distributed tracing instrumentation.

7dd65e8

Add metrics.

7fe1327

Fix Stopwatch elapsed time conversion

9b25148

Stopwatch ticks are not TimeSpan ticks when Stopwatch.IsHighResolution is true (most systems). TimeSpan.FromTicks(stopwatchTicks) produces incorrect durations. Use Stopwatch.GetElapsedTime() which correctly handles the frequency conversion.

Rename db.system to db.system.name per OTel semantic conventions v1.40+

c198a6a

The semantic conventions renamed db.system to db.system.name. The well-known value for Firebird is 'firebirdsql', not 'firebird'. Updated in both FbActivitySource (spans) and FbConnection (metrics).

Set db.namespace attribute on spans from connection database name

4781900

The semantic conventions mark db.namespace as Conditionally Required when available. The connection's Database property is used as the value.

Set error.type attribute on failure spans

4f83e23

The semantic conventions mark error.type as Conditionally Required on failure. For FbException, use the SQLSTATE code; for other exceptions, use the full exception type name.

Read ActivitySource/Meter version from assembly metadata

ecdb4cd

Replace hardcoded '1.0.0' version with the assembly's informational version, keeping the telemetry version in sync with the NuGet package.

Add db.query.summary and db.operation.name for Text commands

98f6f6d

Set db.query.summary as the low-cardinality activity name on all spans. For Text commands, extract the first SQL verb (SELECT, INSERT, etc.) and use it as db.operation.name and in the activity name.

Add debug assertions in TraceCommandStop and TraceCommandException

4aa1e01

Debug.Assert checks that TraceCommandStart was called before the stop or exception path, helping catch lifecycle bugs during development.

Initialize MetricsConnectionAttributes to empty array

43d9462

Prevents potential NullReferenceException if metrics are recorded before the connection string is set.

Avoid dictionary allocation in observable metric callbacks

425ae5a

Change GetMetrics() to return IEnumerable of tuples via Select instead of allocating a Dictionary via ToDictionary on every collection cycle. Also simplify GetConnectionMax to use Select instead of SelectMany.

Add InstrumentAdvice with histogram bucket boundaries on .NET 9+

395a907

Suggest default histogram bucket boundaries for operation duration and connection create time histograms, improving out-of-the-box resolution. Guarded by NET9_0_OR_GREATER since InstrumentAdvice is a .NET 9 API.

Add OpenTelemetry integration documentation

36d884f

Document the ActivitySource and Meter names, available span attributes, opt-in sensitive attributes, and available metrics with their types and semantic convention references.

Fix IndexOfAny compilation on all target frameworks

ec1c68b

Updates run-benchmark.ps1.

0c4a764

Fix solution file: update benchmark project path and folder name

ed46d7a

Add LargeFetchBenchmark for bulk read throughput across data types

04208fe

fdcastel added 10 commits March 22, 2026 23:33

Extract shared BenchmarkConfig to eliminate duplication

8fa39ed

Extract BenchmarkBase with shared ConnectionString and env-var override

5852c9a

Update benchmark toolchain from .NET 8 to .NET 10

3ad15fa

Add GitHub Markdown exporter and fastest-to-slowest ordering

0ecaa3a

Replace single-value [Params] with constants

671604a

Return values from Fetch benchmarks to prevent dead-code elimination

32c46bb

Add async benchmark variants for Execute and Fetch

3c3ce6e

Add connection open/close benchmark

c12627e

Add -Disasm and -Profile flags to run-benchmark.ps1

6f9e2d8

Fix build errors: remove duplicate braces, add missing using

1ec795d

fdcastel force-pushed the refactor-benchmarks branch 2 times, most recently from 2a8789b to 1ec795d Compare March 23, 2026 03:31

Add benchmark documentation

533d57c

fdcastel mentioned this pull request Mar 23, 2026

(Improvements) Rune operations optimization #1252

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor benchmarks#1203

Refactor benchmarks#1203
fdcastel wants to merge 36 commits intoFirebirdSQL:masterfrom
fdcastel:refactor-benchmarks

fdcastel commented Oct 26, 2024

Uh oh!

Uh oh!

fdcastel commented May 18, 2025

Uh oh!

fdcastel commented Mar 23, 2026

Uh oh!

fdcastel commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fdcastel commented Oct 26, 2024

Uh oh!

Uh oh!

fdcastel commented May 18, 2025

Uh oh!

fdcastel commented Mar 23, 2026

Uh oh!

fdcastel commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants