sqlinsight-pro/tests/SQLInsight.Servers.Tests/LocalCimTests.cs
Søren Eggert Lundsteen Olsen 6d8a5d9ad7 Review 1.5: a Windows server that hangs no longer holds up the agent; reads are bounded and backed off
Deadlines
- Event log over RPC (MS-EVEN6): the whole read has a deadline (OperationTimeout per log);
  the token cancels the EventLogSession's operations and is checked before each
  FormatDescription, and the agent stops waiting even if the native call ignores it.
  Reported as ServerAccessProblem.Timeout ("svarede ikke i tide over hændelsesloggens
  protokol (RPC) inden for N sekunder").
- Every CIM call runs on the thread pool and is awaited with the caller's token; a call
  that ignores cancellation is abandoned and the CIM session is closed once it returns.
- Each server cycle has a deadline (new SQLInsight:Servers:CycleTimeout, default 5 min,
  1-30 min, at least 2 x OperationTimeout): the areas not read by then are "ikke målbar"
  ("Serveren svarede ikke inden for N minutter, så indsamlingen blev afbrudt; …"),
  logged once, and the slot is freed.
- Scheduled tasks: run information is read within a time budget (CycleTimeout / 3); the
  rest keep their state without the last run, and the coverage note says so.
- "Test server" has an overall deadline (WindowsServerTester, default 2 min); areas not
  done in time are reported as not answered, so the result is always stored.

Backoff and memory
- A server that does not answer (Unreachable/Timeout, or a timed-out cycle) is backed
  off 1, 2, 5, 10, then 15 minutes, reset when it answers.
- Protocol Auto: the protocol that worked (e.g. DCOM after WS-Man was refused, also for
  access denied) is tried first for an hour, then WS-Man is tried again; a server whose
  event log RPC failed is read over CIM for an hour.
- Over WS-Man with read-only WinRM rights Task Scheduler's GetInfoByName is denied: with
  Auto the tasks are read over DCOM instead (remembered for the server); with WSMan forced
  the reason says to choose Auto or DCOM. The advice "winrm configSDDL default" (broader
  WinRM rights) is gone; texts follow the grant script (-WinRM read right,
  -EnableFirewall -AgentAddress), not Remote Management Users.

Events
- Each log continues from its own mark (ServerEventSnapshot.LogReadUntil, a small additive
  field in Core): a truncated System log no longer loses the events between its last
  read event and the newest Application event.
- CIM fallback: Win32_NTLogEvent is streamed and only the oldest max per log are kept
  (memory bounded, deterministic order), consistent with the RPC read and the coverage
  text, which now names the logs that had more.

Other
- CimInstances (and method results) are mapped to SQLInsight's types and disposed at once.
- Options are clamped when bound (ServerOptions/SyntheticCheckOptions.Normalize):
  OperationTimeout 5 s-5 min, MaxEventsPerCollection 10-10000, MaxWatched 1-200,
  MaxParallelServers/Checks 1-32, retention days >= 1, ForecastHistoryDays 1-365,
  FailuresBeforeFinding/SlowRunsBeforeFinding 1-99.
- Shutdown: the timer's cancellation is caught, running cycles are awaited before the
  throttle is disposed; a cycle's time is taken when it gets its slot; a TaskCanceledException
  that is not shutdown no longer ends the service.
- Retention maintenance also runs (hourly) when SQLInsight:Servers:Enabled is false but
  synthetic checks are on.
- LocalCimTests run alone (collection without parallelization), share one warmed-up
  session, use the same 2 min operation timeout and a 5 min deadline per test, a smaller
  workload, and report timings in assertion messages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 16:02:17 +02:00

178 lines
7.8 KiB
C#

using System.Diagnostics;
using System.Runtime.Versioning;
using SQLInsight.Core.Servers;
using SQLInsight.Servers;
namespace SQLInsight.Servers.Tests;
/// <summary>The local CIM tests run alone: WMI on a busy test machine (SQL Server containers, every test assembly at once) is slow enough already.</summary>
[CollectionDefinition(Name, DisableParallelization = true)]
public sealed class LocalCimGroup
{
public const string Name = "Local CIM";
}
/// <summary>
/// One local session for the class, with WMI warmed up once: the first query of a class can take long on a cold or busy
/// machine (the provider loads), which is not what the tests are about.
/// </summary>
[SupportedOSPlatform("windows")]
public sealed class LocalCimFixture : IAsyncLifetime
{
/// <summary>Generous: the whole test suite (with SQL Server containers) may run at the same time.</summary>
public static readonly ServerConnectionOptions Options = new() { OperationTimeout = TimeSpan.FromMinutes(2) };
public CimWindowsServerSource? Source { get; private set; }
public IWindowsServerSession? Session { get; private set; }
/// <summary>How long the warm-up took, for the assertion messages.</summary>
public string WarmUp { get; private set; } = "no warm-up";
public async Task InitializeAsync()
{
if (!OperatingSystem.IsWindows())
{
return;
}
using var stop = new CancellationTokenSource(TimeSpan.FromMinutes(5));
var watch = Stopwatch.StartNew();
var source = new CimWindowsServerSource(Options);
var session = await Once(() => source.ConnectAsync(Environment.MachineName, stop.Token));
Source = source;
Session = session;
// One read of every class the tests use; a timeout is retried once (a cold provider), anything else fails the tests.
await Once(() => session.GetOperatingSystemAsync(stop.Token));
await Once(() => session.GetPerformanceAsync(DateTimeOffset.UtcNow, stop.Token));
await Once(() => session.GetDisksAsync(stop.Token));
await Once(() => session.GetServiceAsync("EventLog", stop.Token));
await Once(() => session.GetEventsAsync(DateTimeOffset.UtcNow.AddMinutes(-5), 1, stop.Token));
await Once(() => session.GetTasksAsync(_ => false, 1, stop.Token));
WarmUp = $"warm-up {watch.Elapsed.TotalSeconds:0.0} s";
static async Task<T> Once<T>(Func<Task<T>> read)
{
try
{
return await read();
}
catch (ServerAccessException ex) when (ex.Problem == ServerAccessProblem.Timeout)
{
return await read();
}
}
}
public Task DisposeAsync()
{
Session?.Dispose();
return Task.CompletedTask;
}
}
/// <summary>
/// The CIM implementation against this computer (a local CIM session, as for the agent's own server). Remote access over
/// WS-Man/DCOM with least privilege cannot be tested here; it needs a domain with a second server.
/// </summary>
[Trait("Category", "Windows")]
[Collection(LocalCimGroup.Name)]
[SupportedOSPlatform("windows")]
public sealed class LocalCimTests(LocalCimFixture cim) : IClassFixture<LocalCimFixture>
{
/// <summary>Every call in a test shares this: a hang is a failure within minutes, never a stuck test run.</summary>
private static readonly TimeSpan TestDeadline = TimeSpan.FromMinutes(5);
private readonly List<string> _timings = [];
private string Timings => cim.WarmUp + "; " + string.Join(", ", _timings);
private async Task<T> Timed<T>(string what, Func<Task<T>> read)
{
var watch = Stopwatch.StartNew();
try
{
return await read();
}
finally
{
_timings.Add($"{what} {watch.Elapsed.TotalSeconds:0.0} s");
}
}
[Fact]
public async Task Reads_os_performance_disks_services_events_and_tasks()
{
if (cim.Session is not { } session)
{
return;
}
using var stop = new CancellationTokenSource(TestDeadline);
var os = await Timed("os", () => session.GetOperatingSystemAsync(stop.Token));
var first = await Timed("performance", () => session.GetPerformanceAsync(DateTimeOffset.UtcNow, stop.Token));
await Task.Delay(1500, stop.Token);
var second = await Timed("performance", () => session.GetPerformanceAsync(DateTimeOffset.UtcNow, stop.Token));
var sample = ServerPerformanceMath.Sample(first, second, os);
var disks = await Timed("disks", () => session.GetDisksAsync(stop.Token));
var services = await Timed("services", () => session.GetServicesAsync(stop.Token));
var service = await Timed("service", () => session.GetServiceAsync(services[0].Name, stop.Token));
var missing = await Timed("missing service", () => session.GetServiceAsync("SQLInsight-findes-ikke", stop.Token));
var events = await Timed("events", () => session.GetEventsAsync(DateTimeOffset.UtcNow.AddHours(-1), 50, stop.Token));
var tasks = await Timed("tasks", () => session.GetTasksAsync(name => !TaskResults.IsMicrosoftTask(name), 1, stop.Token));
Assert.True(session.Protocol == "lokal", Timings);
Assert.Contains("Windows", os.Caption, StringComparison.OrdinalIgnoreCase);
Assert.True(os.LastBootUpTime < DateTimeOffset.UtcNow, Timings);
Assert.True(os.TotalMemoryMb > 0, Timings);
Assert.True(os.LogicalProcessors > 0, Timings);
Assert.InRange(sample.CpuPercent!.Value, 0, 100);
Assert.True(sample.AvailableMb > 0, Timings);
Assert.InRange(sample.CommitPercent!.Value, 0, 100);
Assert.True(disks.Count > 0, Timings);
Assert.All(disks, d => Assert.False(d.Volume.StartsWith(@"\\?\", StringComparison.Ordinal)));
Assert.All(disks, d => Assert.InRange(d.FreeGb, 0, d.SizeGb));
Assert.True(services.Any(s => s.IsRunning && s.IsAutomatic), Timings);
Assert.True(services[0].Name == service?.Name, Timings);
Assert.True(missing is null, Timings);
Assert.True(events.Method == "Event Log (RPC)", $"{events.Method} {events.Note}; {Timings}");
Assert.True(events.Entries.Count <= 2 * 50, Timings);
Assert.All(events.Entries, e => Assert.InRange(e.Level, 1, 2));
Assert.True(tasks.Visible > 0, Timings);
Assert.True(tasks.Watched.Count <= 1, Timings);
Assert.All(tasks.Watched, t => Assert.False(TaskResults.IsMicrosoftTask(t.FullName)));
}
[Fact]
public async Task The_test_reports_every_area()
{
if (cim.Source is not { } source)
{
return;
}
using var stop = new CancellationTokenSource(TestDeadline);
var result = await Timed("test", () => WindowsServerTester.TestAsync(source, "localhost", "test", DateTimeOffset.UtcNow, TestDeadline, stop.Token));
Assert.True(result.Reachable, $"{result.Error}; {Timings}");
Assert.Equal(ServerArea.All, result.Areas.Select(a => a.Area));
Assert.All(result.Areas, a => Assert.True(a.Ok, $"{a.Area}: {a.Detail}; {Timings}"));
}
[Fact]
public async Task An_unknown_host_is_unreachable_with_a_reason()
{
if (!OperatingSystem.IsWindows())
{
return;
}
using var stop = new CancellationTokenSource(TestDeadline);
var source = new CimWindowsServerSource(new ServerConnectionOptions { OperationTimeout = TimeSpan.FromSeconds(5) });
var error = await Assert.ThrowsAsync<ServerAccessException>(() => Timed("connect", () => source.ConnectAsync("sqlinsight-findes-ikke.invalid", stop.Token)));
Assert.True(error.Problem is ServerAccessProblem.Unreachable or ServerAccessProblem.Timeout, $"{error.Message}; {Timings}");
Assert.Contains("sqlinsight-findes-ikke.invalid", error.Message, StringComparison.Ordinal);
}
}