Deadlines
- Event log over RPC (MS-EVEN6): the whole read has a deadline (OperationTimeout per log);
the token cancels the EventLogSession's operations and is checked before each
FormatDescription, and the agent stops waiting even if the native call ignores it.
Reported as ServerAccessProblem.Timeout ("svarede ikke i tide over hændelsesloggens
protokol (RPC) inden for N sekunder").
- Every CIM call runs on the thread pool and is awaited with the caller's token; a call
that ignores cancellation is abandoned and the CIM session is closed once it returns.
- Each server cycle has a deadline (new SQLInsight:Servers:CycleTimeout, default 5 min,
1-30 min, at least 2 x OperationTimeout): the areas not read by then are "ikke målbar"
("Serveren svarede ikke inden for N minutter, så indsamlingen blev afbrudt; …"),
logged once, and the slot is freed.
- Scheduled tasks: run information is read within a time budget (CycleTimeout / 3); the
rest keep their state without the last run, and the coverage note says so.
- "Test server" has an overall deadline (WindowsServerTester, default 2 min); areas not
done in time are reported as not answered, so the result is always stored.
Backoff and memory
- A server that does not answer (Unreachable/Timeout, or a timed-out cycle) is backed
off 1, 2, 5, 10, then 15 minutes, reset when it answers.
- Protocol Auto: the protocol that worked (e.g. DCOM after WS-Man was refused, also for
access denied) is tried first for an hour, then WS-Man is tried again; a server whose
event log RPC failed is read over CIM for an hour.
- Over WS-Man with read-only WinRM rights Task Scheduler's GetInfoByName is denied: with
Auto the tasks are read over DCOM instead (remembered for the server); with WSMan forced
the reason says to choose Auto or DCOM. The advice "winrm configSDDL default" (broader
WinRM rights) is gone; texts follow the grant script (-WinRM read right,
-EnableFirewall -AgentAddress), not Remote Management Users.
Events
- Each log continues from its own mark (ServerEventSnapshot.LogReadUntil, a small additive
field in Core): a truncated System log no longer loses the events between its last
read event and the newest Application event.
- CIM fallback: Win32_NTLogEvent is streamed and only the oldest max per log are kept
(memory bounded, deterministic order), consistent with the RPC read and the coverage
text, which now names the logs that had more.
Other
- CimInstances (and method results) are mapped to SQLInsight's types and disposed at once.
- Options are clamped when bound (ServerOptions/SyntheticCheckOptions.Normalize):
OperationTimeout 5 s-5 min, MaxEventsPerCollection 10-10000, MaxWatched 1-200,
MaxParallelServers/Checks 1-32, retention days >= 1, ForecastHistoryDays 1-365,
FailuresBeforeFinding/SlowRunsBeforeFinding 1-99.
- Shutdown: the timer's cancellation is caught, running cycles are awaited before the
throttle is disposed; a cycle's time is taken when it gets its slot; a TaskCanceledException
that is not shutdown no longer ends the service.
- Retention maintenance also runs (hourly) when SQLInsight:Servers:Enabled is false but
synthetic checks are on.
- LocalCimTests run alone (collection without parallelization), share one warmed-up
session, use the same 2 min operation timeout and a 5 min deadline per test, a smaller
workload, and report timings in assertion messages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
139 lines
7.1 KiB
C#
139 lines
7.1 KiB
C#
using SQLInsight.Core.Servers;
|
|
using SQLInsight.Core.SyntheticChecks;
|
|
using SQLInsight.Notifications;
|
|
using SQLInsight.Repository;
|
|
using SQLInsight.Rules.Servers;
|
|
using SQLInsight.Servers;
|
|
|
|
namespace SQLInsight.Agent.Servers;
|
|
|
|
/// <summary>Windows servers (SQLInsight:Servers): how and how often to collect, how long to keep it, and the rules' thresholds.</summary>
|
|
public sealed class ServerOptions : ServerRuleSettings
|
|
{
|
|
public const string SectionName = "SQLInsight:Servers";
|
|
|
|
public static readonly TimeSpan MinCycleTimeout = TimeSpan.FromMinutes(1);
|
|
public static readonly TimeSpan MaxCycleTimeout = TimeSpan.FromMinutes(30);
|
|
|
|
/// <summary>Look at Windows servers at all.</summary>
|
|
public bool Enabled { get; set; } = true;
|
|
|
|
/// <summary>The computers behind the monitored SQL Server instances are Windows servers too, without being added.</summary>
|
|
public bool IncludeSqlHosts { get; set; } = true;
|
|
|
|
/// <summary>How many servers are read at the same time (1 to 32).</summary>
|
|
public int MaxParallelServers { get; set; } = 4;
|
|
|
|
/// <summary>Auto (WS-Man, then DCOM), WSMan or DCOM.</summary>
|
|
public string Protocol { get; set; } = "Auto";
|
|
|
|
/// <summary>How long one CIM operation may take (5 seconds to 5 minutes).</summary>
|
|
public TimeSpan OperationTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
|
|
|
/// <summary>
|
|
/// How long one server's round may take at most (1 to 30 minutes, at least twice <see cref="OperationTimeout"/>): what is
|
|
/// not read by then is "ikke målbar" until the next round, so a server that hangs never holds up the others.
|
|
/// </summary>
|
|
public TimeSpan CycleTimeout { get; set; } = TimeSpan.FromMinutes(5);
|
|
|
|
/// <summary>Read event logs with the Event Log service's own protocol (filters on the server); off: CIM only.</summary>
|
|
public bool EventLogRpc { get; set; } = true;
|
|
|
|
/// <summary>Overrides of the collection intervals by area: performance, disks, services, events, tasks.</summary>
|
|
public IDictionary<string, TimeSpan> Intervals { get; } = new Dictionary<string, TimeSpan>(StringComparer.OrdinalIgnoreCase);
|
|
|
|
/// <summary>Per-minute CPU and memory samples (at least 1 day).</summary>
|
|
public int SampleDays { get; set; } = 14;
|
|
|
|
/// <summary>Free space per volume: at least the forecast's history.</summary>
|
|
public int DiskHistoryDays { get; set; } = 35;
|
|
|
|
/// <summary>Grouped Windows events (at least 1 day).</summary>
|
|
public int EventDays { get; set; } = 90;
|
|
|
|
/// <summary>The share of <see cref="CycleTimeout"/> the scheduled tasks' run information may take; the rest is read next time.</summary>
|
|
public TimeSpan TaskBudget => CycleTimeout / 3;
|
|
|
|
public TimeSpan IntervalOf(string area) =>
|
|
Intervals.TryGetValue(area, out var interval) && interval >= TimeSpan.FromSeconds(30) ? interval : ServerArea.DefaultInterval(area);
|
|
|
|
public ServerConnectionOptions Connection => new() { Protocol = Protocol, OperationTimeout = OperationTimeout, EventLogRpc = EventLogRpc };
|
|
|
|
/// <summary>
|
|
/// Keeps the settings within what works: a timeout of 0 or hours, retention of 0 days (maintenance would delete
|
|
/// everything) or unbounded parallelism are moved to the nearest sensible value. Applied when the options are bound.
|
|
/// </summary>
|
|
public ServerOptions Normalize()
|
|
{
|
|
MaxParallelServers = Math.Clamp(MaxParallelServers, 1, 32);
|
|
OperationTimeout = ServerConnectionOptions.BoundTimeout(OperationTimeout);
|
|
var cycle = CycleTimeout < OperationTimeout * 2 ? OperationTimeout * 2 : CycleTimeout;
|
|
CycleTimeout = cycle < MinCycleTimeout ? MinCycleTimeout : cycle > MaxCycleTimeout ? MaxCycleTimeout : cycle;
|
|
SampleDays = Math.Max(1, SampleDays);
|
|
DiskHistoryDays = Math.Max(1, DiskHistoryDays);
|
|
EventDays = Math.Max(1, EventDays);
|
|
Disks.ForecastHistoryDays = Math.Clamp(Disks.ForecastHistoryDays, 1, 365);
|
|
EventLog.MaxEventsPerCollection = Math.Clamp(EventLog.MaxEventsPerCollection, 10, 10000);
|
|
Tasks.MaxWatched = Math.Clamp(Tasks.MaxWatched, 1, 200);
|
|
return this;
|
|
}
|
|
}
|
|
|
|
/// <summary>Synthetic checks (SQLInsight:SyntheticChecks): how many run at once, how long results are kept, and the rules' thresholds.</summary>
|
|
public sealed class SyntheticCheckOptions : SyntheticCheckRuleSettings
|
|
{
|
|
public const string SectionName = "SQLInsight:SyntheticChecks";
|
|
|
|
/// <summary>The most recent results the rules look at: enough for "fejlet i træk", never unbounded.</summary>
|
|
public const int MaxRecentResults = 100;
|
|
|
|
public bool Enabled { get; set; } = true;
|
|
|
|
/// <summary>How many checks run at the same time (1 to 32).</summary>
|
|
public int MaxParallelChecks { get; set; } = 4;
|
|
|
|
/// <summary>Every run's result (at least 1 day).</summary>
|
|
public int ResultDays { get; set; } = 14;
|
|
|
|
/// <summary>Where the Dynamics AX .NET Business Connector is, when not where the AX installer puts it.</summary>
|
|
public string? BusinessConnectorPath { get; set; }
|
|
|
|
/// <summary>How many recent results the rules are given: more than they need to count failures or slow runs in a row.</summary>
|
|
public int RecentResults => Math.Clamp(Math.Max(10, Math.Max(FailuresBeforeFinding, SlowRunsBeforeFinding)) + 1, 11, MaxRecentResults);
|
|
|
|
/// <summary>Keeps the settings within what works (see <see cref="ServerOptions.Normalize"/>). Applied when the options are bound.</summary>
|
|
public SyntheticCheckOptions Normalize()
|
|
{
|
|
MaxParallelChecks = Math.Clamp(MaxParallelChecks, 1, 32);
|
|
ResultDays = Math.Max(1, ResultDays);
|
|
FailuresBeforeFinding = Math.Clamp(FailuresBeforeFinding, 1, MaxRecentResults - 1);
|
|
SlowRunsBeforeFinding = Math.Clamp(SlowRunsBeforeFinding, 1, MaxRecentResults - 1);
|
|
return this;
|
|
}
|
|
|
|
/// <summary>A check's interval as the agent runs it: never more often than <see cref="SyntheticCheckLimits.MinIntervalSeconds"/>.</summary>
|
|
public static TimeSpan IntervalOf(int intervalSeconds) =>
|
|
TimeSpan.FromSeconds(Math.Clamp(intervalSeconds, SyntheticCheckLimits.MinIntervalSeconds, SyntheticCheckLimits.MaxIntervalSeconds));
|
|
}
|
|
|
|
/// <summary>Notifications about Windows servers and synthetic checks, with links to their pages in the web UI.</summary>
|
|
internal static class ServerNotifications
|
|
{
|
|
public static IEnumerable<Notification> For(string name, FindingChanges changes, Uri? webUrl, DateTimeOffset now)
|
|
{
|
|
foreach (var (id, finding) in changes.Opened)
|
|
{
|
|
yield return NotificationFactory.Opened(name, id, finding, webUrl, now) with { Url = Link(webUrl, id) };
|
|
}
|
|
|
|
foreach (var resolved in changes.Resolved)
|
|
{
|
|
yield return NotificationFactory.Resolved(name, resolved.FindingId, resolved.RuleId, resolved.Title, (Core.Findings.Severity)resolved.Severity, webUrl, now)
|
|
with { Url = Link(webUrl, resolved.FindingId) };
|
|
}
|
|
}
|
|
|
|
public static Uri? Link(Uri? webUrl, long findingId) => webUrl is null ? null : new Uri(webUrl, $"servere/fund/{findingId}");
|
|
|
|
public static Uri? ServerPage(Uri? webUrl, int serverId) => webUrl is null ? null : new Uri(webUrl, $"servere/{serverId}");
|
|
}
|