--- title: Reader weight: 2 menu: docs: parent: "vmanomaly-components" weight: 2 tags: - metrics - enterprise aliases: - /anomaly-detection/components/reader.html --- VictoriaMetrics Anomaly Detection (`vmanomaly`) has an input of Prometheus-compatible metrics from either [VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/) accessed with [VmReader](#vm-reader) with [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) queries or from [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) / [VictoriaTraces](https://docs.victoriametrics.com/victoriatraces/) accessed with [VLogsReader](#victorialogs-reader) with [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) queries. Future updates will introduce additional readers, expanding the range of data sources `vmanomaly` can work with. ## Playgrounds To ease the development and testing of queries for `vmanomaly`'s input data, following playgrounds can be used for experimenting with MetricsQL and LogsQL queries: Please see respective sections below for specific reader: - [MetricsQL playground](#metricsql-playground) for `VmReader` - [LogsQL playground](#logsql-playground) for `VLogsReader` ## VM reader {{% collapse name="Queries format migration (to v1.13.0+)" %}} > There is backward-compatible change{{% available_from "v1.13.0" anomaly %}} of [`queries`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) arg of [VmReader](#vm-reader). New format allows to specify per-query parameters, like `step` to reduce amount of data read from VictoriaMetrics TSDB and to allow config flexibility. Please see [per-query parameters](#per-query-parameters) section for the details. Old format like ```yaml # other config sections ... reader: class: 'vm' datasource_url: 'http://localhost:8428' # source victoriametrics/prometheus sampling_period: "10s" # set it <= min(infer_every) in schedulers section queries: # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically vmb: 'avg(vm_blocks)' ``` will be converted to a new one with a warning raised in logs: ```yaml # other config sections ... reader: class: 'vm' datasource_url: 'http://localhost:8428' # source victoriametrics/prometheus sampling_period: '10s' queries: # old format {query_alias: query_expr}, prior to 1.13, will be converted to a new format automatically vmb: expr: 'avg(vm_blocks)' # initial MetricsQL expression step: '10s' # individual step for this query, will be filled with `sampling_period` from the root level data_range: ['-inf', 'inf'] # by default, no constraints applied on data range tz: 'UTC' # by default, tz-free data is used throughout the model lifecycle # new query-level arguments will be added in backward-compatible way in future releases ``` {{% /collapse %}} ### Per-query parameters There is change {{% available_from "v1.13.0" anomaly %}} of [`queries`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) arg format. Now each query alias supports the next (sub)fields, which *override reader-level parameters*, if set: - `expr` (string): MetricsQL/PromQL expression that defines an input for VmReader. As accepted by `/query_range?query=%s`. i.e. `avg(vm_blocks)` - `step` (string): query-level frequency of the points returned, i.e. `30s`. Will be converted to `/query_range?step=%s` param (in seconds). Useful to optimize total amount of data read from VictoriaMetrics, where different queries may have **different frequencies for different [machine learning models](https://docs.victoriametrics.com/anomaly-detection/components/models/)** to run on. > If not set explicitly (or if older config style prior to [v1.13.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130)) is used, then it is set to reader-level `sampling_period` arg. > Having **different** individual `step` args for queries (i.e. `30s` for `q1` and `2m` for `q2`) is not yet supported for [multivariate model](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) if you want to run it on several queries simultaneously (i.e. setting [`queries`](https://docs.victoriametrics.com/anomaly-detection/components/models/#queries) arg of a model to [`q1`, `q2`]). - `data_range`{{% available_from "v1.15.1" anomaly %}} (list[float | string]): It allows defining **valid** data ranges for input per individual query in `queries`, resulting in: - **High anomaly scores** (>1) when the *data falls outside the expected range*, indicating a data range constraint violation (e.g. improperly configured metricsQL query, sensor malfunction, overflows in underlying metrics, etc.). Anomaly scores can be set to a specific value, like `5`, to indicate a strong violation, using the `anomaly_score_outside_data_range` [arg](https://docs.victoriametrics.com/anomaly-detection/components/models/#score-outside-data-range) of a respective model this query is used in. - **Lowest anomaly scores** (=0) when the *model's predictions (`yhat`) fall outside the expected range*, meaning uncertain predictions that does not really aligh with the data. Works together with `anomaly_score_outside_data_range` [arg](https://docs.victoriametrics.com/anomaly-detection/components/models/#score-outside-data-range) of a model to determine the anomaly score for such cases as well as with `clip_predictions` [arg](https://docs.victoriametrics.com/anomaly-detection/components/models/#clip-predictions) of a model to clip the predictions to the expected range. > If not set explicitly (or if older config style prior to [v1.13.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130)) is used, then it is set to reader-level `data_range` arg{{% available_from "v1.18.1" anomaly %}} - `max_points_per_query`{{% available_from "v1.17.0" anomaly %}} (int): Optional arg, overrides how `search.maxPointsPerTimeseries` flag{{% available_from "v1.14.1" anomaly %}} impacts `vmanomaly` on splitting long `fit_window` [queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) into smaller sub-intervals. This helps users avoid hitting the `search.maxQueryDuration` limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead. Set less than `search.maxPointsPerTimeseries` if hitting `maxQueryDuration` limits. If set on a query-level, it overrides the global `max_points_per_query` (reader-level). - `tz`{{% available_from "v1.18.0" anomaly %}} (string): this optional argument enables timezone specification per query, overriding the reader’s default `tz`. This setting helps to account for local timezone shifts, such as [DST](https://en.wikipedia.org/wiki/Daylight_saving_time), in models that are sensitive to seasonal variations (e.g., [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) or [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile)). - `tenant_id` {{% available_from "v1.19.0" anomaly %}} (string): this optional argument enables tenant-level separation for queries (e.g. `query1` to get the data from tenant "0:0", `query2` - from tenant "1:0"). It works as follows: - if *not set, inherits* reader-level `tenant_id` - if *set, overrides* reader-level `tenant_id` - *raises config validation error*, if *reader-level is not set* and *query-level is found* (mixing of VictoriaMetrics [single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and [cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) is prohibited in a single config) - *raises config validation warning*, if `writer.tenant_id` is not explicitly set to `multitenant` when reader uses tenants, meaning [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) will be used for data querying. - also *raises config validation error* if a set of `reader.queries` for [multivariate models](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) has *different* tenant_ids (meaning tenant data is mixed, and special labels like `vm_project_id`, `vm_account_id` will have [ambiguous values](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)) > The recommended approach for using per-query `tenant_id`s is to set both `reader.tenant_id` and `writer.tenant_id` to `multitenant`. See [this section](https://docs.victoriametrics.com/anomaly-detection/components/writer/#multitenancy-support) for more details. Configurations where `reader.tenant_id` equals `writer.tenant_id` and is not `multitenant` are also considered safe, provided there is a single, DISTINCT `tenant_id` defined in the reader (either at the reader level or the query level, if set). - `offset` {{% available_from "v1.25.3" anomaly %}} (string): this optional argument allows specifying a time offset for the query, which can be useful for adjusting the query time range to account for data collection delays or other timing issues. The offset is specified as a string (e.g., "15s", "-20s") and will be applied to the query time range. Valid resolutions are `ms`, `s`, `m`, `h`, `d` (miliseconds, seconds, minutes, hours, days). If not set, defaults to `0s` (0). See [FAQ](https://docs.victoriametrics.com/anomaly-detection/faq/#using-offsets) for more details. ### Per-query config example ```yaml reader: class: 'vm' sampling_period: '1m' datasource_url: 'https://play.victoriametrics.com/' # source victoriametrics/prometheus max_points_per_query: 10000 data_range: [0, 'inf'] tenant_id: 'multitenant' offset: '0s' # optional, defaults to 0s if not set # other reader params ... queries: ingestion_rate_t1: expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0' step: '2m' # overrides global `sampling_period` of 1m data_range: [10, 'inf'] # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1 max_points_per_query: 5000 # overrides reader-level value of 10000 for `ingestion_rate` query tz: 'America/New_York' # to override reader-wise `tz` tenant_id: '1:0' # overriding tenant_id to isolate data ingestion_rate_t2: expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0' step: '2m' # overrides global `sampling_period` of 1m data_range: [10, 'inf'] # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1 max_points_per_query: 5000 # overrides reader-level value of 10000 for `ingestion_rate` query tz: 'America/New_York' # to override reader-wise `tz` tenant_id: '2:0' # overriding tenant_id to isolate data offset: '-15s' # to override reader-wise `offset` and query data 15 seconds earlier to account for data collection delays ``` ### Config parameters
Parameter Example Description
`class` `reader.vm.VmReader` (or `vm`{{% available_from "v1.13.0" anomaly %}}) Name of the class needed to enable reading from VictoriaMetrics or Prometheus. VmReader is the default option, if not specified.
`queries` See [per-query config example](#per-query-config-example) above See [per-query config section](#per-query-parameters) above
`datasource_url` `http://localhost:8481/` Datasource URL address
`tenant_id` `0:0`, `multitenant` For VictoriaMetrics Cluster version only, tenants are identified by `accountID` or `accountID:projectID`. Starting from [v1.16.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1162), `multitenant` [endpoint](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels) is supported, to execute queries over multiple [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). See VictoriaMetrics Cluster [multitenancy docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
`sampling_period` `1h` Frequency of the points returned. Will be converted to `/query_range?step=%s` param (in seconds). **Required** since [v1.9.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v190).
`query_range_path` `/api/v1/query_range` Performs PromQL/MetricsQL range query
`health_path` `health` Absolute or relative URL address where to check availability of the datasource.
`user` `USERNAME` BasicAuth username. If set, it will be used to authenticate the request.
`password` `PASSWORD` BasicAuth password. If set, it will be used to authenticate the request.
`timeout` `30s` Timeout for the requests, passed as a string
`verify_tls` `false` Verify TLS certificate. If `False`, it will not verify the TLS certificate. If `True`, it will verify the certificate using the system's CA store. If a path to a CA bundle file (like `ca.crt`), it will verify the certificate using the provided CA bundle.
`tls_cert_file` `path/to/cert.crt` Path to a file with the client certificate, i.e. `client.crt`{{% available_from "v1.16.3" anomaly %}}.
`tls_key_file` `path/to/key.crt` Path to a file with the client certificate key, i.e. `client.key`{{% available_from "v1.16.3" anomaly %}}.
`bearer_token` `token` Token is passed in the standard format with header: `Authorization: bearer {token}`
`bearer_token_file` `path_to_file` Path to a file, which contains token, that is passed in the standard format with header: `Authorization: bearer {token}`{{% available_from "v1.15.9" anomaly %}}.
`extra_filters` `[]` List of strings with series selector. See: [Prometheus querying API enhancements](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-enhancements)
`query_from_last_seen_timestamp` `False` If True, then query will be performed from the last seen timestamp for a given series. If False, then query will be performed from the start timestamp, based on a schedule period. Defaults to `False`. Useful for `infer` stages in case there were skipped `infer` calls prior to given.
`latency_offset` `1ms` It allows overriding the default `-search.latencyOffset`{{% available_from "v1.15.1" anomaly %}} [flag of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/#list-of-command-line-flags) (30s). The default value is set to 1ms, which should help in cases where `sampling_frequency` is low (10-60s) and `sampling_frequency` equals `infer_every` in the [PeriodicScheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler). This prevents users from receiving `service - WARNING - [Scheduler [scheduler_alias]] No data available for inference.` warnings in logs and allows for consecutive `infer` calls without gaps. To restore the old behavior, set it equal to your `-search.latencyOffset` [flag value](https://docs.victoriametrics.com/victoriametrics/#list-of-command-line-flags).
`max_points_per_query` `10000` Optional arg{{% available_from "v1.17.0" anomaly %}} overrides how `search.maxPointsPerTimeseries` flag{{% available_from "v1.14.1" anomaly %}} impacts `vmanomaly` on splitting long `fit_window` [queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) into smaller sub-intervals. This helps users avoid hitting the `search.maxQueryDuration` limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead. Set less than `search.maxPointsPerTimeseries` if hitting `maxQueryDuration` limits. Can be also set on [per-query](#per-query-parameters) basis to override reader-level settings.
`tz` `UTC` Optional argument{{% available_from "v1.18.0" anomaly %}} specifies the [IANA](https://nodatime.org/TimeZones) timezone to account for local shifts, like [DST](https://en.wikipedia.org/wiki/Daylight_saving_time), in models sensitive to seasonal patterns (e.g., [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) or [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile)). Defaults to `UTC` if not set and can be overridden on a [per-query basis](#per-query-parameters).
`data_range` `["-inf", "inf"]` Optional argument{{% available_from "v1.18.1" anomaly %}} allows defining **valid** data ranges for input of all the queries in `queries`. Defaults to `["-inf", "inf"]` if not set and can be overridden on a [per-query basis](#per-query-parameters).
`offset` `60s` Optional argument{{% available_from "v1.25.3" anomaly %}} allows specifying a time offset for all queries in `queries`. Defaults to `0s` (0) if not set and can be overridden on a [per-query basis](#per-query-parameters).

**Config section example**: ```yaml reader: class: "vm" # or "reader.vm.VmReader" until v1.13.0 datasource_url: "https://play.victoriametrics.com/" tenant_id: '0:0' tz: 'America/New_York' data_range: [1, 'inf'] # reader-level offset: '0s' # reader-level queries: ingestion_rate: expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0' step: '1m' # can override reader-level `sampling_period` on per-query level data_range: [0, 'inf'] # if set, overrides reader-level data_range tz: 'Australia/Sydney' # if set, overrides reader-level tz # tenant_id: '1:0' # if set, overrides reader-level tenant_id # offset: '-15s' # if set, overrides reader-level offset sampling_period: '1m' query_from_last_seen_timestamp: True # false by default latency_offset: '1ms' ``` ### MetricsQL Playground To experiment with MetricsQL queries for `VmReader`, you can use the [VictoriaMetrics MetricsQL Playground](https://play.victoriametrics.com/), which provides an interactive environment to test and visualize your queries against sample data. You can also access embedded version of the playground below: {{% collapse name="VictoriaMetrics Playground" %}}
{{% /collapse %}} ### mTLS protection `vmanomaly` supports [mutual TLS (mTLS)](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} for secure communication across its components, including [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader), [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer), and [Monitoring/Push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters). This allows for mutual authentication between the client and server when querying or writing data to [VictoriaMetrics Enterprise, configured for mTLS](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#mtls-protection). mTLS ensures that both the client and server verify each other's identity using certificates, which enhances security by preventing unauthorized access. To configure mTLS, the following parameters can be set in the [config](#config-parameters): - `verify_tls`: If set to a string, it functions like the `-mtlsCAFile` command-line argument of VictoriaMetrics, specifying the CA bundle to use. Set to `True` to use the system's default certificate store. - `tls_cert_file`: Specifies the path to the client certificate, analogous to the `-tlsCertFile` argument of VictoriaMetrics. - `tls_key_file`: Specifies the path to the client certificate key, similar to the `-tlsKeyFile` argument of VictoriaMetrics. These options allow you to securely interact with mTLS-enabled VictoriaMetrics endpoints. Example configuration to enable mTLS with custom certificates: ```yaml reader: class: "vm" datasource_url: "https://your-victoriametrics-instance-with-mtls" # tenant_id: "0:0" uncomment and set for cluster version queries: vm_blocks_example: expr: 'avg(rate(vm_blocks[5m]))' step: 30s sampling_period: 30s verify_tls: "path/to/ca.crt" # path to CA bundle for TLS verification tls_cert_file: "path/to/client.crt" # path to the client certificate tls_key_file: "path/to/client.key" # path to the client certificate key # additional reader parameters ... # other config sections, like models, schedulers, writer, ... ``` ### Healthcheck metrics `VmReader` exposes [several healthchecks metrics](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#reader-behaviour-metrics). ## VictoriaLogs reader {{% available_from "v1.26.0" anomaly %}} `vmanomaly` can read data from [VictoriaLogs stats queries](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats) endpoint with `VLogsReader`. This reader allows quering and analyzing log data stored in [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/), enabling anomaly detection on metrics generated from logs. **Querying [VictoriaTraces](https://docs.victoriametrics.com/victoriatraces/) is supported with the same reader, as the endpoints for both are equivalent.** Its queries should be expressed in [LogsQL*](https://docs.victoriametrics.com/victorialogs/logsql/) language that both VictoriaLogs and VictoriaTraces support, with the focus on using [stats pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) functions to calculate metrics from logs. > Please be aware that `VLogsReader` is designed to work with a `/select/stats_query_range` endpoint of [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/), so the `` expressions must ends with `stats` [pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) (see [query-examples](#query-examples) section below). The calculated stats is converted into metrics with labels from `by(...)` clause of the `| stats by(...)` pipe, where `stats_func*` is any of the supported [stats function subset](#valid-stats-functions) of [available stats functions](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions), while the `result_name*` is the name of the log field to store the result of the corresponding stats function. The `as` keyword is optional. ### Valid stats functions `VLogsReader` relies on [stats pipe functions](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions) that return **numeric values**, which can be used for anomaly detection on timeseries (metrics). The future addition of similar stats functions in VictoriaLogs will be supported automatically, as long as they return **numeric values**. The supported stats functions currently include: - `avg` - returns the average value over the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `count` - returns the number of log entries. - `count_empty` - returns the number logs with empty [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `count_uniq` - returns the number of unique non-empty values for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `count_uniq_hash` - returns the number of unique hashes for non-empty values at the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `max` - returns the maximum value over the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `median` - returns the [median](https://en.wikipedia.org/wiki/Median) value over the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `min` - returns the minimum value over the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `quantile` - returns the given quantile for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `rate` - returns the average per-second rate of matching logs on the selected time range. - `rate_sum` - returns the average per-second rate of sum for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `sum` - returns the sum for the given numeric [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). - `sum_len` - returns the sum of lengths for the given [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). ### Query Examples #### VictoriaLogs Here are examples of simple [valid LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) queries with stats pipe functions that can be used with `VLogsReader`. 1. Ingestion volume - good baseline time series, for detecting dropouts/spikes without depending on any schema. ```shellhelp * | stats count() as logs ``` 2. Ingestion rate (normalized) - good for detecting dropouts/spikes without depending on any schema, and also for detecting changes in log volume trends. ```shellhelp * | stats rate() as logs_per_sec ``` 3. Per-stream rate - good for detecting dropouts/spikes on individual streams, and also for detecting changes in log volume trends on stream level. ```shellhelp * | stats by (_stream) rate() as logs_per_sec ``` 4. Active stream churn - good for detecting changes in the number of active streams. Catches "new sources exploded"/"sources disappeared" patterns. ```shellhelp * | stats count_uniq(_stream) as active_streams ``` 5. Avg logs per stream - good for detecting changes in log volume trends on stream level, without depending on the number of streams (sources). Catches "new sources exploded"/"sources disappeared" patterns, as well as changes in log volume on stream level. ```shellhelp * | stats count() as logs, count_uniq(_stream_id) as streams | math (logs / max(streams, 1)) as logs_per_stream ``` 6. Max message size - good for detecting changes in log message size patterns, which can be an indicator of changes in log structure or content. ```shellhelp * | len(_msg) as msg_len | stats max(msg_len) as max_msg_len ``` 7. P90 word per message - good for detecting changes in the distribution of words per message, which can indicate changes in log content or structure. ```shellhelp * | unpack_words as words drop_duplicates | json_array_len(words) as words_count | stats quantile(0.9, words_count) as p90_words_per_msg ``` #### VictoriaTraces > [!TIP] Almost identical to VictoriaLogs, but in VictoriaTraces you'll need to exclude index entries. Basically replace `*` with a filter, e.g. `{"resource_attr:service.name"!=""}` to select only spans with non-empty `service.name` resource attribute. 1. Ingestion volume - good baseline time series, for detecting dropouts/spikes without depending on any schema. ``` {"resource_attr:service.name"!=""} | stats count() as spans ``` or for the rate: ```shellhelp {"resource_attr:service.name"!=""} | stats rate() as spans_per_sec ``` 2. Error volume - good for detecting dropouts/spikes in error spans, without depending on any schema. In VictoriaLogs you can use similar query with `status_code` field, if it exists in your logs. ``` # spans with `status_code=2`, see: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto#L323 {"resource_attr:service.name"!=""} AND status_code := "2" | stats count() as error_spans ``` or for the rate: ```shellhelp {"resource_attr:service.name"!=""} AND status_code := "2" | stats rate() as error_spans_per_sec ``` > [!TIP] [The stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) in VictoriaTraces means "service_name & span_name" combination. ```shellhelp {"resource_attr:service.name"!=""} | stats count_uniq(_stream) as active_streams ``` ### LogsQL playground To experiment with LogsQL queries for `VLogsReader`, you can use the [VictoriaLogs LogsQL Playground](https://play-vmlogs.victoriametrics.com/), which provides an interactive environment to test and visualize your queries against sample log data. Similarly, [VictoriaTraces LogsQL Playground](https://play-vtraces.victoriametrics.com/) can be used for testing LogsQL queries against sample trace data. You can also access **embedded version of the playground below** (VictoriaLogs datasource): {{% collapse name="VictoriaLogs LogsQL Playground" %}}
{{% /collapse %}} ### Config parameters
Parameter Example Description
`class` `vlogs` The class name of the reader, must be `vlogs` (or `reader.vlogs.VLogsReader`).
`queries` See [per-query config example](#per-query-config-example-1) below Dictionary of queries. Keys are query aliases, values are LogsQL queries to select data in format: `QUERY_ALIAS:`, as accepted by `/select/logsql/stats_query_range?query=%s` VictoriaLogs/VictoriaTraces endpoint. The `` must contain `stats` [pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe-functions). The calculated stats is converted into metrics with labels from `by(...)` clause of the `| stats by(...)` pipe. Only functions returning numeric values are supported, e.g. `count()`, `sum()`, `avg()`, `count_uniq()`, `median()`, `quantile()`, etc.
`datasource_url` `https://play-vmlogs.victoriametrics.com/` URL address of the VictoriaLogs/VictoriaTraces datasource. Must be a valid URL.
`tenant_id` `0:0` Tenants are identified by `accountID` or `accountID:projectID`. See VictoriaLogs [multitenancy docs](https://docs.victoriametrics.com/victorialogs/#multitenancy).
`sampling_period` `1m` Frequency of the points returned. Will be converted to `/select/stats_query_range?step=%s` param (in seconds).
`data_range` `[0, 'inf']` (Optional) Allows defining **valid** data ranges for input of all the queries in `queries`. Defaults to `["-inf", "inf"]` if not set and can be overridden on a [per-query basis](#per-query-parameters-1).
`tz` `America/New_York` (Optional) Specifies the [IANA](https://nodatime.org/TimeZones) timezone to account for local shifts, like [DST](https://en.wikipedia.org/wiki/Daylight_saving_time), in models sensitive to seasonal patterns (e.g., [`ProphetModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) or [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile)). Defaults to `UTC` if not set and can be overridden on a [per-query basis](#per-query-parameters).
`max_points_per_query` `10000` (Optional) For splitting long `fit_window` [queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vlogs-reader) into smaller sub-intervals. This helps users avoid hitting the timeout limits for individual queries by distributing initial query across multiple subquery requests with minimal overhead. Can be also set on [per-query](#per-query-parameters-1) basis to override reader-level settings.
`offset` `0s` (Optional) Specifies the duration to shift the query window back (or forward) in time. This is useful for accounting for delays in data availability or for aligning the query window with specific events. Can be set on a [per-query basis](#per-query-parameters-1) to override the reader-level setting.
`timeout` `30s` (Optional) Specifies the maximum duration to wait for a query to complete before timing out. Can be set on a [per-query basis](#per-query-parameters-1) to override the reader-level setting.
`verify_tls` `false` Verify TLS certificate. If `False`, it will not verify the TLS certificate. If `True`, it will verify the certificate using the system's CA store. If a path to a CA bundle file (like `ca.crt`), it will verify the certificate using the provided CA bundle.
`tls_cert_file` `path/to/cert.crt` (Optional) Path to a file with the client certificate, i.e. `client.crt`.
`tls_key_file` `path/to/key.key` (Optional) Path to a file with the client key, i.e. `client.key`.
`bearer_token` `token` (Optional) Bearer token for authentication. If set, it will be used to authenticate the request as `Authorization: bearer {token}`.
`bearer_token_file` `path/to/token` (Optional) Path to a file containing the bearer token. If set, it will be used to authenticate the request.
`user` `USERNAME` (Optional) Username for BasicAuth authentication. If set, it will be used to authenticate the request.
`password` `PASSWORD` (Optional) Password for authentication. If set, it will be used to authenticate the request.
### Per-query parameters The names, types and the logic of the per-query parameters subset used in `VLogsReader` are exactly the same as those of [`VmReader`](#vm-reader), please see [per-query parameters](#per-query-parameters) section above for the details. The only difference is that `expr` parameter should contain a valid [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) expression with `stats` [pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe), as described in [query examples](#query-examples) section above. ### Per-query config example ```yaml reader: class: 'vlogs' # or 'reader.vlogs.VLogsReader' # don't include /select/stats_query_range part in the URL, it is added automatically datasource_url: 'https://play-vmlogs.victoriametrics.com/' # source victorialogs # datasource_url: 'https://play-vtraces.victoriametrics.com/' # source victoriatraces # tenant_id: '0:0' # for cluster version only sampling_period: '1m' max_points_per_query: 10000 data_range: [0, 'inf'] # reader-level offset: '0s' # reader-level timeout: '30s' queries: # one query returning 1 result fields (avg_duration), it will have __name__ label (series name) as `duration_30m__avg` duration_avg_30m: expr: "* | stats avg(duration) as avg" # initial LogsQL expression step: '2m' # overrides global `sampling_period` of 1m data_range: [0, 'inf'] # meaning only positive values > 0 are expected, i.e. a value `y` < 0 will trigger anomaly score > 1 tz: 'America/New_York' # to override reader-wise `tz` # tenant_id: '1:0' # overriding tenant_id to isolate data # offset: '-15s' # to override reader-wise `offset` and query data 15 seconds earlier to account for data collection delays # max_points_per_query: 5000 # overrides reader-level value of 10000 for `avg_duration` query # one query returning 3 result fields (p50, p90, p99), they will have __name__ label (series name) as # `duration_quantiles_30m__p50`, `duration_quantiles_30m__p90`, `duration_quantiles_30m__p99`, respectively duration_quantiles_30m: expr: | * | stats quantile(0.5, request_duration_seconds) p50, quantile(0.9, request_duration_seconds) p90, quantile(0.99, request_duration_seconds) p99 step: '2m' # overrides global `sampling_period` of 1m # other per-query parameters as needed # other reader-level parameters as needed # other config sections, like models, schedulers, writer, ... ``` ### mTLS protection Please refer to the [mTLS protection](#mtls-protection) section above for details on how to configure mTLS for `VLogsReader`. It uses the same config parameters as `VmReader` for mTLS setup. ### Healthcheck metrics Similarly to `VmReader`, `VLogsReader` also exposes [several healthchecks metrics](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#reader-behaviour-metrics).