As described above, operands are the leaves of a query expression
tree.
[metric.name] '{metadata qualifiers}' '[time specification]'
Note in most of the query expression examples below, the metadata
qualifiers have been omitted for brevity. In all cases, multiple
time series may qualify, particularly for the hostname label.
In the simple case, a query expression consists of a single
operand and may just be a metric name. In the more general case,
a query expression is either an operand or the argument to a
function, or two operands in a binary arithmetic or logical
expression. Most functions take a single argument (an
expression), though some require additional arguments, e.g.
rescale.
operand | expr operator expr | func(expr[, arg])
This grammar shows expressions may be nested, e.g. using the
addition (+) operator as an example,
func1(func2(expr))
func1(expr) + func2(expr)
expr + func(expr)
func(expr) + expr
expr + expr
Rules governing compatibility of operands in an expression
generally depend on the function and/or operators and are
described below individually. An important rule is that if any
time windows are specified, then all operands must cover the same
number of samples, though the time windows may differ
individually. If no time windows or sample counts are given,
then pmseries will return a series identifier (SID) instead of a
series of timestamps and values. This SID may be used in
subsequent /series/values?series=SID RESTAPI calls, along with a
specific time window.
Arithmetic Operators
pmseries support addition, subtraction, division and
multiplication on each value in the time series of a binary pair
of operands. No unary or ternary operators are supported (yet).
In all cases, the instance domain and the number of samples of
time series operands must be the same. The metadata (units and
dimensions) must also be compatible. Depending on the function,
the result will usually have the same instance domain and (unless
noted otherwise), the same units as the operands. The metadata
dimensions (space, time, count) of the result may differ (see
below).
Expression operands may have different qualifiers, e.g. you can
perform binary arithmetic on metrics qualified by different
labels (such as hostname), or metric names. For example, to add
the two most recent samples of the process context switch
(pswitch) counter metric for hosts node88 and node89, and then
perform rate conversion:
$ pmseries 'rate(kernel.all.pswitch{hostname:node88}[count:2] +
kernel.all.pswitch{hostname:node89}[count:2])'
1cf1a85d5978640ef94c68264d3ae8866cc11f7c
[Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924
Note the resulting time series of values has one less sample than
the expression operand passed to the rate function.
Other rules for arithmetic expressions:
1. if both operands have the semantics of a counter, then only
addition and subtraction are allowed
2. if the left operand is a counter and the right operand is not,
then only multiplication or division are allowed
3. if the left operand is not a counter and the right operand is
a counter, then only multiplication is allowed.
4. addition and subtraction - the dimensions of the result are
the same as the dimensions of the operands.
5. multiplication - the dimensions of the result are the sum of
the dimensions of the operands.
6. division - the dimensions of the result are the difference of
the dimensions of the operands.
Functions
Expression functions operate on vectors of time series values,
and may be nested with other functions or expressions as
described above. When an operand has multiple instances, there
will generally be one result for each series of instances. For
example, the result for
$ pmseries 'min(kernel.all.load[count:100])'
will be the smallest value of the 100 most recent samples,
treating each of the three load average instances as a separate
time series. As an example, for the two most recent samples for
each of the three instances of the load average metric:
$ pmseries 'kernel.all.load[count:2]'
726a325c4c1ba4339ecffcdebd240f441ea77848
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
[Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
Using the min function :
$ pmseries 'min(kernel.all.load[count:2])'
11b965bc5f9598034ed9139fb3a78c6c0b7065ba
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
For singular metrics (with no instance domain), a single value
will result, e.g. for the five most recent samples of the context
switching metric:
$ pmseries 'kernel.all.pswitch[count:5]'
d7832c4fba33bcc980b1a1b614e0508043288480
[Tue Nov 10 12:44:59.380666000 2020] 460774294
[Tue Nov 10 12:44:49.382070000 2020] 460747232
[Tue Nov 10 12:44:39.378545000 2020] 460722370
[Tue Nov 10 12:44:29.379029000 2020] 460697388
[Tue Nov 10 12:44:19.379096000 2020] 460657412
$ pmseries 'min(kernel.all.pswitch[count:5])'
1b6e92fb5bc012372f54452734dd03f0f131fa06
[Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480
Future versions of pmseries may provide functions that perform
aggregation, interpolation, filtering or transforms in other
ways, e.g. across instances instead of time.
Function Reference
max(expr) the maximum value in the time series for each instance
of expr
min(expr) the minimum value in the time series for each instance
of expr
rate(expr) the rate with respect to time of each sample. The
given expr must have counter semantics and the result will have
instant semantics (the time dimension reduced by one). In
addition, the result will have one less sample than the operand -
this is because the first sample cannot be rate converted (two
samples are required).
rescale(expr,scale) rescale the values in the time series for
each instance of expr to scale (units). Note that expr should
have instant or discrete semantics (not counter - rate conversion
should be done first if needed). The time, space and count
dimensions between expr and scale must be compatible. Example:
rate convert the read throughput counter for each disk instance
and then rescale to mbytes per second. Note the native units of
disk.dev.read_bytes is a counter of kbytes read from each device
instance since boot.
$ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'
abs(expr) the absolute value of each value in the time series for
each instance of expr. This has no effect if the type of expr is
unsigned.
floor(expr) rounded down to the nearest integer value of the time
series for each instance of expr.
round(expr) rounded up or down to the nearest integer for each
value in the time series for each instance of expr.
log(expr) logarithm of the values in the time series for each
instance of expr
sqrt(expr) square root of the values in the time series for each
instance of expr
Compatibility
All operands in an expression must have the same number of
samples, but not necessarily the same time window. e.g. you could
subtract some metric time series from today from that of
yesterday by giving different time windows and different metrics
or qualifiers, ensuring the same number of samples are given as
the operands.
Operands in an expression must either all have a time window, or
none. If no operands have a time window, then instead of a
series of time stamps and values, the result will be a time
series identifier (SID) that may be passed to the
/series/values?series=SID REST API function, along with a time
window. For further details, see PMWEBAPI(3).
If the semantics of both operands in an arithmetic expression are
not counter (i.e. PM_SEM_INSTANT or PM_SEM_DISCRETE) then the
result will have semantics PM_SEM_INSTANT unless both operands
are PM_SEM_DISCRETE in which case the result is also
PM_SEM_DISCRETE.
