7

Working in Docker Grafana 8.1.5. Using time series graph, I'm plotting a Prometheus Counter source (that has one label) as a time series (by label), and need to fill all null/missing values as zeros.

This is the query applied to the Prometheus counter source, plotting the label code:

my_metric{code!=""}

The graph display works (just need to see the current counter value for each label variant, and the difference within the selected time range), but the new Grafana time series graph is missing an option that the Graph (old) has under Display > Stacking and null value > null value: null as zero, hence it now ends up with broken lines when null values occur.

Unfortunately, I cannot use the Graph (old) chart as I need the legend value difference, which only is available in the new time series graph.

I tried to add or on() vector(0) to the end of the query, but the condition does not get applied to the data series for each label variant, it rather adds a new data series all filled with zeros...

Thanks for any suggestions !

pete19
  • 333
  • 1
  • 3
  • 17

2 Answers2

18

I had this issue as well and I was not able to use only or on() vector(0) as you mentioned because the main query was returning NaN. In my case I had a division by zero.

I could get around of it by first evaluate if the query has a value >= 0 and then use the or on() vector(0). Try something similar to:

((my_metric{code!=""}) >= 0) OR on() vector(0)
Felipe
  • 7,013
  • 8
  • 44
  • 102
  • 3
    this does not work in my situation to fill all null/missing values as zero, it also simply adds another, separate line that consists of zeros – pete19 Oct 11 '21 at 22:22
  • Hey @pete19 , could you please post a screenshot of the result? It is a little bit hard to understand your problem only by describing it. – Felipe Oct 12 '21 at 05:58
  • Felipe, this works for some queries, but want to note that one often needs to wrap the query then in an extra set of parentheses `()` if the result is used in other functions... – pete19 Oct 16 '21 at 00:21
  • @pete19 I am glad that you made it to work. I edited my answer with your information. In case you think that my post could answer your question, would you mind to acceptas a answer? I thank you in advance – Felipe Oct 16 '21 at 11:36
  • 1
    @Felipe very nice trick on that ` OR on() vector(0)`. Works very well where you want to show a RAG status, and panel doesn't like null series. – Adarsha Feb 14 '22 at 22:53
10

The following technique can be used for filling gaps in a single time series returned from the query q:

sum(q) or vector(0)

The q is wrapped into sum() in order to drop all the labels for a time series returned from q, so the time series can be matched with vector(0) time series according to matching rules for or operator. The query without sum(): q or on() vector(0) would return two time series on the graph instead of a single time series: one time series from q with its own set of labels and another time series with zero labels and with zero value where the q time series contains gaps.

Unfortunately Prometheus doesn't provide an easy ability to fill gaps with zeroes if q returns multiple time series with distinct labelsets. Other Prometheus-like solutions such as VictoriaMetrics (I work on it) provides default operator for this case. For example, the following MetricsQL query fills gaps with zeroes for all the time series returned from q:

q default 0
valyala
  • 11,669
  • 1
  • 59
  • 62