This operator uses the Local Outlier Factor (LOF) algorithm to find anomalies. You should use this operator with a window, because the operator saves all tuples in the current window and uses the normal LOF-algorithm on this set of tuples. If the number of tuples is too high, the algorithm may take a while to process the calculations.

Parameters

  • NEIGHBORS The number of neighbors used, sometimes called k
  • LOFVALUE The value from which the tuples are declared as anomalies. Values near 1 are normal, higher values are anomalies. Standard is 1.5.
  • attribute Name of the attribute which should be analysed
  • GROUP_BY Group by the given attribute, e.g. if you have a context like 'on' and 'off' you want to analyse separately.
  • fastGrouping Use hash code instead of tuple compare to create group. Potentially unsafe!
  • ignoreEqual Set to true, if you want to ignore, if there are a lot of equal values. This can lead to false positives.
  • deliverFirstElements The operator starts to work when the window contains at least the number of neighbors + 1 tuples. If you want to get the first elements before this is reached, you can set this to true. Default is false.

The option "ignoreEqual" is useful, if the data stream has a lot of equal values. If thats the case, the distance between these values is zero. If there is one value, which is very close to these equal values and hence, the k nearest neighbors of this new value are all equal, the LOF will be infinity. That's because the average distance between the neighbors of this value if 0, there even the smallest distance greater than 0 will lead to an infinite LOF. If you set "ignoreEqual" to true, only one equal neighbor will be considered for the calculation of the LOF value.

Example

PQL query for the LOF operator
/// Searches for anomalies in temperature data
#PARSER PQL
#ADDQUERY
temperatureWindow = ELEMENTWINDOW({
                        size = 100
                      },
                      System.temperature
                    )
                    
lofAnalysis = LOFANOMALYDETECTION({
                  lofvalue = 2.5,
                  neighbors = 5,
                  ignoreequal = 'true',
                  attribute = 'temp',
                  deliverFirstElements = 'true'
                },
                temperatureWindow
              )
  • No labels