Persisting metrics over time · Spark Data Quality

To persist the metrics from your ChecksSuite simply pass in a MetricsPersister to your ChecksSuite. You can either use one of the provided implementations or extend MetricsPersister if you wish to use some other storage solution.

Example:

import com.github.timgent.sparkdataquality.checkssuite.ChecksSuite
import com.github.timgent.sparkdataquality.repository.ElasticSearchMetricsPersister
import com.sksamuel.elastic4s.Index

import scala.concurrent.ExecutionContext.Implicits.global

val esHosts = Seq("1.2.3.4")
val esMetricsPersister = ElasticSearchMetricsPersister(esHosts, Index("metrics_index"))
val checksSuite = ChecksSuite("metricsPersister", metricsPersister = esMetricsPersister)

As per qc results persistence you can also pass an ElasticClient to instantiate an ElasticSearchMetricsPersister if you wish to set custom properties like authentication for your ElasticSearch cluster.