Data Flare

Data Flare

  • Docs
  • API

›Persisting your results

Getting Started

  • Introduction
  • Writing your first suite of checks
  • Supported Scala and Spark versions

Available Checks

    Metrics based checks

    • Introduction to metrics and metric based checks
    • Metrics based checks on a single Dataset
    • Metrics based checks on a pair of Datasets
    • Track metrics not involved in any checks
    • Available metrics
  • Arbitrary checks

Persisting your results

  • Persisting results from your checks
  • Persisting metrics over time

Developer docs

  • Developer documentation

Persisting results from your checks

To persist the results from your ChecksSuite simply pass in a QcResultsRepository to your ChecksSuite. You can either use one of the provided implementations or extend QcResultsRepository if you wish to use some other storage solution.

Example:

import com.github.timgent.dataflare.checkssuite._
import com.github.timgent.dataflare.repository.ElasticSearchQcResultsRepository
import com.sksamuel.elastic4s.Index

import scala.concurrent.ExecutionContext.Implicits.global

// List of ElasticSearch hosts
val myEsHosts = Seq("1.2.3.4")
val elasticSearchQcResultsRepository = ElasticSearchQcResultsRepository(hosts = myEsHosts, index = Index("my_index"))
val checksSuite = ChecksSuite("persistingChecksSuite", qcResultsRepository = elasticSearchQcResultsRepository)

Of course in this example you would need to also pass in some checks to perform.

You can also create an ElasticSearchQcResultsRepository with custom properties, for example if you wish to use authentication.

import com.sksamuel.elastic4s.{ElasticClient, ElasticNodeEndpoint, ElasticProperties}
import com.sksamuel.elastic4s.http.JavaClient
val hostList = Seq(ElasticNodeEndpoint(protocol = "http", host = "1.2.3.4", port = 9092, prefix = None))
val client: ElasticClient = ElasticClient(JavaClient(ElasticProperties(hostList, options = Map())))

Check out the elastic4s documentation for more details on constructing a client.

← Arbitrary checksPersisting metrics over time →
Data Flare
Docs
Getting StartedAPI Reference
Community
Stack Overflow
More
GitHubStar