How to Configure Metrics Collector

This guide describes how Katib metrics collector works.

Metrics Collector

In the metricsCollectorSpec section of the Experiment YAML configuration file, you can define how Katib should collect the metrics from each Trial, such as the accuracy and loss metrics.

Your training code can record the metrics into StdOut or into arbitrary output files. Katib collects the metrics using a sidecar container. A sidecar is a utility container that supports the main container in the Kubernetes Pod.

To define the metrics collector for your Experiment:

1. Specify the collector type in the .collector.kind field. Katib’s metrics collector supports the following collector types:

   • StdOut: Katib collects the metrics from the operating system’s default output location (standard output). This is the default metrics collector.

   • File: Katib collects the metrics from an arbitrary file, which you specify in the .source.fileSystemPath.path field. Training container should log metrics to this file in TEXT or JSON format. If you select JSON format, metrics must be line-separated by epoch or step as follows, and the key for timestamp must be timestamp:

     {"epoch": 0, "foo": “bar", “fizz": “buzz", "timestamp": 1638422847.28721…}
     {"epoch": 1, "foo": “bar", “fizz": “buzz", "timestamp": 1638422847.287801…}
     {"epoch": 2, "foo": “bar", “fizz": “buzz", "timestamp": "2021-12-02T14:27:50.000035161+09:00"…}
     {"epoch": 3, "foo": “bar", “fizz": “buzz", "timestamp": "2021-12-02T14:27:50.000037459+09:00"…}
     …
     

     Check the file metrics collector example for TEXT and JSON format. Also, the default file path is /var/log/katib/metrics.log, and the default file format is TEXT.

   • TensorFlowEvent: Katib collects the metrics from a directory path containing a tf.Event. You should specify the path in the .source.fileSystemPath.path field. Check the TFJob example. The default directory path is /var/log/katib/tfevent/.

   • Custom: Specify this value if you need to use a custom way to collect metrics. You must define your custom metrics collector container in the .collector.customCollector field. Check the custom metrics collector example.

   • None: Specify this value if you don’t need to use Katib’s metrics collector. For example, your training code may handle the persistent storage of its own metrics.

2. Write code in your training container to print or save to the file metrics in the format specified in the .source.filter.metricsFormat field. The default metrics format value is:

   ([\w|-]+)\s*=\s*([+-]?\d*(\.\d+)?([Ee][+-]?\d+)?)
   

   Each element is a regular expression with two sub-expressions. The first matched expression is taken as the metric name. The second matched expression is taken as the metric value.

   For example, using the default metrics format and StdOut metrics collector, if the name of your objective metric is loss and the additional metrics are recall and precision, your training code should print the following output:

   epoch 1:
   loss=3.0e-02
   recall=0.5
   precision=.4
   
   epoch 2:
   loss=1.3e-02
   recall=0.55
   precision=.5