evaluator

ChunkEvaluator

class paddle.fluid.evaluator.ChunkEvaluator(input, label, chunk_scheme, num_chunk_types, excluded_chunk_types=None)

Accumulate counter numbers output by chunk_eval from mini-batches and compute the precision recall and F1-score using the accumulated counter numbers.

EditDistance

class paddle.fluid.evaluator.EditDistance(input, label, ignored_tokens=None, **kwargs)

Accumulate edit distance sum and sequence number from mini-batches and compute the average edit_distance and instance error of all batches.

Parameters:
  • input – the sequences predicted by network.
  • label – the target sequences which must has same sequence count
  • input. (with) –
  • ignored_tokens (list of int) – Tokens that should be removed before
  • edit distance. (calculating) –

Example

exe = fluid.executor(place) distance_evaluator = fluid.Evaluator.EditDistance(input, label) for epoch in PASS_NUM:

distance_evaluator.reset(exe) for data in batches:

loss = exe.run(fetch_list=[cost])

distance, instance_error = distance_evaluator.eval(exe)

In the above example: ‘distance’ is the average of the edit distance in a pass. ‘instance_error’ is the instance error rate in a pass.

DetectionMAP

class paddle.fluid.evaluator.DetectionMAP(input, gt_label, gt_box, gt_difficult, class_num, background_label=0, overlap_threshold=0.5, evaluate_difficult=True, ap_version='integral')

Calculate the detection mean average precision (mAP).

TODO (Dang Qingqing): update the following doc. The general steps are as follows: 1. calculate the true positive and false positive according to the input

of detection and labels.
  1. calculate mAP value, support two versions: ‘11 point’ and ‘integral’.
Please get more information from the following articles:
https://sanchom.wordpress.com/tag/average-precision/ https://arxiv.org/abs/1512.02325
Parameters:
  • input (Variable) – The detection results, which is a LoDTensor with shape [M, 6]. The layout is [label, confidence, xmin, ymin, xmax, ymax].
  • gt_label (Variable) – The ground truth label index, which is a LoDTensor with shape [N, 1].
  • gt_difficult (Variable) – Whether this ground truth is a difficult bounding box (bbox), which is a LoDTensor [N, 1].
  • gt_box (Variable) – The ground truth bounding box (bbox), which is a LoDTensor with shape [N, 6]. The layout is [xmin, ymin, xmax, ymax].
  • class_num (int) – The class number.
  • background_label (int) – The index of background label, the background label will be ignored. If set to -1, then all categories will be considered, 0 by defalut.
  • overlap_threshold (float) – The threshold for deciding true/false positive, 0.5 by defalut.
  • evaluate_difficult (bool) – Whether to consider difficult ground truth for evaluation, True by defalut.
  • ap_version (string) – The average precision calculation ways, it must be ‘integral’ or ‘11point’. Please check https://sanchom.wordpress.com/tag/average-precision/ for details. - 11point: the 11-point interpolated average precision. - integral: the natural integral of the precision-recall curve.

Example

exe = fluid.executor(place) map_evaluator = fluid.Evaluator.DetectionMAP(input,

gt_label, gt_difficult, gt_box)

cur_map, accum_map = map_evaluator.get_map_var() fetch = [cost, cur_map, accum_map] for epoch in PASS_NUM:

map_evaluator.reset(exe) for data in batches:

loss, cur_map_v, accum_map_v = exe.run(fetch_list=fetch)

In the above example:

‘cur_map_v’ is the mAP of current mini-batch. ‘accum_map_v’ is the accumulative mAP of one pass.