data_feeder¶

DataFeeder¶

class paddle.fluid.data_feeder.DataFeeder(feed_list, place, program=None)

DataFeeder converts the data that returned by a reader into a data structure that can feed into Executor and ParallelExecutor. The reader usually returns a list of mini-batch data entries. Each data entry in the list is one sample. Each sample is a list or a tuple with one feature or multiple features.

The simple usage shows below:

place = fluid.CPUPlace()
img = fluid.layers.data(name='image', shape=[1, 28, 28])
label = fluid.layers.data(name='label', shape=[1], dtype='int64')
feeder = fluid.DataFeeder([img, label], fluid.CPUPlace())
result = feeder.feed([([0] * 784, [9]), ([1] * 784, [1])])

If you want to feed data into GPU side separately in advance when you use multi-GPU to train a model, you can use decorate_reader function.

place=fluid.CUDAPlace(0)
feeder = fluid.DataFeeder(place=place, feed_list=[data, label])
reader = feeder.decorate_reader(
    paddle.batch(flowers.train(), batch_size=16))

Parameters:

feed_list (list) – The Variables or Variables’name that will feed into model. place (Place) – place indicates feed data into CPU or GPU, if you want to feed data into GPU, please using fluid.CUDAPlace(i) (i represents the GPU id), or if you want to feed data into CPU, please using fluid.CPUPlace(). program (Program) – The Program that will feed data into, if program is None, it will use default_main_program(). Default None.

Raises:

ValueError – If some Variable is not in this Program.

Examples

# ...
place = fluid.CPUPlace()
feed_list = [
    main_program.global_block().var(var_name) for var_name in feed_vars_name
] # feed_vars_name is a list of variables' name.
feeder = fluid.DataFeeder(feed_list, place)
for data in reader():
    outs = exe.run(program=main_program,
                   feed=feeder.feed(data))

feed(iterable)

According to feed_list and iterable, converters the input into a data structure that can feed into Executor and ParallelExecutor.

Parameters:	iterable (list|tuple) – the input data.
Returns:	the result of conversion.
Return type:	dict

feed_parallel(iterable, num_places=None)

Takes multiple mini-batches. Each mini-batch will be feed on each device in advance.

Parameters:	iterable (list|tuple) – the input data. num_places (int) – the number of devices. Default None.
Returns:	the result of conversion.
Return type:	dict

Notes

The number of devices and number of mini-batches must be same.

decorate_reader(reader, multi_devices, num_places=None, drop_last=True)

Converter the input data into a data that returned by reader into multiple mini-batches. Each mini-batch will be feed on each device.

Parameters:	reader (fun) – the input data. multi_devices (bool) – the number of places. Default None. num_places (int) – the number of places. Default None. drop_last (bool) – the number of places. Default None.
Returns:	the result of conversion.
Return type:	dict
Raises:	ValueError – If drop_last is False and the data batch which cannot fit for devices.