Data Reader Interface¶
DataTypes¶

paddle.v2.data_type.
dense_array
(dim, seq_type=0) Dense Array. It means the input feature is dense array with float type. For example, if the input is an image with 28*28 pixels, the input of Paddle neural network could be a dense vector with dimension 784 or a numpy array with shape (28, 28).
For the 2D convolution operation, each sample in one minibatch must have the similarly size in PaddlePaddle now. But, it supports variabledimension feature across minibatch. For the variabledimension, the param dim is not used. While the data reader must yield numpy array and the data feeder will set the data shape correctly.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of input.
Returns: An input type object.
Return type: InputType

paddle.v2.data_type.
dense_vector
(dim, seq_type=0) Dense Array. It means the input feature is dense array with float type. For example, if the input is an image with 28*28 pixels, the input of Paddle neural network could be a dense vector with dimension 784 or a numpy array with shape (28, 28).
For the 2D convolution operation, each sample in one minibatch must have the similarly size in PaddlePaddle now. But, it supports variabledimension feature across minibatch. For the variabledimension, the param dim is not used. While the data reader must yield numpy array and the data feeder will set the data shape correctly.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of input.
Returns: An input type object.
Return type: InputType

paddle.v2.data_type.
dense_vector_sequence
(dim) Data type of a sequence of dense vector.
Parameters: dim (int) – dimension of dense vector. Returns: An input type object Return type: InputType

paddle.v2.data_type.
integer_value
(value_range, seq_type=0) Data type of integer.
Parameters:  seq_type (int) – sequence type of this input.
 value_range (int) – range of this integer.
Returns: An input type object
Return type: InputType

paddle.v2.data_type.
integer_value_sequence
(value_range) Data type of a sequence of integer.
Parameters: value_range (int) – range of each element.

paddle.v2.data_type.
sparse_binary_vector
(dim, seq_type=0) Sparse binary vector. It means the input feature is a sparse vector and the every element in this vector is either zero or one.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of this input.
Returns: An input type object.
Return type: InputType

paddle.v2.data_type.
sparse_binary_vector_sequence
(dim)  Data type of a sequence of sparse vector, which every element is either zero
 or one.
Parameters: dim (int) – dimension of sparse vector. Returns: An input type object Return type: InputType

paddle.v2.data_type.
sparse_float_vector
(dim, seq_type=0) Sparse vector. It means the input feature is a sparse vector. Most of the elements in this vector are zero, others could be any float value.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of this input.
Returns: An input type object.
Return type: InputType

paddle.v2.data_type.
sparse_float_vector_sequence
(dim) Data type of a sequence of sparse vector, which most elements are zero, others could be any float value.
Parameters: dim (int) – dimension of sparse vector. Returns: An input type object Return type: InputType

paddle.v2.data_type.
sparse_non_value_slot
(dim, seq_type=0) Sparse binary vector. It means the input feature is a sparse vector and the every element in this vector is either zero or one.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of this input.
Returns: An input type object.
Return type: InputType

paddle.v2.data_type.
sparse_value_slot
(dim, seq_type=0) Sparse vector. It means the input feature is a sparse vector. Most of the elements in this vector are zero, others could be any float value.
Parameters:  dim (int) – dimension of this vector.
 seq_type (int) – sequence type of this input.
Returns: An input type object.
Return type: InputType

class
paddle.v2.data_type.
InputType
(dim, seq_type, tp) InputType is the base class for paddle input types.
Note
this is a base class, and should never be used by user.
Parameters:  dim (int) – dimension of input. If the input is an integer, it means the value range. Otherwise, it means the size of layer.
 seq_type (int) – sequence type of input. 0 means it is not a sequence. 1 means it is a variable length sequence. 2 means it is a nested sequence.
 type (int) – data type of input.
DataFeeder¶
Reader¶
At training and testing time, PaddlePaddle programs need to read data. To ease the users’ work to write data reading code, we define that
 A reader is a function that reads data (from file, network, random number generator, etc) and yields data items.
 A reader creator is a function that returns a reader function.
 A reader decorator is a function, which accepts one or more readers, and returns a reader.
 A batch reader is a function that reads data (from reader, file, network, random number generator, etc) and yields a batch of data items.
Data Reader Interface¶
Indeed, data reader doesn’t have to be a function that reads and yields data
items. It can be any function with no parameter that creates a iterable
(anything can be used in for x in iterable
):
iterable = data_reader()
Element produced from the iterable should be a single entry of data, not a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of supported type (e.g., numpy 1d array of float32, int, list of int)
An example implementation for single item data reader creator:
def reader_creator_random_image(width, height):
def reader():
while True:
yield numpy.random.uniform(1, 1, size=width*height)
return reader
An example implementation for multiple item data reader creator:
def reader_creator_random_image_and_label(width, height, label):
def reader():
while True:
yield numpy.random.uniform(1, 1, size=width*height), label
return reader
TODO(yuyang18): Should we add whole design doc here?

paddle.v2.reader.
map_readers
(func, *readers) Creates a data reader that outputs return value of function using output of each data readers as arguments.
Parameters:  func – function to use. The type of func should be (Sample) => Sample
 readers – readers whose outputs will be used as arguments of func.
Type: callable
Returns: the created data reader.
Return type: callable

paddle.v2.reader.
buffered
(reader, size) Creates a buffered data reader.
The buffered data reader will read and save data entries into a buffer. Reading from the buffered data reader will proceed as long as the buffer is not empty.
Parameters:  reader (callable) – the data reader to read from.
 size (int) – max buffer size.
Returns: the buffered data reader.

paddle.v2.reader.
compose
(*readers, **kwargs) Creates a data reader whose output is the combination of input readers.
If input readers output following data entries: (1, 2) 3 (4, 5) The composed reader will output: (1, 2, 3, 4, 5)
Parameters:  readers – readers that will be composed together.
 check_alignment (bool) – if True, will check if input readers are aligned correctly. If False, will not check alignment and trailing outputs will be discarded. Defaults to True.
Returns: the new data reader.
Raises: ComposeNotAligned – outputs of readers are not aligned. Will not raise when check_alignment is set to False.

paddle.v2.reader.
chain
(*readers) Creates a data reader whose output is the outputs of input data readers chained together.
If input readers output following data entries: [0, 0, 0] [1, 1, 1] [2, 2, 2] The chained reader will output: [0, 0, 0, 1, 1, 1, 2, 2, 2]
Parameters: readers – input readers. Returns: the new data reader. Return type: callable

paddle.v2.reader.
shuffle
(reader, buf_size) Creates a data reader whose data output is shuffled.
Output from the iterator that created by original reader will be buffered into shuffle buffer, and then shuffled. The size of shuffle buffer is determined by argument buf_size.
Parameters:  reader (callable) – the original reader whose output will be shuffled.
 buf_size (int) – shuffle buffer size.
Returns: the new reader whose output is shuffled.
Return type: callable

paddle.v2.reader.
firstn
(reader, n) Limit the max number of samples that reader could return.
Parameters:  reader (callable) – the data reader to read from.
 n (int) – the max number of samples that return.
Returns: the decorated reader.
Return type: callable

paddle.v2.reader.
xmap_readers
(mapper, reader, process_num, buffer_size, order=False) Use multiprocess to map samples from reader by a mapper defined by user. And this function contains a buffered decorator. :param mapper: a function to map sample. :type mapper: callable :param reader: the data reader to read from :type reader: callable :param process_num: process number to handle original sample :type process_num: int :param buffer_size: max buffer size :type buffer_size: int :param order: keep the order of reader :type order: bool :return: the decarated reader :rtype: callable

paddle.v2.reader.
pipe_reader
(left_cmd, parser, bufsize=8192, file_type='plain', cut_lines=True, line_break='\n') pipe_reader read data by stream from a command, take it’s stdout into a pipe buffer and redirect it to the parser to parse, then yield data as your desired format.
You can using standard linux command or call another program to read data, from HDFS, Ceph, URL, AWS S3 etc:
cmd = “hadoop fs cat /path/to/some/file” cmd = “cat sample_file.tar.gz” cmd = “curl http://someurl” cmd = “python print_s3_bucket.py”
A sample parser:
 def sample_parser(lines):
# parse each line as one sample data, # return a list of samples as batches. ret = [] for l in lines:
ret.append(l.split(” ”)[1:5])return ret
param left_cmd: command to excute to get stdout from. type left_cmd: string param parser: parser function to parse lines of data. if cut_lines is True, parser will receive list of lines. if cut_lines is False, parser will receive a raw buffer each time. parser should return a list of parsed values. type parser: callable param bufsize: the buffer size used for the stdout pipe. type bufsize: int param file_type: can be plain/gzip, stream buffer data type. type file_type: string param cut_lines: whether to pass lines instead of raw buffer to the parser type cut_lines: bool param line_break: line break of the file, like  or
type line_break: string return: the reader generator. rtype: callable
Creator package contains some simple reader creator, which could be used in user program.

paddle.v2.reader.creator.
np_array
(x) Creates a reader that yields elements of x, if it is a numpy vector. Or rows of x, if it is a numpy matrix. Or any subhyperplane indexed by the highest dimension.
Parameters: x – the numpy array to create reader from. Returns: data reader created from x.

paddle.v2.reader.creator.
text_file
(path) Creates a data reader that outputs text line by line from given text file. Trailing new line (‘\n’) of each line will be removed.
Path: path of the text file. Returns: data reader of text file

paddle.v2.reader.creator.
cloud_reader
(paths, etcd_endpoints, timeout_sec=5, buf_size=64)  Create a data reader that yield a record one by one from
 the paths:
Paths: path of recordio files, can be a string or a string list. Etcd_endpoints: the endpoints for etcd cluster Returns: data reader of recordio files.
minibatch¶

paddle.v2.minibatch.
batch
(reader, batch_size) Create a batched reader.
Parameters:  reader (callable) – the data reader to read from.
 batch_size (int) – size of each minibatch
Returns: the batched reader.
Return type: callable