fluid.recordio_writer

convert_reader_to_recordio_file

paddle.fluid.recordio_writer.convert_reader_to_recordio_file(filename, reader_creator, feeder, compressor=Compressor.Snappy, max_num_records=1000, feed_order=None)

Convert a Python Reader to a recordio file.

Please see api_guide_python_reader and api_guide_reader_op for details.

Examples

>>> import paddle.fluid as fluid
>>> import paddle.dataset.mnist as mnist
>>> import paddle
>>>
>>> tmp_program = fluid.Program()
>>> with fluid.program_guard(tmp_program):
>>>     img = fluid.layers.data(name='img', shape=[784])
>>>     label = fluid.layers.data(name='label', shape=[1], dtype='int64')
>>> feeder = fluid.DataFeeder(feed_list=[img, label], place=fluid.CPUPlace())
>>> # mnist.recordio will be generated in current directory
>>> fluid.recordio_writer.convert_reader_to_recordio_file(
>>>                     filename="mnist.recordio",
>>>                     reader_creator=paddle.batch(mnist.train(), batch_size=32),
>>>                     feeder=feeder)
Parameters:
  • filename (str) – The recordio filename.
  • reader_creator (callable) – The Python Reader Creator. See api_guide_python_reader.
  • feeder (DataFeeder) – The DataFeeder instance. Used to convert reader_creator to :code: lod_tensor
  • compressor – Must in fluid.core.RecordIOWriter.Compressor.Snappy or fluid.core.RecordIOWriter.Compressor.NoCompress. Use Snappy by default.
  • max_num_records (int) – Maximum number of records in one chuck. Each record is each return value from reader function
  • feed_order (list) – The order of variable names that the reader returns
Returns:

the number of record that saved.

Return type:

int

convert_reader_to_recordio_files

paddle.fluid.recordio_writer.convert_reader_to_recordio_files(filename, batch_per_file, reader_creator, feeder, compressor=Compressor.Snappy, max_num_records=1000, feed_order=None)

convert a python reader to many recordio files.

This API is basically same as convert_reader_to_recordio_file, instead of it will create many recordio files. Each file contains at most batch_per_file records.

Please reference convert_reader_to_recordio_file for more details.