Single-node training¶
Preparation¶
To perform single-node training in PaddlePaddle Fluid, you need to read Prepare Data and Set up Simple Model . When you have finished reading Set up Simple Model , you can get two fluid.Program
, namely startup_program
and main_program
. By default, you can use fluid.default_startup_program()
and fluid.default_main_program()
to get global fluid.Program
.
For example:
import paddle.fluid as fluid
image = fluid.layers.data(name="image", shape=[784])
label = fluid.layers.data(name="label", shape=[1])
hidden = fluid.layers.fc(input=image, size=100, act='relu')
prediction = fluid.layers.fc(input=hidden, size=10, act='softmax')
loss = fluid.layers.mean(
fluid.layers.cross_entropy(
input=prediction,
label=label
)
)
sgd = fluid.optimizer.SGD(learning_rate=0.001)
sgd.minimize(loss)
# Here the fluid.default_startup_program() and fluid.default_main_program()
# has been constructed.
After the configuration of model, the configurations of fluid.default_startup_program()
and fluid.default_main_program()
have been finished.
Initialize Parameters¶
Random Initialization of Parameters¶
After the configuration of model,the initialization of parameters will be written into fluid.default_startup_program()
. By running this program in fluid.Executor()
, the random initialization of parameters will be finished in global fluid.global_scope()
.For example:
exe = fluid.Executor(fluid.CUDAPlace(0))
exe.run(program=fluid.default_startup_program())
Note that in multi-GPU training, the parameters should be initialized on GPU0 and then will be distributed to multiple graphic cards through fluid.ParallelExecutor
.
Load Predefined Parameters¶
In the neural network training, predefined models are usually loaded to continue training. For how to load predefined parameters, please refer to Save, Load Models or Variables & Incremental Learning.
Single-card Training¶
Single-card training can be performed through calling run()
of fluid.Executor()
to run training fluid.Program
.
In the runtime, feed data with run(feed=...)
and get persistable data with run(fetch=...)
. For example:
...
loss = fluid.layers.mean(...)
exe = fluid.Executor(...)
# the result is an numpy array
result = exe.run(feed={"image": ..., "label": ...}, fetch_list=[loss])
Notes:
- About data type supported by feed, please refer to the article Transfer Train Data to Executor.
- The return value of
Executor.run
is the variable value offetch_list=[...]
.The fetched Variable must be persistable.fetch_list
can be fed with either Variable list or name list of variables .Executor.run
returns Fetch result list. - If the fetched data contain sequence information, you can set
exe.run(return_numpy=False, ...)
to directly getfluid.LoDTensor
. You can directly access the information influid.LoDTensor
.
Multi-card Training¶
In multi-card training, you can use fluid.ParallelExecutor
to run training fluid.Program
. For example:
train_exe = fluid.ParallelExecutor(use_cuda=True, loss_name=loss.name,
main_program=fluid.default_main_program())
train_exe.run(fetch_list=[loss.name], feed={...})
Notes:
- The constructor of
ParallelExecutor
needs to be set withfluid.Program
to be run which can not be modified at runtime. The default value isfluid.default_main_program()
. ParallelExecutor
should be indicated whether to use CUDA to train. In the mode of graphic card training, all graphic cards will be occupied. Users can configure CUDA_VISIBLE_DEVICES to change graphics cards that are being used.