How to resume training?

Yilin_Liu · October 11, 2017, 8:17pm

How can I resume training from halfway? thanks!

vabh · October 11, 2017, 8:39pm

Take a look at this example

Mode state saving code:

pytorch/examples/blob/master/imagenet/main.py#L165-L171


save_checkpoint({
    'epoch': epoch + 1,
    'arch': args.arch,
    'state_dict': model.state_dict(),
    'best_prec1': best_prec1,
    'optimizer' : optimizer.state_dict(),
}, is_best)

Model resume code:

github.com

pytorch/examples/blob/master/imagenet/main.py#L98-L107


if args.resume:
    if os.path.isfile(args.resume):
        print("=> loading checkpoint '{}'".format(args.resume))
        checkpoint = torch.load(args.resume)
        args.start_epoch = checkpoint['epoch']
        best_prec1 = checkpoint['best_prec1']
        model.load_state_dict(checkpoint['state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        print("=> loaded checkpoint '{}' (epoch {})"
              .format(args.resume, checkpoint['epoch']))

nicolasdespres · October 19, 2017, 4:20am

If you run your experiments inside a Docker container you may find this link interesting:

github.com

docker/cli/blob/master/experimental/checkpoint-restore.md

# Docker Checkpoint & Restore

Checkpoint & Restore is a new feature that allows you to freeze a running
container by checkpointing it, which turns its state into a collection of files
on disk. Later, the container can be restored from the point it was frozen.

This is accomplished using a tool called [CRIU](http://criu.org), which is an
external dependency of this feature. A good overview of the history of
checkpoint and restore in Docker is available in this
[Kubernetes blog post](http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html).

## Installing CRIU

If you use a Debian system, you can add the CRIU PPA and install with apt-get
[from the criu launchpad](https://launchpad.net/~criu/+archive/ubuntu/ppa).

Alternatively, you can [build CRIU from source](http://criu.org/Installation).

You need at least version 2.0 of CRIU to run checkpoint/restore in Docker.

This file has been truncated. show original

Lee_Jim · September 20, 2018, 6:27am

nst as interestx or not