Unconstrained Deep Learning

Train unconstrained deep learning for CIFAR-10 classification using modified LeNet5 based on this PyTorch tutorial

Problem Description

We have a simple feed-forward network. The input is an image, which is fed through several layers to obtain the output. The logit output is used to decide the label of the input image. Below is a demo image of LeNet5: image.png

Modules Importing

Import all necessary modules and add PyGRANSO src folder to system path.

[1]:
import time
import torch
from pygranso.pygranso import pygranso
from pygranso.pygransoStruct import pygransoStruct
from pygranso.private.getNvar import getNvarTorch
import torch.nn as nn
import torchvision.transforms as transforms
import torch.nn.functional as F
import torchvision

Data Initialization

Specify torch device, neural network architecture, and generate data.

NOTE: please specify path for downloading data.

Use GPU for this problem. If no cuda device available, please set device = torch.device(‘cpu’)

[2]:
device = torch.device('cuda')

class Net(nn.Module):
     def __init__(self):
             super().__init__()
             self.conv1 = nn.Conv2d(3, 6, 5)
             self.conv1_bn = nn.BatchNorm2d(6)
             self.pool = nn.MaxPool2d(2, 2)
             self.conv2 = nn.Conv2d(6, 8, 9)
             self.conv2_bn = nn.BatchNorm2d(8)
             self.fc1 = nn.Linear(8 * 3 * 3, 30)
             self.fc1_bn = nn.BatchNorm1d(30)
             self.fc2 = nn.Linear(30, 20)
             self.fc2_bn = nn.BatchNorm1d(20)
             self.fc3 = nn.Linear(20, 10)

     def forward(self, x):
             x = self.pool(F.elu( self.conv1_bn(self.conv1(x))  ))
             x = self.pool(F.elu( self.conv2_bn(self.conv2(x))  ))
             x = torch.flatten(x, 1) # flatten all dimensions except batch
             x = F.elu( self.fc1_bn(self.fc1(x)) )
             x = F.elu( self.fc2_bn(self.fc2(x)) )
             x = self.fc3(x)
             return x

# fix model parameters
torch.manual_seed(0)
model = Net().to(device=device, dtype=torch.double)

transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
batch_size = 1000
trainset = torchvision.datasets.CIFAR10(root='/home/buyun/Documents/GitHub/PyGRANSO/examples', train=True, download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=False, num_workers=2)
# data_in
for i, data in enumerate(trainloader, 0):
    if i >= 1:
         break
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

# All the user-provided data (vector/matrix/tensor) must be in torch tensor format.
# As PyTorch tensor is single precision by default, one must explicitly set `dtype=torch.double`.
# Also, please make sure the device of provided torch tensor is the same as opts.torch_device.
labels = labels.to(device=device) # label/target [256]
inputs = inputs.to(device=device, dtype=torch.double) # input data [256,3,32,32]
Files already downloaded and verified

Function Set-Up

Encode the optimization variables, and objective and constraint functions.

Note: please strictly follow the format of comb_fn, which will be used in the PyGRANSO main algortihm.

[3]:
def user_fn(model,inputs,labels):
    # objective function
    outputs = model(inputs)
    criterion = nn.CrossEntropyLoss()
    f = criterion(outputs, labels)
    ci = None
    ce = None
    return [f,ci,ce]

comb_fn = lambda model : user_fn(model,inputs,labels)

User Options

Specify user-defined options for PyGRANSO

[4]:
opts = pygransoStruct()
opts.torch_device = device
nvar = getNvarTorch(model.parameters())
opts.x0 = torch.nn.utils.parameters_to_vector(model.parameters()).detach().reshape(nvar,1)
opts.opt_tol = 1e-3
# opts.fvalquit = 1e-6
opts.print_level = 1
opts.print_frequency = 10
# opts.print_ascii = True

Initial Test

Check initial accuracy of the modified LeNet5 model

[5]:
outputs = model(inputs )
acc = (outputs.max(1)[1] == labels).sum().item()/labels.size(0)

print("Initial acc = {}".format(acc))
Initial acc = 0.102
/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448255797/work/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

Main Algorithm

[6]:
start = time.time()
soln = pygranso(var_spec= model, combined_fn = comb_fn, user_opts = opts)
end = time.time()
print("Total Wall Time: {}s".format(end - start))


╔═════ QP SOLVER NOTICE ════════════════════════════════════════════════════════════════════════╗
║  PyGRANSO requires a quadratic program (QP) solver that has a quadprog-compatible interface,  ║
║  the default is osqp. Users may provide their own wrapper for the QP solver.                  ║
║  To disable this notice, set opts.quadprog_info_msg = False                                   ║
╚═══════════════════════════════════════════════════════════════════════════════════════════════╝
══════════════════════════════════════════════════════════════════════════════════════════════╗
PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation                          ║
Version 1.0.0                                                                                 ║
Licensed under the AGPLv3, Copyright (C) 2021 Tim Mitchell and Buyun Liang                    ║
══════════════════════════════════════════════════════════════════════════════════════════════╣
Problem specifications:                                                                       ║
 # of variables                     :   7500                                                  ║
 # of inequality constraints        :      0                                                  ║
 # of equality constraints          :      0                                                  ║
═════╦════════════╦════════════════╦═════════════╦═══════════════════════╦════════════════════╣
     ║ Penalty Fn ║                ║  Violation  ║ <--- Line Search ---> ║ <- Stationarity -> ║
Iter ║ Mu │ Value ║    Objective   ║ Ineq │  Eq  ║ SD │ Evals │     t    ║ Grads │    Value   ║
═════╬════════════╬════════════════╬═════════════╬═══════════════════════╬════════════════════╣
   0 ║  - │   -   ║  2.39541706590 ║   -  │   -  ║ -  │     1 │ 0.000000 ║     1 │ 1.473948   ║
  10 ║  - │   -   ║  1.63783103375 ║   -  │   -  ║ QN │     1 │ 1.000000 ║     1 │ 0.269115   ║
  20 ║  - │   -   ║  1.26055629664 ║   -  │   -  ║ QN │     1 │ 1.000000 ║     1 │ 0.863316   ║
  30 ║  - │   -   ║  0.84374002729 ║   -  │   -  ║ QN │     1 │ 1.000000 ║     1 │ 0.222527   ║
  40 ║  - │   -   ║  0.39057278862 ║   -  │   -  ║ QN │     1 │ 1.000000 ║     1 │ 0.224242   ║
  50 ║  - │   -   ║  0.08692818754 ║   -  │   -  ║ QN │     1 │ 1.000000 ║     1 │ 0.498670   ║
  60 ║  - │   -   ║  0.00766217133 ║   -  │   -  ║ QN │     3 │ 4.000000 ║     1 │ 0.106859   ║
  70 ║  - │   -   ║  0.00111545451 ║   -  │   -  ║ QN │     2 │ 2.000000 ║     1 │ 0.028518   ║
  80 ║  - │   -   ║  3.9223359e-04 ║   -  │   -  ║ QN │     2 │ 2.000000 ║     1 │ 0.005710   ║
═════╩════════════╩════════════════╩═════════════╩═══════════════════════╩════════════════════╣
Optimization results:                                                                         ║
F = final iterate, B = Best (to tolerance), MF = Most Feasible                                ║
═════╦════════════╦════════════════╦═════════════╦═══════════════════════╦════════════════════╣
   F ║    │       ║  1.9363262e-04 ║   -  │   -  ║    │       │          ║       │            ║
   B ║    │       ║  1.9363262e-04 ║   -  │   -  ║    │       │          ║       │            ║
═════╩════════════╩════════════════╩═════════════╩═══════════════════════╩════════════════════╣
Iterations:              89                                                                   ║
Function evaluations:    154                                                                  ║
PyGRANSO termination code: 0 --- converged to stationarity tolerance.                         ║
══════════════════════════════════════════════════════════════════════════════════════════════╝
Total Wall Time: 6.698632478713989s

Train Accuracy

[7]:
torch.nn.utils.vector_to_parameters(soln.final.x, model.parameters())
outputs = model(inputs)
acc = (outputs.max(1)[1] == labels).sum().item()/labels.size(0)
print("Train acc = {}".format(acc))
Train acc = 1.0