{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5257bb27",
   "metadata": {},
   "source": [
    "# Perceptual Attack\n",
    "\n",
    "Apply neural perceptual attack to images taken from: Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. [\"Perceptual adversarial robustness: Defense against unseen threat models.\"](https://arxiv.org/abs/2006.12655) arXiv preprint arXiv:2006.12655 (2020)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4364b1e6",
   "metadata": {},
   "source": [
    "## Problem Description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344f7377",
   "metadata": {},
   "source": [
    "Given a classifier $f$ which maps any input image $x \\in X$ to its label $y = f(x) \\in Y$. The goal of neural perceptual attack is to find an input $\\widetilde{x}$ that is perceptually similar to the original input $x$ but can fool the classifier $f$. This can be formulated as:\n",
    "\n",
    "$$\\max_{\\widetilde{x}} L (f(\\widetilde{x}),y),$$\n",
    "$$\\text{s.t.}\\;\\; d(x,\\widetilde{x}) = ||\\phi(x) - \\phi (\\tilde{x}) ||_{2} \\leq \\epsilon$$\n",
    "Here $$L (f({x}),y) = \\max_{i\\neq y} (z_i(x) - z_y(x) ),$$\n",
    "where $z_i(x)$ is the $i$-th logit output of $f(x)$, and $\\phi(\\cdot)$ is a function that maps the input $x$ to  normalized, flattened activations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08dfdd50",
   "metadata": {},
   "source": [
    "## Modules Importing\n",
    "Import all necessary modules and add PyGRANSO src folder to system path. \n",
    "\n",
    "NOTE: the perceptual advex package (https://github.com/cassidylaidlaw/perceptual-advex.git) is required to calculate the distance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "23c19f28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install required package\n",
    "try:\n",
    "    import perceptual_advex\n",
    "except ImportError:\n",
    "    !pip install perceptual-advex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "90ed32f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import torch\n",
    "import sys\n",
    "from pygranso.pygranso import pygranso\n",
    "from pygranso.pygransoStruct import pygransoStruct\n",
    "from pygranso.private.getNvar import getNvarTorch\n",
    "from perceptual_advex.utilities import get_dataset_model\n",
    "from perceptual_advex.perceptual_attacks import get_lpips_model\n",
    "from perceptual_advex.distances import normalize_flatten_features\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0c82d9",
   "metadata": {},
   "source": [
    "## Download Pretrained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "beedb704",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Download ResNet model\n",
    "if not os.path.exists('data/checkpoints/cifar_pgd_l2_1.pt'):\n",
    "    !mkdir -p data/checkpoints\n",
    "    !curl -o data/checkpoints/cifar_pgd_l2_1.pt https://perceptual-advex.s3.us-east-2.amazonaws.com/cifar_pgd_l2_1_cpu.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a1b7fe",
   "metadata": {},
   "source": [
    "## Data Initialization \n",
    "\n",
    "Specify torch device, neural network architecture, and generate data.\n",
    "\n",
    "NOTE: please specify path for downloading data.\n",
    "\n",
    "Use GPU for this problem. If no cuda device available, please set *device = torch.device('cpu')*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8b4842e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=> loading checkpoint 'data/checkpoints/cifar_pgd_l2_1.pt'\n",
      "==> Preparing dataset cifar..\n",
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/multiprocessing/queues.py\", line 251, in _feed\n",
      "    send_bytes(obj)\n",
      "  File \"/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/multiprocessing/connection.py\", line 205, in send_bytes\n",
      "    self._send_bytes(m[offset:offset + size])\n",
      "  File \"/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/multiprocessing/connection.py\", line 416, in _send_bytes\n",
      "    self._send(header + buf)\n",
      "  File \"/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/multiprocessing/connection.py\", line 373, in _send\n",
      "    n = write(self._handle, buf)\n",
      "BrokenPipeError: [Errno 32] Broken pipe\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cuda')\n",
    "\n",
    "dataset, model = get_dataset_model(\n",
    "dataset='cifar',\n",
    "arch='resnet50',\n",
    "checkpoint_fname='data/checkpoints/cifar_pgd_l2_1.pt',\n",
    ")\n",
    "model = model.to(device=device, dtype=torch.double)\n",
    "# Create a validation set loader.\n",
    "batch_size = 1\n",
    "_, val_loader = dataset.make_loaders(1, batch_size, only_val=True, shuffle_val=False)\n",
    "\n",
    "inputs, labels = next(iter(val_loader))\n",
    "\n",
    "# All the user-provided data (vector/matrix/tensor) must be in torch tensor format. \n",
    "# As PyTorch tensor is single precision by default, one must explicitly set `dtype=torch.double`.\n",
    "# Also, please make sure the device of provided torch tensor is the same as opts.torch_device.\n",
    "inputs = inputs.to(device=device, dtype=torch.double)\n",
    "labels = labels.to(device=device)\n",
    "\n",
    "# externally-bounded attack: AlexNet for constraint while ResNet for objective\n",
    "lpips_model = get_lpips_model('alexnet_cifar', model).to(device=device, dtype=torch.double)\n",
    "\n",
    "# Don't reccoment use in the current version. self-bounded attack: AlexNet for both constraint and objective\n",
    "# model = get_lpips_model('alexnet_cifar', model).to(device=device, dtype=torch.double)\n",
    "\n",
    "# attack_type = 'L_2'\n",
    "# attack_type = 'L_inf'\n",
    "attack_type = 'Perceptual'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec80716b",
   "metadata": {},
   "source": [
    "## Function Set-Up\n",
    "\n",
    "Encode the optimization variables, and objective and constraint functions.\n",
    "\n",
    "Note: please strictly follow the format of comb_fn, which will be used in the PyGRANSO main algortihm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fb360e75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# variables and corresponding dimensions.\n",
    "var_in = {\"x_tilde\": list(inputs.shape)}\n",
    "\n",
    "def MarginLoss(logits,labels):\n",
    "    correct_logits = torch.gather(logits, 1, labels.view(-1, 1))\n",
    "    max_2_logits, argmax_2_logits = torch.topk(logits, 2, dim=1)\n",
    "    top_max, second_max = max_2_logits.chunk(2, dim=1)\n",
    "    top_argmax, _ = argmax_2_logits.chunk(2, dim=1)\n",
    "    labels_eq_max = top_argmax.squeeze().eq(labels).float().view(-1, 1)\n",
    "    labels_ne_max = top_argmax.squeeze().ne(labels).float().view(-1, 1)\n",
    "    max_incorrect_logits = labels_eq_max * second_max + labels_ne_max * top_max\n",
    "    loss = -(max_incorrect_logits - correct_logits).clamp(max=1).squeeze().sum()\n",
    "    return loss\n",
    "\n",
    "def user_fn(X_struct,inputs,labels,lpips_model,model):\n",
    "    adv_inputs = X_struct.x_tilde\n",
    "    \n",
    "    # objective function\n",
    "    # 8/255 for L_inf, 1 for L_2, 0.5 for PPGD/LPA\n",
    "    if attack_type == 'L_2':\n",
    "        epsilon = 1\n",
    "    elif attack_type == 'L_inf':\n",
    "        epsilon = 8/255\n",
    "    else:\n",
    "        epsilon = 0.5\n",
    "\n",
    "    logits_outputs = model(adv_inputs)\n",
    "\n",
    "    f = MarginLoss(logits_outputs,labels)\n",
    "\n",
    "    # inequality constraint\n",
    "    ci = pygransoStruct()\n",
    "    if attack_type == 'L_2':\n",
    "        ci.c1 = torch.norm((inputs - adv_inputs).reshape(inputs.size()[0], -1)) - epsilon\n",
    "    elif attack_type == 'L_inf':\n",
    "        ci.c1 = torch.norm((inputs - adv_inputs).reshape(inputs.size()[0], -1), float('inf')) - epsilon\n",
    "    else:\n",
    "        input_features = normalize_flatten_features( lpips_model.features(inputs)).detach()\n",
    "        adv_features = lpips_model.features(adv_inputs)\n",
    "        adv_features = normalize_flatten_features(adv_features)\n",
    "        lpips_dists = (adv_features - input_features).norm(dim=1)\n",
    "        ci.c1 = lpips_dists - epsilon\n",
    "    \n",
    "    # equality constraint \n",
    "    ce = None\n",
    "\n",
    "    return [f,ci,ce]\n",
    "\n",
    "comb_fn = lambda X_struct : user_fn(X_struct,inputs,labels,lpips_model,model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f55ace",
   "metadata": {},
   "source": [
    "## User Options\n",
    "Specify user-defined options for PyGRANSO "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f3a65b57",
   "metadata": {},
   "outputs": [],
   "source": [
    "opts = pygransoStruct()\n",
    "opts.torch_device = device\n",
    "opts.maxit = 100\n",
    "opts.opt_tol = 1e-6\n",
    "opts.print_frequency = 1\n",
    "opts.x0 = torch.reshape(inputs,(torch.numel(inputs),1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bca18c7",
   "metadata": {},
   "source": [
    "## Main Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "632976b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/buyun/anaconda3/envs/cuosqp_pygranso/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448255797/work/c10/core/TensorImpl.h:1156.)\n",
      "  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[33m╔═════ QP SOLVER NOTICE ════════════════════════════════════════════════════════════════════════╗\n",
      "\u001b[0m\u001b[33m║  PyGRANSO requires a quadratic program (QP) solver that has a quadprog-compatible interface,  ║\n",
      "\u001b[0m\u001b[33m║  the default is osqp. Users may provide their own wrapper for the QP solver.                  ║\n",
      "\u001b[0m\u001b[33m║  To disable this notice, set opts.quadprog_info_msg = False                                   ║\n",
      "\u001b[0m\u001b[33m╚═══════════════════════════════════════════════════════════════════════════════════════════════╝\n",
      "\u001b[0m═════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗\n",
      "PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation                                             ║ \n",
      "Version 1.0.0                                                                                                    ║ \n",
      "Licensed under the AGPLv3, Copyright (C) 2021 Tim Mitchell and Buyun Liang                                       ║ \n",
      "═════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣\n",
      "Problem specifications:                                                                                          ║ \n",
      " # of variables                     :   3072                                                                     ║ \n",
      " # of inequality constraints        :      1                                                                     ║ \n",
      " # of equality constraints          :      0                                                                     ║ \n",
      "═════╦═══════════════════════════╦════════════════╦═════════════════╦═══════════════════════╦════════════════════╣\n",
      "     ║ <--- Penalty Function --> ║                ║ Total Violation ║ <--- Line Search ---> ║ <- Stationarity -> ║ \n",
      "Iter ║    Mu    │      Value     ║    Objective   ║   Ineq   │  Eq  ║ SD │ Evals │     t    ║ Grads │    Value   ║ \n",
      "═════╬═══════════════════════════╬════════════════╬═════════════════╬═══════════════════════╬════════════════════╣\n",
      "   0 ║ 1.000000 │  0.50909392739 ║  0.50909392739 ║ 0.000000 │   -  ║ -  │     1 │ 0.000000 ║     1 │ 0.752374   ║ \n",
      "   1 ║ 1.000000 │  0.13139580076 ║  0.13139580076 ║ 0.000000 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.569827   ║ \n",
      "   2 ║ 1.000000 │ -0.01353464659 ║ -0.11231957917 ║ 0.098785 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.364019   ║ \n",
      "   3 ║ 1.000000 │ -0.03305865013 ║ -0.23057790915 ║ 0.197519 │   -  ║ S  │     2 │ 0.500000 ║     1 │ 0.391431   ║ \n",
      "   4 ║ 1.000000 │ -0.07512002132 ║ -0.36057738209 ║ 0.285457 │   -  ║ S  │     2 │ 0.500000 ║     1 │ 0.436613   ║ \n",
      "   5 ║ 1.000000 │ -0.11627463660 ║ -0.49942430204 ║ 0.383150 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.349554   ║ \n",
      "   6 ║ 1.000000 │ -0.17664891011 ║ -0.58140452016 ║ 0.404756 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.484603   ║ \n",
      "   7 ║ 1.000000 │ -0.29911013557 ║ -0.68329983929 ║ 0.384190 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.357295   ║ \n",
      "   8 ║ 1.000000 │ -0.35472121343 ║ -0.75604941979 ║ 0.401328 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.760431   ║ \n",
      "   9 ║ 1.000000 │ -0.46871129466 ║ -0.80585740448 ║ 0.337146 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 1.010051   ║ \n",
      "  10 ║ 1.000000 │ -0.50872108700 ║ -0.88639971711 ║ 0.377679 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.548706   ║ \n",
      "  11 ║ 1.000000 │ -0.62605323050 ║ -0.88862023590 ║ 0.262567 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.638928   ║ \n",
      "  12 ║ 1.000000 │ -0.64317710100 ║ -0.93760108907 ║ 0.294424 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.852888   ║ \n",
      "  13 ║ 1.000000 │ -0.74714953697 ║ -0.98111740251 ║ 0.233968 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.616422   ║ \n",
      "  14 ║ 1.000000 │ -0.79240880586 ║ -1.00000000000 ║ 0.207591 │   -  ║ S  │     2 │ 0.500000 ║     1 │ 0.509754   ║ \n",
      "  15 ║ 1.000000 │ -0.83347523436 ║ -1.00000000000 ║ 0.166525 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.253215   ║ \n",
      "  16 ║ 1.000000 │ -0.86121027483 ║ -0.95196903482 ║ 0.090759 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 1.270118   ║ \n",
      "  17 ║ 1.000000 │ -0.91384321895 ║ -1.00000000000 ║ 0.086157 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.187817   ║ \n",
      "  18 ║ 1.000000 │ -0.93207484436 ║ -0.97803478945 ║ 0.045960 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.748240   ║ \n",
      "  19 ║ 1.000000 │ -0.94498666126 ║ -1.00000000000 ║ 0.055013 │   -  ║ S  │     2 │ 0.500000 ║     1 │ 0.112843   ║ \n",
      "═════╬═══════════════════════════╬════════════════╬═════════════════╬═══════════════════════╬════════════════════╣\n",
      "     ║ <--- Penalty Function --> ║                ║ Total Violation ║ <--- Line Search ---> ║ <- Stationarity -> ║ \n",
      "Iter ║    Mu    │      Value     ║    Objective   ║   Ineq   │  Eq  ║ SD │ Evals │     t    ║ Grads │    Value   ║ \n",
      "═════╬═══════════════════════════╬════════════════╬═════════════════╬═══════════════════════╬════════════════════╣\n",
      "  20 ║ 1.000000 │ -0.96019197251 ║ -0.98795223931 ║ 0.027760 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 1.182802   ║ \n",
      "  21 ║ 1.000000 │ -0.96657925961 ║ -1.00000000000 ║ 0.033421 │   -  ║ S  │     3 │ 0.250000 ║     1 │ 0.065142   ║ \n",
      "  22 ║ 1.000000 │ -0.99053087965 ║ -1.00000000000 ║ 0.009469 │   -  ║ S  │     1 │ 1.000000 ║     1 │ 0.032242   ║ \n",
      "  23 ║ 1.000000 │ -1.00000000000 ║ -1.00000000000 ║ 0.000000 │   -  ║ S  │     2 │ 2.000000 ║     1 │ 0.000000   ║ \n",
      "═════╩═══════════════════════════╩════════════════╩═════════════════╩═══════════════════════╩════════════════════╣\n",
      "F = final iterate, B = Best (to tolerance), MF = Most Feasible                                                   ║ \n",
      "Optimization results:                                                                                            ║ \n",
      "═════╦═══════════════════════════╦════════════════╦═════════════════╦═══════════════════════╦════════════════════╣\n",
      "   F ║          │                ║ -1.00000000000 ║ 0.000000 │   -  ║    │       │          ║       │            ║ \n",
      "   B ║          │                ║ -1.00000000000 ║ 0.000000 │   -  ║    │       │          ║       │            ║ \n",
      "  MF ║          │                ║ -1.00000000000 ║ 0.000000 │   -  ║    │       │          ║       │            ║ \n",
      "═════╩═══════════════════════════╩════════════════╩═════════════════╩═══════════════════════╩════════════════════╣\n",
      "Iterations:              23                                                                                      ║ \n",
      "Function evaluations:    31                                                                                      ║ \n",
      "PyGRANSO termination code: 0 --- converged to stationarity and feasibility tolerances.                           ║ \n",
      "═════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝\n",
      "Total Wall Time: 4.690735816955566s\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "soln = pygranso(var_spec = var_in,combined_fn = comb_fn,user_opts = opts)\n",
    "end = time.time()\n",
    "print(\"Total Wall Time: {}s\".format(end - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc1ca84",
   "metadata": {},
   "source": [
    "## Batch Attacks\n",
    "\n",
    "Apply attacks to multiple images by repeating above steps and calculate the success rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "49584c22",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i = 0\n",
      "i = 8\n",
      "i = 11\n",
      "i = 14\n",
      "i = 18\n",
      "i = 23\n",
      "i = 28\n",
      "i = 34\n",
      "i = 38\n",
      "i = 42\n",
      "i = 45\n",
      "i = 46\n",
      "\n",
      "\n",
      "\n",
      "Model train acc on the original image = 0.24\n",
      "Success rate of attack = 1.0\n",
      "Average distance between attacked image and original image = 0.49154774143691704\n",
      "Average run time of PyGRANSO = 3.7453556855519614s, mean f_eval = 30.833333333333332 iters\n"
     ]
    }
   ],
   "source": [
    "total_count = 50\n",
    "total_diff = 0\n",
    "original_count = 0\n",
    "attack_count = 0\n",
    "total_time = 0\n",
    "total_iterations = 0  \n",
    "i = 0\n",
    "it = iter(val_loader)\n",
    "\n",
    "for i in range(total_count):\n",
    "    # Get a batch from the validation set.\n",
    "    inputs, labels = next(it)\n",
    "    inputs = inputs.to(device=device, dtype=torch.double)\n",
    "    labels = labels.to(device=device)\n",
    "\n",
    "    # variables and corresponding dimensions.\n",
    "    var_in = {\"x_tilde\": list(inputs.shape)}\n",
    "\n",
    "    opts.x0 = torch.reshape(inputs,(torch.numel(inputs),1))\n",
    "    # suppress output\n",
    "    opts.print_level = 0\n",
    "\n",
    "    pred_labels = model(inputs).argmax(1)\n",
    "    if pred_labels == labels:\n",
    "        original_count += 1\n",
    "    else:\n",
    "        continue\n",
    "    \n",
    "    start = time.time()\n",
    "    soln = pygranso(var_spec = var_in,combined_fn = comb_fn,user_opts = opts)\n",
    "    end = time.time()\n",
    "    \n",
    "    # Garbage Collector\n",
    "    gc.collect()\n",
    "    \n",
    "    print(\"i = %d\"%i)\n",
    "    \n",
    "    total_time += end - start\n",
    "    total_iterations += soln.fn_evals\n",
    "\n",
    "    final_adv_input = torch.reshape(soln.final.x,inputs.shape)\n",
    "    pred_labels2 = model(final_adv_input.to(device=device, dtype=torch.double)).argmax(1)\n",
    "\n",
    "    if pred_labels2 == labels:\n",
    "        attack_count += 1\n",
    "        \n",
    "    if attack_type == 'L_2':\n",
    "            diff = torch.norm((inputs.to(device=device, dtype=torch.double) - final_adv_input).reshape(inputs.size()[0], -1))\n",
    "    elif attack_type == 'L_inf':\n",
    "        diff = ( torch.norm((inputs.to(device=device, dtype=torch.double) - final_adv_input).reshape(inputs.size()[0], -1), float('inf') ) )\n",
    "    else:\n",
    "        input_features = normalize_flatten_features( lpips_model.features(inputs)).detach()\n",
    "        adv_features = lpips_model.features(final_adv_input)\n",
    "        adv_features = normalize_flatten_features(adv_features)\n",
    "        lpips_dists = torch.mean((adv_features - input_features).norm(dim=1))\n",
    "        diff = lpips_dists\n",
    "\n",
    "    total_diff += diff\n",
    "\n",
    "print(\"\\n\\n\\nModel train acc on the original image = {}\".format(( original_count/total_count )))\n",
    "print(\"Success rate of attack = {}\".format( (original_count-attack_count)/original_count ))\n",
    "print(\"Average distance between attacked image and original image = {}\".format(total_diff/original_count))\n",
    "print(\"Average run time of PyGRANSO = {}s, mean f_eval = {} iters\".format(total_time/original_count,total_iterations/original_count))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}