Deep Learning

Keith Dillon
Spring 2019

Topic 12: Representations and Features

Outline:

• Representation Learning
• Using Pre-trained Models
• Extracting & Visualizing Features

Reading:

• Chollet 1.1.3,1.1.4, 5.3, 5.4

Perceptron as testing a single feature

Break the perceptron into two steps, the linear part plus the activation function.

Consider what each part does to some data.

Perceptron as computing a feature

A perceptron can be viewed as consisting of two steps:

1. a feature computed by the products with weights
2. a simple test at the activation function to determine if that feature is present

Features mathematically

Recall the function that a perceptron computes is...

$$ y = \sigma\left(\sum_{i=1}^N w_i x_i + b \right) $$

View this as

1. Computing the "matching" inner product.
2. Checking if the "match" is greater than a threshold defined by $b$

Perceptron summary

Raecall that a perceptron finds a linear decision surface to separate classes

Now what happens when there's hidden layers?

Answer: we can get a nonlinear decision surface:

Deep network as "Base" + Perceptron

Consider the output of the last hidden layer.

So what does the base do?

The Representation perspective

A DNN can be viewed as consisting of two major steps:

I. a highly-complex nonlinear transformation performed by hidden layers

II. a simple linear perceptron at the final layer

The purpose of the transformation is to make the data easy to separate

Exercise: what computations are needed to achieve this picture?

The Representation perspective

The hidden layers apply a transformation to the data which we can describe with a function like all neural networks

$$ \mathbf{y}_0 = f(\mathbf x) $$

Then the final output is $y = \sigma(\mathbf w^T \mathbf y_0 + b)$, the perceptron with inputs $\mathbf{y}_0$.

This intermediate output $\mathbf{y}_0$ is viewed as features of the data.

Note that "feature"can be used to mean either:

1. the component of the data, e.g. the pixel values of an eye or nose in an image
2. the "amount" of a feature present. E.g. +1 if the image contains a nose, and 0 if not.

Extracting and Visualizing Features

1. View weights themselves
2. View outputs of neurons for a given input
3. Estimate "receptive fields"

Examine the features computed by your CNN

Save your optimized CNN in your other notebook:

model.save('myCNN1.h5')

from keras import layers from keras import models model = models.Sequential() model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1))) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.Flatten()) model.add(layers.Dense(64, activation='relu')) model.add(layers.Dense(10, activation='softmax'))

# load CNN
from keras.models import load_model

model1 = load_model('myCNN1.h5')
model1.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
_________________________________________________________________
flatten_1 (Flatten)          (None, 7744)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                495680    
_________________________________________________________________
dense_2 (Dense)              (None, 10)                650       
=================================================================
Total params: 515,146
Trainable params: 515,146
Non-trainable params: 0
_________________________________________________________________

Looking at the weights directly

W = model1.get_weights()
print(W)

[array([[[[ 0.12393805,  0.29555812, -0.03443781,  0.10047762,
           0.2740227 , -0.15360133,  0.14966604, -0.37192473,
           0.27105087, -0.32160634,  0.07860904, -0.19558953,
           0.10390168, -0.02034206, -0.30026528, -0.05415158,
           0.14401165, -0.35657546,  0.37038136,  0.1859861 ,
          -0.09194922,  0.26008603, -0.08468723, -0.3625537 ,
          -0.0507572 , -0.06364607,  0.07816954, -0.06191834,
          -0.08335934, -0.0635187 , -0.03148477, -0.15850236]],

        [[-0.24822272,  0.11184768,  0.13335153,  0.08462547,
           0.127243  , -0.41219476, -0.24194096, -0.19882804,
           0.06798392, -0.23773661,  0.0186049 ,  0.02222762,
           0.04680707,  0.05471738, -0.13836642,  0.19851135,
           0.09741544, -0.06913515, -0.4951402 ,  0.08979689,
          -0.38929164,  0.03240624, -0.01615073, -0.36284545,
          -0.47164044,  0.03782435,  0.23953198,  0.22659536,
           0.37381583,  0.12238607, -0.00916939, -0.42900953]],

        [[-0.26856807,  0.1392773 , -0.18961774,  0.41461176,
          -0.04509732, -0.34555224,  0.21346642, -0.07448883,
           0.20474686, -0.29164743, -0.10065962,  0.00422999,
          -0.2539399 , -0.12202153,  0.1397122 ,  0.27534872,
          -0.42934847,  0.11625656, -0.05292435,  0.25224766,
          -0.3292182 , -0.09514748, -0.11023691, -0.14532875,
           0.11225843,  0.10854486, -0.129986  ,  0.23094979,
           0.04776447, -0.2298256 ,  0.01923954, -0.2282658 ]]],


       [[[ 0.1668913 ,  0.04071714, -0.02588835, -0.24602601,
           0.04806589,  0.11980241, -0.1489243 ,  0.19288488,
          -0.17243458, -0.09478007, -0.05166049,  0.03093583,
           0.05174795,  0.01471144, -0.3730575 , -0.17756388,
          -0.46611366, -0.00780934, -0.65050703, -0.04529419,
           0.38779408,  0.2731649 , -0.1129197 , -0.31066474,
           0.00142124,  0.13128506, -0.08610269,  0.10401179,
           0.27428082, -0.01169131,  0.02876723,  0.14822632]],

        [[ 0.2150311 ,  0.04567486,  0.1002462 , -0.25379914,
           0.02431531,  0.07217376, -0.3182986 ,  0.36076963,
          -0.35877088, -0.05731074,  0.05557048,  0.03643727,
          -0.27821106,  0.18090881,  0.07466038,  0.02074893,
          -0.5299377 ,  0.31816655, -0.11251707,  0.15579082,
          -0.03801886,  0.01997136, -0.07771192,  0.06912947,
          -0.28732628,  0.10439617,  0.06328747,  0.01594514,
           0.15998435, -0.0564676 ,  0.02079309,  0.19407962]],

        [[-0.35002303, -0.22171135, -0.16922559, -0.04179449,
          -0.34948352, -0.05636424,  0.23259026,  0.19714142,
          -0.05239949, -0.07166538, -0.04310193, -0.15610601,
           0.02561595, -0.2532092 ,  0.29916826,  0.2209564 ,
          -0.08132499,  0.01879224,  0.37333858, -0.06468213,
          -0.27677524, -0.39835986,  0.07932521,  0.09673759,
           0.2069168 , -0.19497092, -0.01406067,  0.02231553,
          -0.3711613 , -0.35295367, -0.02648262, -0.05890309]]],


       [[[ 0.04622341, -0.3394741 , -0.36368313, -0.19185725,
          -0.2640078 ,  0.14920147,  0.12692942,  0.08246649,
           0.00826664,  0.2919026 , -0.11644731, -0.02316476,
           0.02563399, -0.24131653, -0.34861442, -0.45298064,
          -0.425062  ,  0.05727489, -0.49799997, -0.29876208,
           0.30964154,  0.25526586,  0.10057381, -0.12711   ,
           0.44445905,  0.28775164, -0.35413033, -0.04173014,
          -0.11769666,  0.03034481, -0.03833997,  0.14338265]],

        [[ 0.35251644, -0.28897318,  0.19790012, -0.49865988,
          -0.4207049 ,  0.17000246, -0.04071616, -0.1908288 ,
           0.03709246, -0.03335654,  0.06122651, -0.05774596,
           0.06536673,  0.2568737 , -0.04076527, -0.47682056,
           0.30471343, -0.00446914,  0.0460033 , -0.30743912,
           0.13318692, -0.22945042, -0.18717206,  0.35826412,
          -0.14119938, -0.02401813,  0.01470838, -0.3887071 ,
          -0.1635798 ,  0.02222394,  0.01411585,  0.17973416]],

        [[ 0.04212068, -0.08341552,  0.06394381, -0.28096956,
          -0.2810748 ,  0.196831  , -0.06265098, -0.23665293,
           0.17336074, -0.24377821, -0.04962803, -0.00534903,
           0.04903032,  0.04170244,  0.23406324, -0.27751464,
           0.40706107, -0.14052317,  0.20382592, -0.14133932,
          -0.01120193, -0.18425177, -0.1294573 ,  0.08158613,
          -0.25761947, -0.39402527,  0.17837296, -0.25433525,
          -0.16911371, -0.03170805, -0.0836859 ,  0.25749287]]]],
      dtype=float32), array([-0.17092925,  0.04827565, -0.17189223, -0.00559161, -0.00394613,
       -0.06119642, -0.14307174, -0.02644083, -0.21604455,  0.04209705,
       -0.30551127, -0.07701088, -0.31083563, -0.11654025, -0.09306484,
       -0.04075203,  0.02106469, -0.19440316, -0.0029317 , -0.03296823,
       -0.02075277, -0.06048743, -0.10177936, -0.0547804 , -0.01377105,
       -0.13521011, -0.20464337, -0.05702955, -0.11036282, -0.133393  ,
       -0.26295224, -0.04232927], dtype=float32), array([[[[-4.37147588e-01,  3.31441224e-01, -3.54274303e-01, ...,
          -3.62939745e-01,  3.36853527e-02, -4.51682135e-02],
         [-8.73924494e-02, -7.69407004e-02, -1.95835888e-01, ...,
           1.91837270e-02, -4.48113531e-01,  1.28666058e-01],
         [-9.15526412e-03, -4.42057773e-02,  1.68848991e-01, ...,
          -2.34872326e-01, -1.82369038e-01, -2.52886325e-01],
         ...,
         [-2.16621444e-01,  8.18139240e-02, -2.48104542e-01, ...,
          -4.03857023e-01,  5.67574054e-02, -2.04376161e-01],
         [ 1.25338927e-01, -1.51047230e-01,  3.09814624e-02, ...,
          -4.21439737e-01, -1.34130130e-02, -1.72517195e-01],
         [-3.20258617e-01,  2.19882969e-02, -9.26471353e-02, ...,
           9.04942024e-03, -3.66989747e-02, -3.27173203e-01]],

        [[ 1.34638011e-01, -4.14067388e-01,  3.82067412e-02, ...,
          -2.13749871e-01, -2.16597140e-01, -5.40116280e-02],
         [-1.03807271e-01, -3.04965496e-01, -5.83295107e-01, ...,
           4.85898927e-02, -2.82014102e-01,  5.66588156e-02],
         [-2.71875076e-02,  1.66306347e-01,  5.86352572e-02, ...,
          -5.07192612e-02,  2.43766676e-03,  3.36879715e-02],
         ...,
         [ 8.11636671e-02, -1.01830475e-01,  8.26127455e-02, ...,
          -3.25657219e-01,  4.16316763e-02, -1.87730372e-01],
         [ 1.27491400e-01,  3.34545001e-02, -1.96939241e-02, ...,
          -6.62713647e-02, -6.26005530e-02, -2.37186365e-02],
         [ 3.38199399e-02,  1.22895554e-01, -4.09838930e-02, ...,
          -1.41911522e-01, -1.46261171e-01, -2.94466585e-01]],

        [[-3.98681639e-03, -2.11308405e-01,  8.68455544e-02, ...,
          -1.53529882e-01,  3.59258167e-02,  1.41138107e-01],
         [-1.80236086e-01, -4.62905347e-01,  1.16321631e-01, ...,
           1.70952410e-01, -1.50619950e-02, -5.88682175e-01],
         [-1.87575087e-01,  1.30383363e-02, -1.72292128e-01, ...,
          -5.77368587e-03, -1.17551185e-01,  2.26292089e-01],
         ...,
         [-1.58441253e-02, -3.69946882e-02,  6.56319857e-02, ...,
          -5.39943017e-02, -3.47217522e-03,  7.57845417e-02],
         [-7.07195103e-02,  1.25823036e-01,  2.93516479e-02, ...,
           1.77228618e-02, -2.69725956e-02,  1.52479142e-01],
         [-2.34410554e-01,  1.67367741e-01,  1.28187202e-02, ...,
          -4.97147411e-01, -3.63208085e-01, -1.05173208e-01]]],


       [[[-6.76879406e-01, -1.60532996e-01, -6.10169232e-01, ...,
          -2.89676398e-01, -1.65631138e-02, -2.02041000e-01],
         [-2.57971913e-01,  1.44627824e-01, -4.81692590e-02, ...,
          -1.82291150e-01, -6.47546127e-02, -1.27967626e-01],
         [ 6.40491471e-02, -1.93015561e-01, -1.64642528e-01, ...,
          -3.52580965e-01, -2.16332391e-01,  1.92499235e-02],
         ...,
         [-2.94749349e-01, -1.01444311e-01, -1.30585626e-01, ...,
          -1.37937844e-01,  1.24433495e-01, -9.59236994e-02],
         [ 5.07352017e-02, -3.04834962e-01,  1.31796330e-01, ...,
           8.37097131e-03, -8.83890539e-02, -2.44530272e-02],
         [-1.86882764e-01, -1.32332548e-01, -1.68695778e-01, ...,
           2.68231362e-01, -2.07357779e-01, -1.39940053e-01]],

        [[ 9.34353992e-02, -5.66296875e-01, -4.95342480e-04, ...,
           1.30839586e-01, -9.04313996e-02, -4.74784076e-01],
         [-8.43484774e-02, -2.28009656e-01, -2.11830199e-01, ...,
           3.52460630e-02, -2.55055517e-01, -1.32222503e-01],
         [-1.33088738e-01, -7.25896731e-02,  2.09301710e-02, ...,
          -1.25658900e-01, -1.98163331e-01, -9.16924775e-02],
         ...,
         [-9.36522111e-02, -3.08624923e-01, -1.90736502e-02, ...,
          -1.15550436e-01,  4.92178230e-03, -1.91067532e-01],
         [ 1.03400968e-01,  1.49176084e-02, -4.59595351e-03, ...,
          -2.60424260e-02,  5.94788007e-02, -1.02535248e-01],
         [ 1.25080913e-01, -1.47207662e-01, -2.13400736e-01, ...,
           3.66163135e-01, -3.75050426e-01, -4.81071323e-01]],

        [[ 2.08642215e-01, -1.37235031e-01,  3.61211509e-01, ...,
           1.12294488e-01, -2.42273007e-02,  2.21805140e-01],
         [-4.35178995e-01, -2.42706209e-01, -4.50771809e-01, ...,
           8.46396312e-02, -3.45169082e-02, -2.40960374e-01],
         [-1.30000755e-01,  7.64806047e-02,  5.83241917e-02, ...,
          -1.42246172e-01, -1.45704433e-01,  1.46477789e-01],
         ...,
         [ 3.06809880e-02, -1.64565109e-02,  6.80551082e-02, ...,
           2.98059639e-02,  3.73690426e-02,  8.20716470e-02],
         [-1.39721185e-01,  7.40714446e-02,  4.20065150e-02, ...,
          -8.56121406e-02, -1.79513603e-01,  1.10942863e-01],
         [ 3.65407988e-02, -1.96021006e-01,  1.91911966e-01, ...,
           2.14971527e-01, -2.58672297e-01, -2.01588441e-02]]],


       [[[-4.97740030e-01, -3.09916556e-01,  4.95646782e-02, ...,
          -9.77326855e-02, -2.82290459e-01,  1.00294016e-02],
         [-9.19062570e-02, -8.24612752e-02,  5.12591936e-02, ...,
          -3.76940239e-03,  6.26002550e-02, -6.63519204e-02],
         [ 6.60980567e-02, -1.09259635e-01, -1.91534758e-01, ...,
           2.46625632e-01, -2.96720177e-01, -1.33776590e-01],
         ...,
         [-3.29860389e-01, -2.47844726e-01, -2.06333641e-02, ...,
          -3.91734578e-02, -5.72764538e-02, -8.88518095e-02],
         [ 5.58591746e-02, -1.04299992e-01,  3.00490484e-02, ...,
           9.41738784e-02,  3.35734747e-02, -1.34073704e-01],
         [-2.51120687e-01,  4.04942036e-02, -3.52228343e-01, ...,
           3.96691747e-02, -6.79363251e-01,  1.64845601e-01]],

        [[-4.16523777e-02, -3.47243905e-01, -1.75802514e-01, ...,
           2.24170074e-01,  1.85409039e-02, -5.48021123e-02],
         [ 1.90247258e-03, -3.83644134e-01, -7.43854642e-02, ...,
          -2.11268827e-01, -8.56200680e-02,  6.07667752e-02],
         [-5.70949987e-02,  8.93104728e-03,  9.54431966e-02, ...,
           2.15063691e-01, -1.25950284e-03, -1.82883129e-01],
         ...,
         [ 3.07388194e-02, -1.83093667e-01, -2.27828890e-01, ...,
           2.99052661e-03,  9.88369659e-02, -2.98270464e-01],
         [ 1.26731560e-01, -6.38014497e-03, -5.84385544e-02, ...,
           7.56727532e-02, -1.02719426e-01, -4.78872247e-02],
         [ 6.07875586e-02,  4.82898317e-02, -2.00863749e-01, ...,
           1.93532288e-01, -1.26606181e-01,  7.01787248e-02]],

        [[ 1.34718224e-01,  8.89057070e-02,  3.09266210e-01, ...,
           1.14778511e-01,  2.48811003e-02,  1.57112837e-01],
         [-5.56967631e-02, -8.64299983e-02, -8.79690796e-03, ...,
          -1.14010401e-01, -4.92396988e-02, -3.02588671e-01],
         [-3.06036413e-01, -8.08913559e-02,  1.68385565e-01, ...,
           2.19255269e-01, -5.94564946e-03,  2.60016292e-01],
         ...,
         [ 1.66210145e-01,  3.42071913e-02,  4.71208841e-02, ...,
           8.16620439e-02, -2.66355928e-02,  3.11962646e-02],
         [ 4.67161685e-02,  9.46098343e-02, -7.63248205e-02, ...,
           9.51910838e-02, -4.32659127e-03,  9.37160626e-02],
         [-8.14746991e-02, -1.65592149e-01,  2.01788291e-01, ...,
           1.72578722e-01, -9.17824730e-03, -2.32606512e-02]]]],
      dtype=float32), array([ 0.032965  , -0.04759627, -0.08019461, -0.04698042, -0.04286314,
       -0.13247484, -0.13290064, -0.02096165, -0.07402146, -0.02310568,
       -0.15379825, -0.13416721, -0.10986349,  0.05559081, -0.1631614 ,
        0.01430344, -0.05238677, -0.17334378, -0.07095388, -0.14520007,
       -0.14565195, -0.26706278, -0.2209013 , -0.01291561, -0.10036796,
       -0.19429679,  0.02357538, -0.04641526, -0.08346023, -0.11184764,
       -0.07583415, -0.0479938 , -0.02194562, -0.09130105, -0.06401855,
       -0.08995284, -0.0824646 , -0.0801158 , -0.01307435, -0.04007895,
       -0.06200149, -0.07999768, -0.08491305, -0.06288613, -0.19517672,
       -0.12440615, -0.04435901, -0.14292918,  0.00142239, -0.18893103,
       -0.1555429 , -0.1069064 , -0.0811101 , -0.1175898 , -0.05936356,
       -0.13832006, -0.05462597, -0.1972611 , -0.00175535, -0.1581607 ,
       -0.01287095, -0.08373403, -0.17783903, -0.13234885], dtype=float32), array([[ 0.11926277, -0.01837908,  0.02574452, ..., -0.16927598,
         0.19209668, -0.0415886 ],
       [ 0.1977712 ,  0.15826967, -0.00684569, ..., -0.12439491,
        -0.0132242 , -0.09029384],
       [ 0.07699034,  0.11964823, -0.0288853 , ..., -0.20037042,
         0.2021265 ,  0.16295193],
       ...,
       [ 0.26607114, -0.20542282,  0.02039455, ...,  0.00554273,
         0.09380985, -0.08535405],
       [ 0.30002096,  0.35949206, -0.01994079, ...,  0.19220781,
        -0.0101005 ,  0.18977262],
       [-0.35114333,  0.29958028,  0.02163729, ...,  0.15140225,
         0.0619433 , -0.12142856]], dtype=float32), array([-0.03951303,  0.04707881, -0.02526065, -0.06517662, -0.04485786,
       -0.04939875,  0.02815304, -0.1148628 , -0.03886691, -0.16257545,
        0.01017456, -0.0793589 , -0.02498941, -0.01197931, -0.00920686,
       -0.14661393, -0.2060713 ,  0.10223962, -0.01896688,  0.03222625,
       -0.06999557, -0.01427148, -0.08237164, -0.06256299, -0.03683901,
       -0.00600544, -0.14304265, -0.02039435,  0.06397778, -0.09767182,
       -0.10186456, -0.2131798 , -0.02953694, -0.06424407, -0.10835618,
       -0.02425328, -0.01585833, -0.02304908,  0.02287285, -0.06320777,
       -0.13896787,  0.00461668, -0.2043601 ,  0.01171023, -0.06589849,
       -0.01324438, -0.03593405, -0.07897043,  0.00712475, -0.04263547,
       -0.01284288, -0.10268302,  0.01793077,  0.05831348, -0.00508396,
       -0.01256575, -0.0086979 , -0.05661709, -0.04145973,  0.02000856,
        0.01720177, -0.07598212, -0.03908115, -0.04232547], dtype=float32), array([[-2.59543598e-01,  2.16902681e-02,  2.81356014e-02,
        -3.13205391e-01,  3.81556302e-02, -4.56728399e-01,
         2.57009178e-01, -3.27231318e-01, -1.24626691e-02,
         2.04357311e-01],
       [-4.31414932e-01,  9.82786492e-02,  8.43839198e-02,
         2.37590730e-01, -5.54773867e-01,  2.23203987e-01,
         1.17428839e-01,  5.50991260e-02,  7.64718428e-02,
        -6.08447194e-01],
       [ 2.15727836e-01, -1.02624759e-01, -2.76404262e-01,
        -1.62425473e-01,  2.10823361e-02, -1.21458918e-01,
        -1.22080736e-01,  8.96469876e-02,  1.10748723e-01,
        -2.51234949e-01],
       [ 3.62848431e-01,  3.95167656e-02,  5.75220108e-01,
         2.65450776e-02, -5.71763515e-01,  2.45048255e-01,
        -1.95901304e-01, -1.93517596e-01, -1.30344048e-01,
        -2.13938970e-02],
       [ 2.50623882e-01, -2.67900497e-01,  3.49786758e-01,
        -2.39008721e-02,  3.35083544e-01, -7.14903921e-02,
         2.40875572e-01, -5.54043412e-01, -4.38209213e-02,
        -2.24991471e-01],
       [-5.23245156e-01, -1.16823576e-02, -6.69922411e-01,
         1.85946777e-01, -3.53667676e-01,  1.48609892e-01,
         1.39351368e-01, -3.63621041e-02, -1.42648235e-01,
        -1.11156859e-01],
       [-5.99463545e-02,  1.10785998e-01,  7.58719891e-02,
         3.00377697e-01, -4.98042136e-01, -2.34478682e-01,
        -4.92102891e-01,  2.16698766e-01, -2.70870864e-01,
        -1.22436464e-01],
       [ 1.02456391e-01, -3.81114811e-01, -1.08758420e-01,
        -2.20061734e-01, -2.22631618e-01,  1.84371546e-01,
         1.27987981e-01,  2.39907075e-02, -2.46187299e-02,
         1.23078130e-01],
       [ 1.63250983e-01, -1.68810800e-01, -4.98569816e-01,
        -1.04680784e-01, -4.46266979e-01, -1.63979782e-03,
        -3.24046947e-02, -1.77312025e-03,  2.09705710e-01,
         4.92740311e-02],
       [-1.17777579e-01, -4.65784311e-01, -1.50246397e-01,
        -1.49695918e-01,  1.00790836e-01, -3.63112390e-01,
        -1.95873156e-01,  3.00400674e-01, -5.70838153e-01,
         1.71360716e-01],
       [-2.09371254e-01,  1.24115735e-01,  5.17831519e-02,
        -3.95494580e-01, -4.17893976e-01, -2.03564182e-01,
         4.22529668e-01,  9.71148461e-02, -5.12920558e-01,
        -5.56593657e-01],
       [-4.31740344e-01,  6.24969602e-02,  3.05701882e-01,
        -4.27815914e-01, -3.35274488e-01,  1.34820923e-01,
         2.17691123e-01, -2.32433394e-01,  2.20464170e-02,
        -2.06559360e-01],
       [-1.29940480e-01,  1.51519135e-01,  1.04006551e-01,
         2.27598578e-01, -7.77706364e-03, -2.00641662e-01,
         1.64000496e-01, -2.96422720e-01,  4.07167859e-02,
        -3.65987092e-01],
       [ 2.55478472e-01,  1.56662300e-01, -1.13494219e-02,
         6.27755150e-02, -4.25622851e-01, -2.14432165e-01,
         1.61838263e-01, -1.07325867e-01,  1.41580820e-01,
        -1.65273979e-01],
       [ 1.39180198e-01, -1.86820462e-01, -5.18343508e-01,
         9.95139033e-03,  1.27736881e-01,  1.24397591e-01,
        -1.29542217e-01, -1.02651969e-01,  2.20964804e-01,
         2.69723255e-02],
       [ 2.67558277e-01, -4.38170999e-01,  1.69161171e-01,
        -2.23864898e-01,  6.24648929e-02, -2.05586687e-01,
         2.57094651e-01,  3.80343087e-02,  1.49112448e-01,
         1.68718725e-01],
       [ 5.99703416e-02,  3.33724767e-02, -2.19289765e-01,
        -4.54077154e-01,  3.98389548e-02, -5.40813878e-02,
        -1.91525161e-01,  1.24301918e-01, -1.79584637e-01,
        -1.47394538e-02],
       [-4.74388272e-01,  3.19509283e-02,  1.56984240e-01,
         4.69464809e-02,  2.84727186e-01,  8.58311057e-02,
         1.46022052e-01,  2.58611709e-01,  7.67779201e-02,
        -3.52085650e-01],
       [ 3.74782175e-01, -9.62216333e-02,  3.53694975e-01,
        -2.40433633e-01,  2.80648738e-01, -3.00913215e-01,
        -3.34889352e-01, -2.73288578e-01, -5.78128994e-01,
        -7.38518909e-02],
       [ 1.89503253e-01,  9.81867164e-02, -1.03771381e-01,
        -5.24435222e-01,  2.41089925e-01, -1.68760747e-01,
         1.20574169e-01,  3.45691144e-02,  3.98573935e-01,
        -3.86521637e-01],
       [ 1.29097611e-01, -6.23944521e-01,  7.91106522e-02,
         2.71635592e-01,  1.36089623e-01,  1.96638227e-01,
         1.61767319e-01, -6.69964969e-01, -1.02391042e-01,
         9.58049521e-02],
       [ 2.40423232e-01,  1.43291429e-02, -2.49067545e-01,
        -1.81277648e-01,  5.49674220e-02,  1.28527299e-01,
         1.88483894e-02, -5.46489917e-02, -5.60769513e-02,
         1.46517873e-01],
       [-1.39303520e-01,  3.68744209e-02, -1.89110011e-01,
        -1.56435743e-01, -1.24742210e-01,  1.53428083e-02,
        -5.85594416e-01,  2.09736422e-01, -6.06585085e-01,
         2.69687176e-01],
       [-5.29769480e-01, -1.75515577e-01,  4.91337255e-02,
         1.91839673e-02, -3.52053732e-01, -9.04163942e-02,
         5.15744202e-02,  1.45863313e-02,  3.41941088e-01,
         2.31882438e-01],
       [-1.10011753e-02,  2.37681121e-02,  1.75213233e-01,
         2.30779305e-01,  1.36817575e-01,  1.94366008e-01,
        -2.30391115e-01, -2.12665185e-01, -7.96542838e-02,
         6.44236952e-02],
       [-1.87153667e-01, -2.34555513e-01, -8.61962810e-02,
         9.06504765e-02,  6.13325275e-02,  3.01459786e-02,
        -2.47608557e-01, -1.06228493e-01,  1.04583077e-01,
         1.79534033e-02],
       [ 2.96172835e-02,  1.23220146e-01, -4.12426054e-01,
         3.89065564e-01, -1.03639185e-01, -3.64075333e-01,
        -1.72721565e-01, -1.91362962e-01, -1.77530184e-01,
        -1.86390355e-01],
       [ 9.59043875e-02,  8.25120509e-02,  7.70280734e-02,
        -2.24920601e-01, -1.52547956e-01,  1.26366541e-01,
         1.13916084e-01, -2.44646385e-01, -2.39463091e-01,
        -6.58766180e-02],
       [-2.89638668e-01,  2.34216243e-01, -3.82164121e-02,
        -1.65712103e-01,  2.75099009e-01, -5.53515851e-01,
        -3.72214049e-01,  1.13291673e-01, -2.23806009e-01,
         2.22252890e-01],
       [ 2.70258576e-01, -5.71652412e-01,  2.42970318e-01,
         1.88765451e-01,  5.39087579e-02,  8.58047158e-02,
        -8.61209854e-02, -4.88066494e-01,  2.46045470e-01,
         2.10983545e-01],
       [-2.60957807e-01, -3.99796695e-01, -2.56569535e-01,
        -3.62970948e-01, -2.20374856e-02, -1.78207427e-01,
         1.37099519e-01, -5.32357633e-01,  9.94453058e-02,
         4.05070990e-01],
       [-5.03320396e-01,  3.55873823e-01, -1.77013680e-01,
        -5.20856865e-02, -3.41552734e-01, -2.35873997e-01,
        -4.49899584e-01,  2.04734784e-02, -2.95220017e-01,
         3.79008204e-02],
       [-9.93234068e-02,  3.02770615e-01, -6.56923413e-01,
        -2.55853474e-01,  7.84171000e-02,  2.46386781e-01,
        -3.95807177e-01,  1.92383602e-01, -2.71288082e-02,
         2.15684235e-01],
       [-4.69548404e-01,  1.90080434e-01,  6.08033016e-02,
         1.52849883e-01,  2.13650852e-01,  1.81689888e-01,
        -2.98260540e-01, -7.49957263e-02, -3.34309369e-01,
         1.57840699e-02],
       [-3.50625604e-01, -4.25658047e-01,  2.25623518e-01,
        -1.06252357e-01,  2.27001488e-01,  5.79719283e-02,
        -1.90556511e-01, -2.29112700e-01,  1.38666600e-01,
         1.95174009e-01],
       [-9.54157040e-02,  1.55218795e-01,  4.17091064e-02,
        -2.31757268e-01,  6.61306009e-02, -2.43641868e-01,
        -3.80015433e-01, -9.88941640e-02,  1.95834544e-02,
         1.96066320e-01],
       [-1.43502116e-01, -1.18240481e-02,  7.48008639e-02,
        -1.57033205e-01, -1.98415101e-01, -5.05537450e-01,
        -3.78246784e-01, -2.75915954e-04,  1.67257071e-01,
         1.55258760e-01],
       [-9.46263745e-02,  5.72356135e-02, -2.35265762e-01,
        -4.95791405e-01,  1.15399867e-01, -2.35183045e-01,
        -5.26071906e-01,  2.30919614e-01, -2.14966103e-01,
         1.34628415e-01],
       [-2.82392710e-01,  2.58280635e-01, -4.42604274e-01,
        -2.77332246e-01,  1.73260123e-01,  4.32056576e-01,
        -9.61348489e-02, -4.10355479e-01,  1.77729934e-01,
         1.38066765e-02],
       [ 2.88156420e-01,  1.85991287e-01, -4.54265743e-01,
        -4.46733415e-01, -3.47976267e-01,  3.12545858e-02,
         1.06577449e-01, -1.25274271e-01, -2.40515471e-01,
        -1.28222525e-01],
       [-3.64954919e-01, -1.20660774e-01, -5.05286038e-01,
         1.63469017e-01,  2.20667019e-01, -3.54030758e-01,
        -1.85053974e-01, -3.01645640e-02,  1.31451249e-01,
         1.29577825e-02],
       [ 1.28329977e-01, -2.12173387e-01, -3.20038646e-01,
        -2.87861228e-01,  1.61352590e-01,  6.02975562e-02,
         3.13553870e-01, -5.37717164e-01,  2.26437002e-01,
        -3.03609282e-01],
       [ 8.48934427e-02,  2.88496576e-02, -1.31569117e-01,
         1.49817154e-01,  1.47523642e-01,  7.53719360e-02,
        -1.24106139e-01,  1.06258355e-01, -3.34234506e-01,
         4.72777262e-02],
       [ 1.55614614e-01,  2.55256981e-01,  1.28867954e-01,
         9.66213495e-02,  3.15318882e-01, -1.89925626e-01,
         1.10020384e-01, -3.12961370e-01, -3.39624673e-01,
        -1.48505419e-01],
       [-5.04510045e-01,  2.96874255e-01,  9.09399688e-02,
         1.62764311e-01, -3.55245948e-01,  2.62266815e-01,
        -3.90153438e-01, -1.22111782e-01, -2.35629212e-02,
         2.43598685e-01],
       [ 1.12947695e-01,  1.28254086e-01,  2.39035860e-01,
         1.31588727e-01, -6.16865717e-02,  2.12252960e-01,
        -1.19073026e-01, -1.20601110e-01,  2.09337711e-01,
         1.94588169e-01],
       [-1.19118684e-03, -1.53159156e-01,  2.42594540e-01,
        -2.00991511e-01, -3.13274324e-01, -5.65823674e-01,
        -2.15117022e-01,  3.01331878e-01, -2.54383773e-01,
         2.54868567e-01],
       [-4.24529873e-02, -1.95301950e-01,  1.06023267e-01,
        -1.54251456e-01, -1.41877830e-01,  1.38406023e-01,
        -2.67419904e-01, -3.51010077e-02,  4.03090447e-01,
        -7.19807744e-02],
       [ 1.78090930e-01, -1.36166707e-01,  2.83072680e-01,
        -2.14478180e-01, -2.35732287e-01, -5.98598242e-01,
         1.27002463e-01,  1.32786334e-01, -1.62550598e-01,
        -2.98590481e-01],
       [ 1.80656150e-01, -8.64234269e-02, -5.12276068e-02,
         5.82883097e-02, -4.01430517e-01, -1.40703142e-01,
        -3.52094710e-01, -5.72035909e-01, -1.33127272e-01,
         1.10533409e-01],
       [ 5.51272966e-02, -1.67962179e-01,  2.28646487e-01,
        -1.42697871e-01, -4.28372532e-01, -4.62841123e-01,
        -4.37851161e-01,  6.59916997e-02,  6.27002269e-02,
        -1.35540858e-01],
       [-9.97540131e-02, -2.87052006e-01, -4.61099818e-02,
         3.60196918e-01,  3.61189693e-01, -2.55415022e-01,
        -2.26382002e-01,  1.88661188e-01,  2.91068829e-03,
         3.30247998e-01],
       [-2.98667729e-01,  2.10747406e-01, -3.10678154e-01,
        -2.84058869e-01, -2.50201136e-01, -3.15758049e-01,
         1.74270812e-02,  5.04483581e-01, -3.37414682e-01,
        -8.22458491e-02],
       [-5.11474252e-01, -2.52376646e-02, -5.22838943e-02,
        -1.93627641e-01,  2.69053161e-01,  1.72315672e-01,
        -4.34532911e-01,  1.31506473e-01, -6.81542382e-02,
        -1.66250616e-02],
       [-3.22471231e-01, -9.96440798e-02,  1.00074507e-01,
         3.01131755e-01, -7.17882276e-01, -4.22175586e-01,
        -4.13364768e-01,  2.61002004e-01,  4.12679045e-04,
        -3.34390461e-01],
       [ 1.15283996e-01, -3.60620409e-01, -3.93232286e-01,
         2.62975663e-01, -5.00972331e-01,  8.06024596e-02,
         1.54376164e-01,  2.50913560e-01, -2.14352623e-01,
        -2.31403038e-01],
       [-5.65564454e-01, -3.62165540e-01, -2.42525637e-02,
         2.01102152e-01, -3.68907571e-01,  3.25433500e-02,
        -4.35582638e-01, -3.15824211e-01,  3.63448977e-01,
        -3.85847427e-02],
       [ 3.49537991e-02, -8.33523721e-02, -4.26971942e-01,
         4.17935103e-01, -7.12941885e-01,  2.44917825e-01,
        -2.57787496e-01,  2.03375029e-03, -2.14628633e-02,
         1.14008248e-01],
       [ 1.31246939e-01, -3.04900378e-01,  2.31430635e-01,
         1.82093456e-02, -3.91582787e-01,  1.40807807e-01,
         2.09609553e-01,  3.17412198e-01, -1.95102155e-01,
        -5.42398036e-01],
       [-2.17445329e-01,  8.85255635e-02, -4.88399625e-01,
        -5.18065572e-01, -8.84875059e-02,  1.61469460e-01,
         1.21210217e-01, -9.86769646e-02,  1.06237628e-01,
        -1.54053301e-01],
       [ 1.50614217e-01,  1.42387107e-01,  2.52528787e-01,
        -3.17736641e-02, -4.14587051e-01, -2.23790072e-02,
        -4.61497009e-01,  7.44739398e-02,  2.89066166e-01,
        -4.58382607e-01],
       [ 2.75025666e-01, -5.11224389e-01, -3.35430354e-01,
        -4.34547663e-01, -2.05700859e-01,  2.52158344e-01,
         3.65170505e-04, -1.64568126e-01, -4.96027172e-01,
         1.90451369e-01],
       [ 1.30437851e-01,  6.14309013e-01, -5.49006939e-01,
        -2.01670960e-01, -3.52438301e-01,  2.08187371e-01,
        -2.36225992e-01, -5.19124746e-01, -2.85000384e-01,
         6.65354207e-02],
       [-2.82304972e-01, -4.38806638e-02, -2.60806903e-02,
         3.17924023e-01,  9.18172076e-02, -4.25526381e-01,
        -1.79323196e-01, -1.93694517e-01, -2.43870363e-01,
        -3.44869606e-02]], dtype=float32), array([-0.00868695,  0.10249853, -0.08740643, -0.06786465,  0.11382794,
       -0.08030652, -0.05060877, -0.04772485,  0.13290028, -0.04500815],
      dtype=float32)]

print(W[0].shape)
#plt.imshow(W[0][:,:,0,2])
showCon5x5(W[0])

(3, 3, 1, 32)

print(W[1].shape)
plt.plot(W[1]);

(32,)

Looking at the activations

This means looking at the output of a layer for a single input

There are multiple ways to do this. We will remove the top layer(s) of the model.

from keras.models import load_model

model1 = load_model('myCNN1.h5')
model1.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
_________________________________________________________________
flatten_1 (Flatten)          (None, 7744)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                495680    
_________________________________________________________________
dense_2 (Dense)              (None, 10)                650       
=================================================================
Total params: 515,146
Trainable params: 515,146
Non-trainable params: 0
_________________________________________________________________

Remove Top Layer(s)

model1.pop()
model1.pop()
model1.pop()
model1.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
=================================================================
Total params: 18,816
Trainable params: 18,816
Non-trainable params: 0
_________________________________________________________________

mnist = keras.datasets.mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train / 255.0
X_test = X_test / 255.0

out = model1.predict(np.reshape(X_test[0],(1,28,28,1)))

print(out.shape)
showCon5x5_2(out[0])

(1, 11, 11, 64)

Receptive Fields

The region(s) of inputs mapped to output signals of a neuron

What was this for a simple cell in the vision system?

What is this for a perceptron?

What about a neuron in a deep network?

Using pre-optimized models: pre-made "feature extractors"

Recall we can view a deep feed-forward network as a series of hidden layers plus a final classifier layer.

We reuse the hidden layers for a new task and add (and optimize) a new classifier.

Note: you may have noticed there's no standard direction from input to output; input on top/bottom/left. Even your textbooks can change directions from one page to the next.

Pre-trained models in Keras

dir(keras.applications)

['DenseNet121',
 'DenseNet169',
 'DenseNet201',
 'InceptionResNetV2',
 'InceptionV3',
 'MobileNet',
 'MobileNetV2',
 'NASNetLarge',
 'NASNetMobile',
 'ResNet50',
 'VGG16',
 'VGG19',
 'Xception',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'absolute_import',
 'backend',
 'densenet',
 'division',
 'inception_resnet_v2',
 'inception_v3',
 'keras_applications',
 'keras_modules_injection',
 'layers',
 'mobilenet',
 'mobilenet_v2',
 'models',
 'nasnet',
 'print_function',
 'resnet50',
 'utils',
 'vgg16',
 'vgg19',
 'xception']

Chollet 5.3

https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb

from keras.applications import VGG16

conv_base = VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(150, 150, 3))

conv_base.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         (None, 150, 150, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 150, 150, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 150, 150, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 75, 75, 64)        0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 75, 75, 128)       73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 75, 75, 128)       147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 37, 37, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 37, 37, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 18, 18, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 18, 18, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 9, 9, 512)         0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 4, 4, 512)         0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________

Exercise

Load and summarize a couple other models in Keras.

Exercise: Use VGG16 on MNIST

Min size is 32x32x3, so we need to make our data fit

We can use our generators to zero pad as data is loaded.

But for MNIST we can just adjust them in memory

from keras.applications import VGG16

conv_base = VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(32, 32, 3))

conv_base.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 32, 32, 3)         0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 32, 32, 64)        1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 32, 32, 64)        36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 16, 16, 64)        0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 16, 16, 128)       73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 16, 16, 128)       147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 8, 8, 128)         0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 8, 8, 256)         295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 8, 8, 256)         590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 8, 8, 256)         590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 4, 4, 256)         0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 4, 4, 512)         1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 4, 4, 512)         2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 4, 4, 512)         2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 2, 2, 512)         0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 1, 1, 512)         0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________

#mnist = keras.datasets.mnist
#(X_train, y_train), (X_test, y_test) = mnist.load_data()

from keras.datasets import fashion_mnist
(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()

X_train = X_train / 255.0
X_test = X_test / 255.0

# # note new step needed - convolutional layers expect channel dimension
# X_train = X_train.reshape((60000, 28, 28, 1))
# X_test = X_test.reshape((10000, 28, 28, 1))

X1_train = np.zeros((len(X_train),32,32,3))
for k in range(0,len(X_train)):
    X1_train[k,2:30,2:30,0] = X_train[k]
    X1_train[k,2:30,2:30,1] = X_train[k]
    X1_train[k,2:30,2:30,2] = X_train[k]

X1_test = np.zeros((len(X_test),32,32,3))
for k in range(0,len(X_test)):
    X1_test[k,2:30,2:30,0] = X_test[k]
    X1_test[k,2:30,2:30,1] = X_test[k]
    X1_test[k,2:30,2:30,2] = X_test[k]

plt.imshow(X1_test[0][:,:,1]);

Look at features

y0 = conv_base.predict(X1_train[0:2])
y0.shape

(2, 1, 1, 512)

plt.plot(np.reshape(y0[0],np.prod(y0[0].shape)));
plt.plot(np.reshape(y0[1],np.prod(y0[1].shape)));

Compute Representation of Data

X1_train.shape

(60000, 32, 32, 3)

y0_train = conv_base.predict(X1_train)
y0_test = conv_base.predict(X1_test)

Now make a dense network that classifies using that as input

"Flatten" the data by converting each sample output to a simple vector

y1_train = np.zeros((len(y0_train),512)
for k in range(0,len(x0_train)):
    y1_train[k] = np.reshape(y0_train[k],(512,1))

x0_test = np.zeros((len(y0_test),np.prod(y0_test[0].shape)))
for k in range(0,len(x0_test)):
    x0_test[k] = np.reshape(y0_test[k],np.prod(y0_test[k].shape))    

from keras import models
from keras import layers
from keras import optimizers

model = models.Sequential()
model.add(layers.Dense(256, activation='relu', input_dim=512))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer=optimizers.RMSprop(lr=2e-5),
              loss='binary_crossentropy',
              metrics=['acc'])

history = model.fit(x0_train, y_train,
                    epochs=3,
                    batch_size=20,
                    validation_data=(x0_test, y_test))

Train on 50000 samples, validate on 10000 samples
Epoch 1/3
50000/50000 [==============================] - 16s 323us/step - loss: -54.4193 - acc: 0.0999 - val_loss: -55.7984 - val_acc: 0.1000
Epoch 2/3
50000/50000 [==============================] - 14s 289us/step - loss: -55.7707 - acc: 0.0999 - val_loss: -55.7984 - val_acc: 0.1000
Epoch 3/3
50000/50000 [==============================] - 13s 258us/step - loss: -55.7792 - acc: 0.1000 - val_loss: -55.7984 - val_acc: 0.1000

Plan B: Put new perceptron classifier on network

from keras import models
from keras import layers

model2 = models.Sequential()
model2.add(conv_base)
model2.add(layers.Flatten())
model2.add(layers.Dense(256, activation='relu'))
model2.add(layers.Dense(10, activation='sigmoid'))

model2.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vgg16 (Model)                (None, 1, 1, 512)         14714688  
_________________________________________________________________
flatten_2 (Flatten)          (None, 512)               0         
_________________________________________________________________
dense_9 (Dense)              (None, 256)               131328    
_________________________________________________________________
dense_10 (Dense)             (None, 10)                2570      
=================================================================
Total params: 14,848,586
Trainable params: 133,898
Non-trainable params: 14,714,688
_________________________________________________________________

Freeze convolutional base and fit

conv_base.trainable = False

from keras import optimizers
model2.compile(loss='sparse_categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model2.fit(X1_train, y_train,
                    epochs=10,
                    batch_size=20,
                    validation_data=(X1_test, y_test))

Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 823s 14ms/step - loss: 1.5351 - acc: 0.6240 - val_loss: 0.9877 - val_acc: 0.7661
Epoch 2/10
60000/60000 [==============================] - 802s 13ms/step - loss: 0.8086 - acc: 0.7905 - val_loss: 0.6653 - val_acc: 0.8271
Epoch 3/10
 4320/60000 [=>............................] - ETA: 10:25 - loss: 0.6622 - acc: 0.8234

Exercise

Use NASNet to classify the CIFAR10 dataset

from keras.applications import NASNetMobile

conv_base = NASNetMobile(weights='imagenet',
                  include_top=False,
                  input_shape=(32, 32, 3))

Downloading data from https://github.com/titu1994/Keras-NASNet/releases/download/v1.2/NASNet-mobile-no-top.h5
19996672/19993432 [==============================] - 7s 0us/step

conv_base.summary()

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 32, 32, 3)    0                                            
__________________________________________________________________________________________________
stem_conv1 (Conv2D)             (None, 15, 15, 32)   864         input_1[0][0]                    
__________________________________________________________________________________________________
stem_bn1 (BatchNormalization)   (None, 15, 15, 32)   128         stem_conv1[0][0]                 
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 15, 15, 32)   0           stem_bn1[0][0]                   
__________________________________________________________________________________________________
reduction_conv_1_stem_1 (Conv2D (None, 15, 15, 11)   352         activation_1[0][0]               
__________________________________________________________________________________________________
reduction_bn_1_stem_1 (BatchNor (None, 15, 15, 11)   44          reduction_conv_1_stem_1[0][0]    
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 15, 15, 11)   0           reduction_bn_1_stem_1[0][0]      
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 15, 15, 32)   0           stem_bn1[0][0]                   
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 19, 19, 11)   0           activation_2[0][0]               
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 21, 21, 32)   0           activation_4[0][0]               
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 8, 8, 11)     396         separable_conv_1_pad_reduction_le
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 8, 8, 11)     1920        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 8, 8, 11)     44          separable_conv_1_reduction_left1_
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_1_reduction_right1
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 8, 8, 11)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 8, 8, 11)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 8, 8, 11)     396         activation_3[0][0]               
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 8, 8, 11)     660         activation_5[0][0]               
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 15, 15, 32)   0           stem_bn1[0][0]                   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 8, 8, 11)     44          separable_conv_2_reduction_left1_
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_2_reduction_right1
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 21, 21, 32)   0           activation_6[0][0]               
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 15, 15, 32)   0           stem_bn1[0][0]                   
__________________________________________________________________________________________________
reduction_add_1_stem_1 (Add)    (None, 8, 8, 11)     0           separable_conv_2_bn_reduction_lef
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 8, 8, 11)     1920        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 19, 19, 32)   0           activation_8[0][0]               
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 8, 8, 11)     0           reduction_add_1_stem_1[0][0]     
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_1_reduction_right2
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 8, 8, 11)     1152        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 8, 8, 11)     220         activation_10[0][0]              
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 8, 8, 11)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_1_reduction_right3
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 8, 8, 11)     44          separable_conv_1_reduction_left4_
__________________________________________________________________________________________________
reduction_pad_1_stem_1 (ZeroPad (None, 17, 17, 11)   0           reduction_bn_1_stem_1[0][0]      
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 8, 8, 11)     660         activation_7[0][0]               
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 8, 8, 11)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 8, 8, 11)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
reduction_left2_stem_1 (MaxPool (None, 8, 8, 11)     0           reduction_pad_1_stem_1[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_2_reduction_right2
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 8, 8, 11)     396         activation_9[0][0]               
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 8, 8, 11)     220         activation_11[0][0]              
__________________________________________________________________________________________________
adjust_relu_1_stem_2 (Activatio (None, 15, 15, 32)   0           stem_bn1[0][0]                   
__________________________________________________________________________________________________
reduction_add_2_stem_1 (Add)    (None, 8, 8, 11)     0           reduction_left2_stem_1[0][0]     
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
reduction_left3_stem_1 (Average (None, 8, 8, 11)     0           reduction_pad_1_stem_1[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 8, 8, 11)     44          separable_conv_2_reduction_right3
__________________________________________________________________________________________________
reduction_left4_stem_1 (Average (None, 8, 8, 11)     0           reduction_add_1_stem_1[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 8, 8, 11)     44          separable_conv_2_reduction_left4_
__________________________________________________________________________________________________
reduction_right5_stem_1 (MaxPoo (None, 8, 8, 11)     0           reduction_pad_1_stem_1[0][0]     
__________________________________________________________________________________________________
zero_padding2d_1 (ZeroPadding2D (None, 16, 16, 32)   0           adjust_relu_1_stem_2[0][0]       
__________________________________________________________________________________________________
reduction_add3_stem_1 (Add)     (None, 8, 8, 11)     0           reduction_left3_stem_1[0][0]     
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
add_1 (Add)                     (None, 8, 8, 11)     0           reduction_add_2_stem_1[0][0]     
                                                                 reduction_left4_stem_1[0][0]     
__________________________________________________________________________________________________
reduction_add4_stem_1 (Add)     (None, 8, 8, 11)     0           separable_conv_2_bn_reduction_lef
                                                                 reduction_right5_stem_1[0][0]    
__________________________________________________________________________________________________
cropping2d_1 (Cropping2D)       (None, 15, 15, 32)   0           zero_padding2d_1[0][0]           
__________________________________________________________________________________________________
reduction_concat_stem_1 (Concat (None, 8, 8, 44)     0           reduction_add_2_stem_1[0][0]     
                                                                 reduction_add3_stem_1[0][0]      
                                                                 add_1[0][0]                      
                                                                 reduction_add4_stem_1[0][0]      
__________________________________________________________________________________________________
adjust_avg_pool_1_stem_2 (Avera (None, 8, 8, 32)     0           adjust_relu_1_stem_2[0][0]       
__________________________________________________________________________________________________
adjust_avg_pool_2_stem_2 (Avera (None, 8, 8, 32)     0           cropping2d_1[0][0]               
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 8, 8, 44)     0           reduction_concat_stem_1[0][0]    
__________________________________________________________________________________________________
adjust_conv_1_stem_2 (Conv2D)   (None, 8, 8, 11)     352         adjust_avg_pool_1_stem_2[0][0]   
__________________________________________________________________________________________________
adjust_conv_2_stem_2 (Conv2D)   (None, 8, 8, 11)     352         adjust_avg_pool_2_stem_2[0][0]   
__________________________________________________________________________________________________
reduction_conv_1_stem_2 (Conv2D (None, 8, 8, 22)     968         activation_12[0][0]              
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 8, 8, 22)     0           adjust_conv_1_stem_2[0][0]       
                                                                 adjust_conv_2_stem_2[0][0]       
__________________________________________________________________________________________________
reduction_bn_1_stem_2 (BatchNor (None, 8, 8, 22)     88          reduction_conv_1_stem_2[0][0]    
__________________________________________________________________________________________________
adjust_bn_stem_2 (BatchNormaliz (None, 8, 8, 22)     88          concatenate_1[0][0]              
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 8, 8, 22)     0           reduction_bn_1_stem_2[0][0]      
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 8, 8, 22)     0           adjust_bn_stem_2[0][0]           
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 11, 11, 22)   0           activation_13[0][0]              
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 13, 13, 22)   0           activation_15[0][0]              
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 4, 4, 22)     1034        separable_conv_1_pad_reduction_le
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 4, 4, 22)     1562        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 4, 4, 22)     88          separable_conv_1_reduction_left1_
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_1_reduction_right1
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 4, 4, 22)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 4, 4, 22)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 4, 4, 22)     1034        activation_14[0][0]              
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 4, 4, 22)     1562        activation_16[0][0]              
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 8, 8, 22)     0           adjust_bn_stem_2[0][0]           
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 4, 4, 22)     88          separable_conv_2_reduction_left1_
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_2_reduction_right1
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 13, 13, 22)   0           activation_17[0][0]              
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 8, 8, 22)     0           adjust_bn_stem_2[0][0]           
__________________________________________________________________________________________________
reduction_add_1_stem_2 (Add)    (None, 4, 4, 22)     0           separable_conv_2_bn_reduction_lef
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 4, 4, 22)     1562        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 11, 11, 22)   0           activation_19[0][0]              
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 4, 4, 22)     0           reduction_add_1_stem_2[0][0]     
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_1_reduction_right2
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 4, 4, 22)     1034        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 4, 4, 22)     682         activation_21[0][0]              
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 4, 4, 22)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_1_reduction_right3
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 4, 4, 22)     88          separable_conv_1_reduction_left4_
__________________________________________________________________________________________________
reduction_pad_1_stem_2 (ZeroPad (None, 9, 9, 22)     0           reduction_bn_1_stem_2[0][0]      
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 4, 4, 22)     1562        activation_18[0][0]              
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 4, 4, 22)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 4, 4, 22)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
reduction_left2_stem_2 (MaxPool (None, 4, 4, 22)     0           reduction_pad_1_stem_2[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_2_reduction_right2
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 4, 4, 22)     1034        activation_20[0][0]              
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 4, 4, 22)     682         activation_22[0][0]              
__________________________________________________________________________________________________
adjust_relu_1_0 (Activation)    (None, 8, 8, 44)     0           reduction_concat_stem_1[0][0]    
__________________________________________________________________________________________________
reduction_add_2_stem_2 (Add)    (None, 4, 4, 22)     0           reduction_left2_stem_2[0][0]     
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
reduction_left3_stem_2 (Average (None, 4, 4, 22)     0           reduction_pad_1_stem_2[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 4, 4, 22)     88          separable_conv_2_reduction_right3
__________________________________________________________________________________________________
reduction_left4_stem_2 (Average (None, 4, 4, 22)     0           reduction_add_1_stem_2[0][0]     
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 4, 4, 22)     88          separable_conv_2_reduction_left4_
__________________________________________________________________________________________________
reduction_right5_stem_2 (MaxPoo (None, 4, 4, 22)     0           reduction_pad_1_stem_2[0][0]     
__________________________________________________________________________________________________
zero_padding2d_2 (ZeroPadding2D (None, 9, 9, 44)     0           adjust_relu_1_0[0][0]            
__________________________________________________________________________________________________
reduction_add3_stem_2 (Add)     (None, 4, 4, 22)     0           reduction_left3_stem_2[0][0]     
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
add_2 (Add)                     (None, 4, 4, 22)     0           reduction_add_2_stem_2[0][0]     
                                                                 reduction_left4_stem_2[0][0]     
__________________________________________________________________________________________________
reduction_add4_stem_2 (Add)     (None, 4, 4, 22)     0           separable_conv_2_bn_reduction_lef
                                                                 reduction_right5_stem_2[0][0]    
__________________________________________________________________________________________________
cropping2d_2 (Cropping2D)       (None, 8, 8, 44)     0           zero_padding2d_2[0][0]           
__________________________________________________________________________________________________
reduction_concat_stem_2 (Concat (None, 4, 4, 88)     0           reduction_add_2_stem_2[0][0]     
                                                                 reduction_add3_stem_2[0][0]      
                                                                 add_2[0][0]                      
                                                                 reduction_add4_stem_2[0][0]      
__________________________________________________________________________________________________
adjust_avg_pool_1_0 (AveragePoo (None, 4, 4, 44)     0           adjust_relu_1_0[0][0]            
__________________________________________________________________________________________________
adjust_avg_pool_2_0 (AveragePoo (None, 4, 4, 44)     0           cropping2d_2[0][0]               
__________________________________________________________________________________________________
adjust_conv_1_0 (Conv2D)        (None, 4, 4, 22)     968         adjust_avg_pool_1_0[0][0]        
__________________________________________________________________________________________________
adjust_conv_2_0 (Conv2D)        (None, 4, 4, 22)     968         adjust_avg_pool_2_0[0][0]        
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 4, 4, 88)     0           reduction_concat_stem_2[0][0]    
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (None, 4, 4, 44)     0           adjust_conv_1_0[0][0]            
                                                                 adjust_conv_2_0[0][0]            
__________________________________________________________________________________________________
normal_conv_1_0 (Conv2D)        (None, 4, 4, 44)     3872        activation_23[0][0]              
__________________________________________________________________________________________________
adjust_bn_0 (BatchNormalization (None, 4, 4, 44)     176         concatenate_2[0][0]              
__________________________________________________________________________________________________
normal_bn_1_0 (BatchNormalizati (None, 4, 4, 44)     176         normal_conv_1_0[0][0]            
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_0[0][0]              
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 4, 4, 44)     0           adjust_bn_0[0][0]                
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 4, 4, 44)     0           adjust_bn_0[0][0]                
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 4, 4, 44)     0           adjust_bn_0[0][0]                
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_0[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_0 (None, 4, 4, 44)     3036        activation_24[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 4, 4, 44)     2332        activation_26[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_0 (None, 4, 4, 44)     3036        activation_28[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 4, 4, 44)     2332        activation_30[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left5_0 (None, 4, 4, 44)     2332        activation_32[0][0]              
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left1_0[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right1_0[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left2_0[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right2_0[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left5_0[0
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_0 (None, 4, 4, 44)     3036        activation_25[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 4, 4, 44)     2332        activation_27[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_0 (None, 4, 4, 44)     3036        activation_29[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 4, 4, 44)     2332        activation_31[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left5_0 (None, 4, 4, 44)     2332        activation_33[0][0]              
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left1_0[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right1_0[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left2_0[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right2_0[
__________________________________________________________________________________________________
normal_left3_0 (AveragePooling2 (None, 4, 4, 44)     0           normal_bn_1_0[0][0]              
__________________________________________________________________________________________________
normal_left4_0 (AveragePooling2 (None, 4, 4, 44)     0           adjust_bn_0[0][0]                
__________________________________________________________________________________________________
normal_right4_0 (AveragePooling (None, 4, 4, 44)     0           adjust_bn_0[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left5_0[0
__________________________________________________________________________________________________
normal_add_1_0 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_0 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_0 (Add)            (None, 4, 4, 44)     0           normal_left3_0[0][0]             
                                                                 adjust_bn_0[0][0]                
__________________________________________________________________________________________________
normal_add_4_0 (Add)            (None, 4, 4, 44)     0           normal_left4_0[0][0]             
                                                                 normal_right4_0[0][0]            
__________________________________________________________________________________________________
normal_add_5_0 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_0[0][0]              
__________________________________________________________________________________________________
normal_concat_0 (Concatenate)   (None, 4, 4, 264)    0           adjust_bn_0[0][0]                
                                                                 normal_add_1_0[0][0]             
                                                                 normal_add_2_0[0][0]             
                                                                 normal_add_3_0[0][0]             
                                                                 normal_add_4_0[0][0]             
                                                                 normal_add_5_0[0][0]             
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 4, 4, 88)     0           reduction_concat_stem_2[0][0]    
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 4, 4, 264)    0           normal_concat_0[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_1 (Conv2 (None, 4, 4, 44)     3872        activation_34[0][0]              
__________________________________________________________________________________________________
normal_conv_1_1 (Conv2D)        (None, 4, 4, 44)     11616       activation_35[0][0]              
__________________________________________________________________________________________________
adjust_bn_1 (BatchNormalization (None, 4, 4, 44)     176         adjust_conv_projection_1[0][0]   
__________________________________________________________________________________________________
normal_bn_1_1 (BatchNormalizati (None, 4, 4, 44)     176         normal_conv_1_1[0][0]            
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_1[0][0]              
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 4, 4, 44)     0           adjust_bn_1[0][0]                
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 4, 4, 44)     0           adjust_bn_1[0][0]                
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 4, 4, 44)     0           adjust_bn_1[0][0]                
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_1[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_1 (None, 4, 4, 44)     3036        activation_36[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 4, 4, 44)     2332        activation_38[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_1 (None, 4, 4, 44)     3036        activation_40[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 4, 4, 44)     2332        activation_42[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left5_1 (None, 4, 4, 44)     2332        activation_44[0][0]              
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left1_1[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right1_1[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left2_1[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right2_1[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left5_1[0
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_1 (None, 4, 4, 44)     3036        activation_37[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 4, 4, 44)     2332        activation_39[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_1 (None, 4, 4, 44)     3036        activation_41[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 4, 4, 44)     2332        activation_43[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left5_1 (None, 4, 4, 44)     2332        activation_45[0][0]              
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left1_1[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right1_1[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left2_1[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right2_1[
__________________________________________________________________________________________________
normal_left3_1 (AveragePooling2 (None, 4, 4, 44)     0           normal_bn_1_1[0][0]              
__________________________________________________________________________________________________
normal_left4_1 (AveragePooling2 (None, 4, 4, 44)     0           adjust_bn_1[0][0]                
__________________________________________________________________________________________________
normal_right4_1 (AveragePooling (None, 4, 4, 44)     0           adjust_bn_1[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left5_1[0
__________________________________________________________________________________________________
normal_add_1_1 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_1 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_1 (Add)            (None, 4, 4, 44)     0           normal_left3_1[0][0]             
                                                                 adjust_bn_1[0][0]                
__________________________________________________________________________________________________
normal_add_4_1 (Add)            (None, 4, 4, 44)     0           normal_left4_1[0][0]             
                                                                 normal_right4_1[0][0]            
__________________________________________________________________________________________________
normal_add_5_1 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_1[0][0]              
__________________________________________________________________________________________________
normal_concat_1 (Concatenate)   (None, 4, 4, 264)    0           adjust_bn_1[0][0]                
                                                                 normal_add_1_1[0][0]             
                                                                 normal_add_2_1[0][0]             
                                                                 normal_add_3_1[0][0]             
                                                                 normal_add_4_1[0][0]             
                                                                 normal_add_5_1[0][0]             
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 4, 4, 264)    0           normal_concat_0[0][0]            
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 4, 4, 264)    0           normal_concat_1[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_2 (Conv2 (None, 4, 4, 44)     11616       activation_46[0][0]              
__________________________________________________________________________________________________
normal_conv_1_2 (Conv2D)        (None, 4, 4, 44)     11616       activation_47[0][0]              
__________________________________________________________________________________________________
adjust_bn_2 (BatchNormalization (None, 4, 4, 44)     176         adjust_conv_projection_2[0][0]   
__________________________________________________________________________________________________
normal_bn_1_2 (BatchNormalizati (None, 4, 4, 44)     176         normal_conv_1_2[0][0]            
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_2[0][0]              
__________________________________________________________________________________________________
activation_50 (Activation)      (None, 4, 4, 44)     0           adjust_bn_2[0][0]                
__________________________________________________________________________________________________
activation_52 (Activation)      (None, 4, 4, 44)     0           adjust_bn_2[0][0]                
__________________________________________________________________________________________________
activation_54 (Activation)      (None, 4, 4, 44)     0           adjust_bn_2[0][0]                
__________________________________________________________________________________________________
activation_56 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_2[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_2 (None, 4, 4, 44)     3036        activation_48[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 4, 4, 44)     2332        activation_50[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_2 (None, 4, 4, 44)     3036        activation_52[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 4, 4, 44)     2332        activation_54[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left5_2 (None, 4, 4, 44)     2332        activation_56[0][0]              
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left1_2[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right1_2[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left2_2[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right2_2[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left5_2[0
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_51 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_53 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_55 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_57 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_2 (None, 4, 4, 44)     3036        activation_49[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 4, 4, 44)     2332        activation_51[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_2 (None, 4, 4, 44)     3036        activation_53[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 4, 4, 44)     2332        activation_55[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left5_2 (None, 4, 4, 44)     2332        activation_57[0][0]              
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left1_2[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right1_2[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left2_2[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right2_2[
__________________________________________________________________________________________________
normal_left3_2 (AveragePooling2 (None, 4, 4, 44)     0           normal_bn_1_2[0][0]              
__________________________________________________________________________________________________
normal_left4_2 (AveragePooling2 (None, 4, 4, 44)     0           adjust_bn_2[0][0]                
__________________________________________________________________________________________________
normal_right4_2 (AveragePooling (None, 4, 4, 44)     0           adjust_bn_2[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left5_2[0
__________________________________________________________________________________________________
normal_add_1_2 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_2 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_2 (Add)            (None, 4, 4, 44)     0           normal_left3_2[0][0]             
                                                                 adjust_bn_2[0][0]                
__________________________________________________________________________________________________
normal_add_4_2 (Add)            (None, 4, 4, 44)     0           normal_left4_2[0][0]             
                                                                 normal_right4_2[0][0]            
__________________________________________________________________________________________________
normal_add_5_2 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_2[0][0]              
__________________________________________________________________________________________________
normal_concat_2 (Concatenate)   (None, 4, 4, 264)    0           adjust_bn_2[0][0]                
                                                                 normal_add_1_2[0][0]             
                                                                 normal_add_2_2[0][0]             
                                                                 normal_add_3_2[0][0]             
                                                                 normal_add_4_2[0][0]             
                                                                 normal_add_5_2[0][0]             
__________________________________________________________________________________________________
activation_58 (Activation)      (None, 4, 4, 264)    0           normal_concat_1[0][0]            
__________________________________________________________________________________________________
activation_59 (Activation)      (None, 4, 4, 264)    0           normal_concat_2[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_3 (Conv2 (None, 4, 4, 44)     11616       activation_58[0][0]              
__________________________________________________________________________________________________
normal_conv_1_3 (Conv2D)        (None, 4, 4, 44)     11616       activation_59[0][0]              
__________________________________________________________________________________________________
adjust_bn_3 (BatchNormalization (None, 4, 4, 44)     176         adjust_conv_projection_3[0][0]   
__________________________________________________________________________________________________
normal_bn_1_3 (BatchNormalizati (None, 4, 4, 44)     176         normal_conv_1_3[0][0]            
__________________________________________________________________________________________________
activation_60 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_3[0][0]              
__________________________________________________________________________________________________
activation_62 (Activation)      (None, 4, 4, 44)     0           adjust_bn_3[0][0]                
__________________________________________________________________________________________________
activation_64 (Activation)      (None, 4, 4, 44)     0           adjust_bn_3[0][0]                
__________________________________________________________________________________________________
activation_66 (Activation)      (None, 4, 4, 44)     0           adjust_bn_3[0][0]                
__________________________________________________________________________________________________
activation_68 (Activation)      (None, 4, 4, 44)     0           normal_bn_1_3[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_3 (None, 4, 4, 44)     3036        activation_60[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 4, 4, 44)     2332        activation_62[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_3 (None, 4, 4, 44)     3036        activation_64[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 4, 4, 44)     2332        activation_66[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left5_3 (None, 4, 4, 44)     2332        activation_68[0][0]              
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left1_3[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right1_3[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left2_3[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_1_normal_right2_3[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 4, 4, 44)     176         separable_conv_1_normal_left5_3[0
__________________________________________________________________________________________________
activation_61 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_63 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_65 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_67 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_69 (Activation)      (None, 4, 4, 44)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_3 (None, 4, 4, 44)     3036        activation_61[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 4, 4, 44)     2332        activation_63[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_3 (None, 4, 4, 44)     3036        activation_65[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 4, 4, 44)     2332        activation_67[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left5_3 (None, 4, 4, 44)     2332        activation_69[0][0]              
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left1_3[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right1_3[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left2_3[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 4, 4, 44)     176         separable_conv_2_normal_right2_3[
__________________________________________________________________________________________________
normal_left3_3 (AveragePooling2 (None, 4, 4, 44)     0           normal_bn_1_3[0][0]              
__________________________________________________________________________________________________
normal_left4_3 (AveragePooling2 (None, 4, 4, 44)     0           adjust_bn_3[0][0]                
__________________________________________________________________________________________________
normal_right4_3 (AveragePooling (None, 4, 4, 44)     0           adjust_bn_3[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 4, 4, 44)     176         separable_conv_2_normal_left5_3[0
__________________________________________________________________________________________________
normal_add_1_3 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_3 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_3 (Add)            (None, 4, 4, 44)     0           normal_left3_3[0][0]             
                                                                 adjust_bn_3[0][0]                
__________________________________________________________________________________________________
normal_add_4_3 (Add)            (None, 4, 4, 44)     0           normal_left4_3[0][0]             
                                                                 normal_right4_3[0][0]            
__________________________________________________________________________________________________
normal_add_5_3 (Add)            (None, 4, 4, 44)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_3[0][0]              
__________________________________________________________________________________________________
normal_concat_3 (Concatenate)   (None, 4, 4, 264)    0           adjust_bn_3[0][0]                
                                                                 normal_add_1_3[0][0]             
                                                                 normal_add_2_3[0][0]             
                                                                 normal_add_3_3[0][0]             
                                                                 normal_add_4_3[0][0]             
                                                                 normal_add_5_3[0][0]             
__________________________________________________________________________________________________
activation_71 (Activation)      (None, 4, 4, 264)    0           normal_concat_3[0][0]            
__________________________________________________________________________________________________
activation_70 (Activation)      (None, 4, 4, 264)    0           normal_concat_2[0][0]            
__________________________________________________________________________________________________
reduction_conv_1_reduce_4 (Conv (None, 4, 4, 88)     23232       activation_71[0][0]              
__________________________________________________________________________________________________
adjust_conv_projection_reduce_4 (None, 4, 4, 88)     23232       activation_70[0][0]              
__________________________________________________________________________________________________
reduction_bn_1_reduce_4 (BatchN (None, 4, 4, 88)     352         reduction_conv_1_reduce_4[0][0]  
__________________________________________________________________________________________________
adjust_bn_reduce_4 (BatchNormal (None, 4, 4, 88)     352         adjust_conv_projection_reduce_4[0
__________________________________________________________________________________________________
activation_72 (Activation)      (None, 4, 4, 88)     0           reduction_bn_1_reduce_4[0][0]    
__________________________________________________________________________________________________
activation_74 (Activation)      (None, 4, 4, 88)     0           adjust_bn_reduce_4[0][0]         
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 7, 7, 88)     0           activation_72[0][0]              
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 9, 9, 88)     0           activation_74[0][0]              
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 2, 2, 88)     9944        separable_conv_1_pad_reduction_le
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 2, 2, 88)     12056       separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 2, 2, 88)     352         separable_conv_1_reduction_left1_
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_1_reduction_right1
__________________________________________________________________________________________________
activation_73 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
activation_75 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 2, 2, 88)     9944        activation_73[0][0]              
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 2, 2, 88)     12056       activation_75[0][0]              
__________________________________________________________________________________________________
activation_76 (Activation)      (None, 4, 4, 88)     0           adjust_bn_reduce_4[0][0]         
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 2, 2, 88)     352         separable_conv_2_reduction_left1_
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_2_reduction_right1
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 9, 9, 88)     0           activation_76[0][0]              
__________________________________________________________________________________________________
activation_78 (Activation)      (None, 4, 4, 88)     0           adjust_bn_reduce_4[0][0]         
__________________________________________________________________________________________________
reduction_add_1_reduce_4 (Add)  (None, 2, 2, 88)     0           separable_conv_2_bn_reduction_lef
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 2, 2, 88)     12056       separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 7, 7, 88)     0           activation_78[0][0]              
__________________________________________________________________________________________________
activation_80 (Activation)      (None, 2, 2, 88)     0           reduction_add_1_reduce_4[0][0]   
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_1_reduction_right2
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 2, 2, 88)     9944        separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 2, 2, 88)     8536        activation_80[0][0]              
__________________________________________________________________________________________________
activation_77 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_1_reduction_right3
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 2, 2, 88)     352         separable_conv_1_reduction_left4_
__________________________________________________________________________________________________
reduction_pad_1_reduce_4 (ZeroP (None, 5, 5, 88)     0           reduction_bn_1_reduce_4[0][0]    
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 2, 2, 88)     12056       activation_77[0][0]              
__________________________________________________________________________________________________
activation_79 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
activation_81 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
reduction_left2_reduce_4 (MaxPo (None, 2, 2, 88)     0           reduction_pad_1_reduce_4[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_2_reduction_right2
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 2, 2, 88)     9944        activation_79[0][0]              
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 2, 2, 88)     8536        activation_81[0][0]              
__________________________________________________________________________________________________
adjust_relu_1_5 (Activation)    (None, 4, 4, 264)    0           normal_concat_3[0][0]            
__________________________________________________________________________________________________
reduction_add_2_reduce_4 (Add)  (None, 2, 2, 88)     0           reduction_left2_reduce_4[0][0]   
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
reduction_left3_reduce_4 (Avera (None, 2, 2, 88)     0           reduction_pad_1_reduce_4[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 2, 2, 88)     352         separable_conv_2_reduction_right3
__________________________________________________________________________________________________
reduction_left4_reduce_4 (Avera (None, 2, 2, 88)     0           reduction_add_1_reduce_4[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 2, 2, 88)     352         separable_conv_2_reduction_left4_
__________________________________________________________________________________________________
reduction_right5_reduce_4 (MaxP (None, 2, 2, 88)     0           reduction_pad_1_reduce_4[0][0]   
__________________________________________________________________________________________________
zero_padding2d_3 (ZeroPadding2D (None, 5, 5, 264)    0           adjust_relu_1_5[0][0]            
__________________________________________________________________________________________________
reduction_add3_reduce_4 (Add)   (None, 2, 2, 88)     0           reduction_left3_reduce_4[0][0]   
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
add_3 (Add)                     (None, 2, 2, 88)     0           reduction_add_2_reduce_4[0][0]   
                                                                 reduction_left4_reduce_4[0][0]   
__________________________________________________________________________________________________
reduction_add4_reduce_4 (Add)   (None, 2, 2, 88)     0           separable_conv_2_bn_reduction_lef
                                                                 reduction_right5_reduce_4[0][0]  
__________________________________________________________________________________________________
cropping2d_3 (Cropping2D)       (None, 4, 4, 264)    0           zero_padding2d_3[0][0]           
__________________________________________________________________________________________________
reduction_concat_reduce_4 (Conc (None, 2, 2, 352)    0           reduction_add_2_reduce_4[0][0]   
                                                                 reduction_add3_reduce_4[0][0]    
                                                                 add_3[0][0]                      
                                                                 reduction_add4_reduce_4[0][0]    
__________________________________________________________________________________________________
adjust_avg_pool_1_5 (AveragePoo (None, 2, 2, 264)    0           adjust_relu_1_5[0][0]            
__________________________________________________________________________________________________
adjust_avg_pool_2_5 (AveragePoo (None, 2, 2, 264)    0           cropping2d_3[0][0]               
__________________________________________________________________________________________________
adjust_conv_1_5 (Conv2D)        (None, 2, 2, 44)     11616       adjust_avg_pool_1_5[0][0]        
__________________________________________________________________________________________________
adjust_conv_2_5 (Conv2D)        (None, 2, 2, 44)     11616       adjust_avg_pool_2_5[0][0]        
__________________________________________________________________________________________________
activation_82 (Activation)      (None, 2, 2, 352)    0           reduction_concat_reduce_4[0][0]  
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 2, 2, 88)     0           adjust_conv_1_5[0][0]            
                                                                 adjust_conv_2_5[0][0]            
__________________________________________________________________________________________________
normal_conv_1_5 (Conv2D)        (None, 2, 2, 88)     30976       activation_82[0][0]              
__________________________________________________________________________________________________
adjust_bn_5 (BatchNormalization (None, 2, 2, 88)     352         concatenate_3[0][0]              
__________________________________________________________________________________________________
normal_bn_1_5 (BatchNormalizati (None, 2, 2, 88)     352         normal_conv_1_5[0][0]            
__________________________________________________________________________________________________
activation_83 (Activation)      (None, 2, 2, 88)     0           normal_bn_1_5[0][0]              
__________________________________________________________________________________________________
activation_85 (Activation)      (None, 2, 2, 88)     0           adjust_bn_5[0][0]                
__________________________________________________________________________________________________
activation_87 (Activation)      (None, 2, 2, 88)     0           adjust_bn_5[0][0]                
__________________________________________________________________________________________________
activation_89 (Activation)      (None, 2, 2, 88)     0           adjust_bn_5[0][0]                
__________________________________________________________________________________________________
activation_91 (Activation)      (None, 2, 2, 88)     0           normal_bn_1_5[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_5 (None, 2, 2, 88)     9944        activation_83[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 2, 2, 88)     8536        activation_85[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_5 (None, 2, 2, 88)     9944        activation_87[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 2, 2, 88)     8536        activation_89[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left5_5 (None, 2, 2, 88)     8536        activation_91[0][0]              
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left1_5[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right1_5[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left2_5[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right2_5[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left5_5[0
__________________________________________________________________________________________________
activation_84 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_86 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_88 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_90 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_92 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_5 (None, 2, 2, 88)     9944        activation_84[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 2, 2, 88)     8536        activation_86[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_5 (None, 2, 2, 88)     9944        activation_88[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 2, 2, 88)     8536        activation_90[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left5_5 (None, 2, 2, 88)     8536        activation_92[0][0]              
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left1_5[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right1_5[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left2_5[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right2_5[
__________________________________________________________________________________________________
normal_left3_5 (AveragePooling2 (None, 2, 2, 88)     0           normal_bn_1_5[0][0]              
__________________________________________________________________________________________________
normal_left4_5 (AveragePooling2 (None, 2, 2, 88)     0           adjust_bn_5[0][0]                
__________________________________________________________________________________________________
normal_right4_5 (AveragePooling (None, 2, 2, 88)     0           adjust_bn_5[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left5_5[0
__________________________________________________________________________________________________
normal_add_1_5 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_5 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_5 (Add)            (None, 2, 2, 88)     0           normal_left3_5[0][0]             
                                                                 adjust_bn_5[0][0]                
__________________________________________________________________________________________________
normal_add_4_5 (Add)            (None, 2, 2, 88)     0           normal_left4_5[0][0]             
                                                                 normal_right4_5[0][0]            
__________________________________________________________________________________________________
normal_add_5_5 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_5[0][0]              
__________________________________________________________________________________________________
normal_concat_5 (Concatenate)   (None, 2, 2, 528)    0           adjust_bn_5[0][0]                
                                                                 normal_add_1_5[0][0]             
                                                                 normal_add_2_5[0][0]             
                                                                 normal_add_3_5[0][0]             
                                                                 normal_add_4_5[0][0]             
                                                                 normal_add_5_5[0][0]             
__________________________________________________________________________________________________
activation_93 (Activation)      (None, 2, 2, 352)    0           reduction_concat_reduce_4[0][0]  
__________________________________________________________________________________________________
activation_94 (Activation)      (None, 2, 2, 528)    0           normal_concat_5[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_6 (Conv2 (None, 2, 2, 88)     30976       activation_93[0][0]              
__________________________________________________________________________________________________
normal_conv_1_6 (Conv2D)        (None, 2, 2, 88)     46464       activation_94[0][0]              
__________________________________________________________________________________________________
adjust_bn_6 (BatchNormalization (None, 2, 2, 88)     352         adjust_conv_projection_6[0][0]   
__________________________________________________________________________________________________
normal_bn_1_6 (BatchNormalizati (None, 2, 2, 88)     352         normal_conv_1_6[0][0]            
__________________________________________________________________________________________________
activation_95 (Activation)      (None, 2, 2, 88)     0           normal_bn_1_6[0][0]              
__________________________________________________________________________________________________
activation_97 (Activation)      (None, 2, 2, 88)     0           adjust_bn_6[0][0]                
__________________________________________________________________________________________________
activation_99 (Activation)      (None, 2, 2, 88)     0           adjust_bn_6[0][0]                
__________________________________________________________________________________________________
activation_101 (Activation)     (None, 2, 2, 88)     0           adjust_bn_6[0][0]                
__________________________________________________________________________________________________
activation_103 (Activation)     (None, 2, 2, 88)     0           normal_bn_1_6[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_6 (None, 2, 2, 88)     9944        activation_95[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 2, 2, 88)     8536        activation_97[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left2_6 (None, 2, 2, 88)     9944        activation_99[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 2, 2, 88)     8536        activation_101[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_6 (None, 2, 2, 88)     8536        activation_103[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left1_6[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right1_6[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left2_6[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right2_6[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left5_6[0
__________________________________________________________________________________________________
activation_96 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_98 (Activation)      (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_100 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_102 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_104 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_6 (None, 2, 2, 88)     9944        activation_96[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 2, 2, 88)     8536        activation_98[0][0]              
__________________________________________________________________________________________________
separable_conv_2_normal_left2_6 (None, 2, 2, 88)     9944        activation_100[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 2, 2, 88)     8536        activation_102[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_6 (None, 2, 2, 88)     8536        activation_104[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left1_6[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right1_6[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left2_6[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right2_6[
__________________________________________________________________________________________________
normal_left3_6 (AveragePooling2 (None, 2, 2, 88)     0           normal_bn_1_6[0][0]              
__________________________________________________________________________________________________
normal_left4_6 (AveragePooling2 (None, 2, 2, 88)     0           adjust_bn_6[0][0]                
__________________________________________________________________________________________________
normal_right4_6 (AveragePooling (None, 2, 2, 88)     0           adjust_bn_6[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left5_6[0
__________________________________________________________________________________________________
normal_add_1_6 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_6 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_6 (Add)            (None, 2, 2, 88)     0           normal_left3_6[0][0]             
                                                                 adjust_bn_6[0][0]                
__________________________________________________________________________________________________
normal_add_4_6 (Add)            (None, 2, 2, 88)     0           normal_left4_6[0][0]             
                                                                 normal_right4_6[0][0]            
__________________________________________________________________________________________________
normal_add_5_6 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_6[0][0]              
__________________________________________________________________________________________________
normal_concat_6 (Concatenate)   (None, 2, 2, 528)    0           adjust_bn_6[0][0]                
                                                                 normal_add_1_6[0][0]             
                                                                 normal_add_2_6[0][0]             
                                                                 normal_add_3_6[0][0]             
                                                                 normal_add_4_6[0][0]             
                                                                 normal_add_5_6[0][0]             
__________________________________________________________________________________________________
activation_105 (Activation)     (None, 2, 2, 528)    0           normal_concat_5[0][0]            
__________________________________________________________________________________________________
activation_106 (Activation)     (None, 2, 2, 528)    0           normal_concat_6[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_7 (Conv2 (None, 2, 2, 88)     46464       activation_105[0][0]             
__________________________________________________________________________________________________
normal_conv_1_7 (Conv2D)        (None, 2, 2, 88)     46464       activation_106[0][0]             
__________________________________________________________________________________________________
adjust_bn_7 (BatchNormalization (None, 2, 2, 88)     352         adjust_conv_projection_7[0][0]   
__________________________________________________________________________________________________
normal_bn_1_7 (BatchNormalizati (None, 2, 2, 88)     352         normal_conv_1_7[0][0]            
__________________________________________________________________________________________________
activation_107 (Activation)     (None, 2, 2, 88)     0           normal_bn_1_7[0][0]              
__________________________________________________________________________________________________
activation_109 (Activation)     (None, 2, 2, 88)     0           adjust_bn_7[0][0]                
__________________________________________________________________________________________________
activation_111 (Activation)     (None, 2, 2, 88)     0           adjust_bn_7[0][0]                
__________________________________________________________________________________________________
activation_113 (Activation)     (None, 2, 2, 88)     0           adjust_bn_7[0][0]                
__________________________________________________________________________________________________
activation_115 (Activation)     (None, 2, 2, 88)     0           normal_bn_1_7[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_7 (None, 2, 2, 88)     9944        activation_107[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 2, 2, 88)     8536        activation_109[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_7 (None, 2, 2, 88)     9944        activation_111[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 2, 2, 88)     8536        activation_113[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_7 (None, 2, 2, 88)     8536        activation_115[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left1_7[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right1_7[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left2_7[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right2_7[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left5_7[0
__________________________________________________________________________________________________
activation_108 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_110 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_112 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_114 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_116 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_7 (None, 2, 2, 88)     9944        activation_108[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 2, 2, 88)     8536        activation_110[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_7 (None, 2, 2, 88)     9944        activation_112[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 2, 2, 88)     8536        activation_114[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_7 (None, 2, 2, 88)     8536        activation_116[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left1_7[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right1_7[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left2_7[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right2_7[
__________________________________________________________________________________________________
normal_left3_7 (AveragePooling2 (None, 2, 2, 88)     0           normal_bn_1_7[0][0]              
__________________________________________________________________________________________________
normal_left4_7 (AveragePooling2 (None, 2, 2, 88)     0           adjust_bn_7[0][0]                
__________________________________________________________________________________________________
normal_right4_7 (AveragePooling (None, 2, 2, 88)     0           adjust_bn_7[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left5_7[0
__________________________________________________________________________________________________
normal_add_1_7 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_7 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_7 (Add)            (None, 2, 2, 88)     0           normal_left3_7[0][0]             
                                                                 adjust_bn_7[0][0]                
__________________________________________________________________________________________________
normal_add_4_7 (Add)            (None, 2, 2, 88)     0           normal_left4_7[0][0]             
                                                                 normal_right4_7[0][0]            
__________________________________________________________________________________________________
normal_add_5_7 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_7[0][0]              
__________________________________________________________________________________________________
normal_concat_7 (Concatenate)   (None, 2, 2, 528)    0           adjust_bn_7[0][0]                
                                                                 normal_add_1_7[0][0]             
                                                                 normal_add_2_7[0][0]             
                                                                 normal_add_3_7[0][0]             
                                                                 normal_add_4_7[0][0]             
                                                                 normal_add_5_7[0][0]             
__________________________________________________________________________________________________
activation_117 (Activation)     (None, 2, 2, 528)    0           normal_concat_6[0][0]            
__________________________________________________________________________________________________
activation_118 (Activation)     (None, 2, 2, 528)    0           normal_concat_7[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_8 (Conv2 (None, 2, 2, 88)     46464       activation_117[0][0]             
__________________________________________________________________________________________________
normal_conv_1_8 (Conv2D)        (None, 2, 2, 88)     46464       activation_118[0][0]             
__________________________________________________________________________________________________
adjust_bn_8 (BatchNormalization (None, 2, 2, 88)     352         adjust_conv_projection_8[0][0]   
__________________________________________________________________________________________________
normal_bn_1_8 (BatchNormalizati (None, 2, 2, 88)     352         normal_conv_1_8[0][0]            
__________________________________________________________________________________________________
activation_119 (Activation)     (None, 2, 2, 88)     0           normal_bn_1_8[0][0]              
__________________________________________________________________________________________________
activation_121 (Activation)     (None, 2, 2, 88)     0           adjust_bn_8[0][0]                
__________________________________________________________________________________________________
activation_123 (Activation)     (None, 2, 2, 88)     0           adjust_bn_8[0][0]                
__________________________________________________________________________________________________
activation_125 (Activation)     (None, 2, 2, 88)     0           adjust_bn_8[0][0]                
__________________________________________________________________________________________________
activation_127 (Activation)     (None, 2, 2, 88)     0           normal_bn_1_8[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_8 (None, 2, 2, 88)     9944        activation_119[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 2, 2, 88)     8536        activation_121[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_8 (None, 2, 2, 88)     9944        activation_123[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 2, 2, 88)     8536        activation_125[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_8 (None, 2, 2, 88)     8536        activation_127[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left1_8[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right1_8[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left2_8[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_1_normal_right2_8[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 2, 2, 88)     352         separable_conv_1_normal_left5_8[0
__________________________________________________________________________________________________
activation_120 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_122 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_124 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_126 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_128 (Activation)     (None, 2, 2, 88)     0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_8 (None, 2, 2, 88)     9944        activation_120[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 2, 2, 88)     8536        activation_122[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_8 (None, 2, 2, 88)     9944        activation_124[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 2, 2, 88)     8536        activation_126[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_8 (None, 2, 2, 88)     8536        activation_128[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left1_8[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right1_8[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left2_8[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 2, 2, 88)     352         separable_conv_2_normal_right2_8[
__________________________________________________________________________________________________
normal_left3_8 (AveragePooling2 (None, 2, 2, 88)     0           normal_bn_1_8[0][0]              
__________________________________________________________________________________________________
normal_left4_8 (AveragePooling2 (None, 2, 2, 88)     0           adjust_bn_8[0][0]                
__________________________________________________________________________________________________
normal_right4_8 (AveragePooling (None, 2, 2, 88)     0           adjust_bn_8[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 2, 2, 88)     352         separable_conv_2_normal_left5_8[0
__________________________________________________________________________________________________
normal_add_1_8 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_8 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_8 (Add)            (None, 2, 2, 88)     0           normal_left3_8[0][0]             
                                                                 adjust_bn_8[0][0]                
__________________________________________________________________________________________________
normal_add_4_8 (Add)            (None, 2, 2, 88)     0           normal_left4_8[0][0]             
                                                                 normal_right4_8[0][0]            
__________________________________________________________________________________________________
normal_add_5_8 (Add)            (None, 2, 2, 88)     0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_8[0][0]              
__________________________________________________________________________________________________
normal_concat_8 (Concatenate)   (None, 2, 2, 528)    0           adjust_bn_8[0][0]                
                                                                 normal_add_1_8[0][0]             
                                                                 normal_add_2_8[0][0]             
                                                                 normal_add_3_8[0][0]             
                                                                 normal_add_4_8[0][0]             
                                                                 normal_add_5_8[0][0]             
__________________________________________________________________________________________________
activation_130 (Activation)     (None, 2, 2, 528)    0           normal_concat_8[0][0]            
__________________________________________________________________________________________________
activation_129 (Activation)     (None, 2, 2, 528)    0           normal_concat_7[0][0]            
__________________________________________________________________________________________________
reduction_conv_1_reduce_8 (Conv (None, 2, 2, 176)    92928       activation_130[0][0]             
__________________________________________________________________________________________________
adjust_conv_projection_reduce_8 (None, 2, 2, 176)    92928       activation_129[0][0]             
__________________________________________________________________________________________________
reduction_bn_1_reduce_8 (BatchN (None, 2, 2, 176)    704         reduction_conv_1_reduce_8[0][0]  
__________________________________________________________________________________________________
adjust_bn_reduce_8 (BatchNormal (None, 2, 2, 176)    704         adjust_conv_projection_reduce_8[0
__________________________________________________________________________________________________
activation_131 (Activation)     (None, 2, 2, 176)    0           reduction_bn_1_reduce_8[0][0]    
__________________________________________________________________________________________________
activation_133 (Activation)     (None, 2, 2, 176)    0           adjust_bn_reduce_8[0][0]         
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 5, 5, 176)    0           activation_131[0][0]             
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 7, 7, 176)    0           activation_133[0][0]             
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 1, 1, 176)    35376       separable_conv_1_pad_reduction_le
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 1, 1, 176)    39600       separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 1, 1, 176)    704         separable_conv_1_reduction_left1_
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_1_reduction_right1
__________________________________________________________________________________________________
activation_132 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
activation_134 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 1, 1, 176)    35376       activation_132[0][0]             
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 1, 1, 176)    39600       activation_134[0][0]             
__________________________________________________________________________________________________
activation_135 (Activation)     (None, 2, 2, 176)    0           adjust_bn_reduce_8[0][0]         
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 1, 1, 176)    704         separable_conv_2_reduction_left1_
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_2_reduction_right1
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 7, 7, 176)    0           activation_135[0][0]             
__________________________________________________________________________________________________
activation_137 (Activation)     (None, 2, 2, 176)    0           adjust_bn_reduce_8[0][0]         
__________________________________________________________________________________________________
reduction_add_1_reduce_8 (Add)  (None, 1, 1, 176)    0           separable_conv_2_bn_reduction_lef
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 1, 1, 176)    39600       separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_pad_reduction_ (None, 5, 5, 176)    0           activation_137[0][0]             
__________________________________________________________________________________________________
activation_139 (Activation)     (None, 1, 1, 176)    0           reduction_add_1_reduce_8[0][0]   
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_1_reduction_right2
__________________________________________________________________________________________________
separable_conv_1_reduction_righ (None, 1, 1, 176)    35376       separable_conv_1_pad_reduction_ri
__________________________________________________________________________________________________
separable_conv_1_reduction_left (None, 1, 1, 176)    32560       activation_139[0][0]             
__________________________________________________________________________________________________
activation_136 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_1_reduction_right3
__________________________________________________________________________________________________
separable_conv_1_bn_reduction_l (None, 1, 1, 176)    704         separable_conv_1_reduction_left4_
__________________________________________________________________________________________________
reduction_pad_1_reduce_8 (ZeroP (None, 3, 3, 176)    0           reduction_bn_1_reduce_8[0][0]    
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 1, 1, 176)    39600       activation_136[0][0]             
__________________________________________________________________________________________________
activation_138 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_reduction_rig
__________________________________________________________________________________________________
activation_140 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_reduction_lef
__________________________________________________________________________________________________
reduction_left2_reduce_8 (MaxPo (None, 1, 1, 176)    0           reduction_pad_1_reduce_8[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_2_reduction_right2
__________________________________________________________________________________________________
separable_conv_2_reduction_righ (None, 1, 1, 176)    35376       activation_138[0][0]             
__________________________________________________________________________________________________
separable_conv_2_reduction_left (None, 1, 1, 176)    32560       activation_140[0][0]             
__________________________________________________________________________________________________
adjust_relu_1_9 (Activation)    (None, 2, 2, 528)    0           normal_concat_8[0][0]            
__________________________________________________________________________________________________
reduction_add_2_reduce_8 (Add)  (None, 1, 1, 176)    0           reduction_left2_reduce_8[0][0]   
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
reduction_left3_reduce_8 (Avera (None, 1, 1, 176)    0           reduction_pad_1_reduce_8[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_r (None, 1, 1, 176)    704         separable_conv_2_reduction_right3
__________________________________________________________________________________________________
reduction_left4_reduce_8 (Avera (None, 1, 1, 176)    0           reduction_add_1_reduce_8[0][0]   
__________________________________________________________________________________________________
separable_conv_2_bn_reduction_l (None, 1, 1, 176)    704         separable_conv_2_reduction_left4_
__________________________________________________________________________________________________
reduction_right5_reduce_8 (MaxP (None, 1, 1, 176)    0           reduction_pad_1_reduce_8[0][0]   
__________________________________________________________________________________________________
zero_padding2d_4 (ZeroPadding2D (None, 3, 3, 528)    0           adjust_relu_1_9[0][0]            
__________________________________________________________________________________________________
reduction_add3_reduce_8 (Add)   (None, 1, 1, 176)    0           reduction_left3_reduce_8[0][0]   
                                                                 separable_conv_2_bn_reduction_rig
__________________________________________________________________________________________________
add_4 (Add)                     (None, 1, 1, 176)    0           reduction_add_2_reduce_8[0][0]   
                                                                 reduction_left4_reduce_8[0][0]   
__________________________________________________________________________________________________
reduction_add4_reduce_8 (Add)   (None, 1, 1, 176)    0           separable_conv_2_bn_reduction_lef
                                                                 reduction_right5_reduce_8[0][0]  
__________________________________________________________________________________________________
cropping2d_4 (Cropping2D)       (None, 2, 2, 528)    0           zero_padding2d_4[0][0]           
__________________________________________________________________________________________________
reduction_concat_reduce_8 (Conc (None, 1, 1, 704)    0           reduction_add_2_reduce_8[0][0]   
                                                                 reduction_add3_reduce_8[0][0]    
                                                                 add_4[0][0]                      
                                                                 reduction_add4_reduce_8[0][0]    
__________________________________________________________________________________________________
adjust_avg_pool_1_9 (AveragePoo (None, 1, 1, 528)    0           adjust_relu_1_9[0][0]            
__________________________________________________________________________________________________
adjust_avg_pool_2_9 (AveragePoo (None, 1, 1, 528)    0           cropping2d_4[0][0]               
__________________________________________________________________________________________________
adjust_conv_1_9 (Conv2D)        (None, 1, 1, 88)     46464       adjust_avg_pool_1_9[0][0]        
__________________________________________________________________________________________________
adjust_conv_2_9 (Conv2D)        (None, 1, 1, 88)     46464       adjust_avg_pool_2_9[0][0]        
__________________________________________________________________________________________________
activation_141 (Activation)     (None, 1, 1, 704)    0           reduction_concat_reduce_8[0][0]  
__________________________________________________________________________________________________
concatenate_4 (Concatenate)     (None, 1, 1, 176)    0           adjust_conv_1_9[0][0]            
                                                                 adjust_conv_2_9[0][0]            
__________________________________________________________________________________________________
normal_conv_1_9 (Conv2D)        (None, 1, 1, 176)    123904      activation_141[0][0]             
__________________________________________________________________________________________________
adjust_bn_9 (BatchNormalization (None, 1, 1, 176)    704         concatenate_4[0][0]              
__________________________________________________________________________________________________
normal_bn_1_9 (BatchNormalizati (None, 1, 1, 176)    704         normal_conv_1_9[0][0]            
__________________________________________________________________________________________________
activation_142 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_9[0][0]              
__________________________________________________________________________________________________
activation_144 (Activation)     (None, 1, 1, 176)    0           adjust_bn_9[0][0]                
__________________________________________________________________________________________________
activation_146 (Activation)     (None, 1, 1, 176)    0           adjust_bn_9[0][0]                
__________________________________________________________________________________________________
activation_148 (Activation)     (None, 1, 1, 176)    0           adjust_bn_9[0][0]                
__________________________________________________________________________________________________
activation_150 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_9[0][0]              
__________________________________________________________________________________________________
separable_conv_1_normal_left1_9 (None, 1, 1, 176)    35376       activation_142[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 1, 1, 176)    32560       activation_144[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_9 (None, 1, 1, 176)    35376       activation_146[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 1, 1, 176)    32560       activation_148[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_9 (None, 1, 1, 176)    32560       activation_150[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left1_9[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right1_9[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left2_9[0
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right2_9[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left5_9[0
__________________________________________________________________________________________________
activation_143 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_145 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_147 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_149 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_151 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_9 (None, 1, 1, 176)    35376       activation_143[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 1, 1, 176)    32560       activation_145[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_9 (None, 1, 1, 176)    35376       activation_147[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 1, 1, 176)    32560       activation_149[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_9 (None, 1, 1, 176)    32560       activation_151[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left1_9[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right1_9[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left2_9[0
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right2_9[
__________________________________________________________________________________________________
normal_left3_9 (AveragePooling2 (None, 1, 1, 176)    0           normal_bn_1_9[0][0]              
__________________________________________________________________________________________________
normal_left4_9 (AveragePooling2 (None, 1, 1, 176)    0           adjust_bn_9[0][0]                
__________________________________________________________________________________________________
normal_right4_9 (AveragePooling (None, 1, 1, 176)    0           adjust_bn_9[0][0]                
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left5_9[0
__________________________________________________________________________________________________
normal_add_1_9 (Add)            (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_9 (Add)            (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_9 (Add)            (None, 1, 1, 176)    0           normal_left3_9[0][0]             
                                                                 adjust_bn_9[0][0]                
__________________________________________________________________________________________________
normal_add_4_9 (Add)            (None, 1, 1, 176)    0           normal_left4_9[0][0]             
                                                                 normal_right4_9[0][0]            
__________________________________________________________________________________________________
normal_add_5_9 (Add)            (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_9[0][0]              
__________________________________________________________________________________________________
normal_concat_9 (Concatenate)   (None, 1, 1, 1056)   0           adjust_bn_9[0][0]                
                                                                 normal_add_1_9[0][0]             
                                                                 normal_add_2_9[0][0]             
                                                                 normal_add_3_9[0][0]             
                                                                 normal_add_4_9[0][0]             
                                                                 normal_add_5_9[0][0]             
__________________________________________________________________________________________________
activation_152 (Activation)     (None, 1, 1, 704)    0           reduction_concat_reduce_8[0][0]  
__________________________________________________________________________________________________
activation_153 (Activation)     (None, 1, 1, 1056)   0           normal_concat_9[0][0]            
__________________________________________________________________________________________________
adjust_conv_projection_10 (Conv (None, 1, 1, 176)    123904      activation_152[0][0]             
__________________________________________________________________________________________________
normal_conv_1_10 (Conv2D)       (None, 1, 1, 176)    185856      activation_153[0][0]             
__________________________________________________________________________________________________
adjust_bn_10 (BatchNormalizatio (None, 1, 1, 176)    704         adjust_conv_projection_10[0][0]  
__________________________________________________________________________________________________
normal_bn_1_10 (BatchNormalizat (None, 1, 1, 176)    704         normal_conv_1_10[0][0]           
__________________________________________________________________________________________________
activation_154 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_10[0][0]             
__________________________________________________________________________________________________
activation_156 (Activation)     (None, 1, 1, 176)    0           adjust_bn_10[0][0]               
__________________________________________________________________________________________________
activation_158 (Activation)     (None, 1, 1, 176)    0           adjust_bn_10[0][0]               
__________________________________________________________________________________________________
activation_160 (Activation)     (None, 1, 1, 176)    0           adjust_bn_10[0][0]               
__________________________________________________________________________________________________
activation_162 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_10[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left1_1 (None, 1, 1, 176)    35376       activation_154[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 1, 1, 176)    32560       activation_156[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_1 (None, 1, 1, 176)    35376       activation_158[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 1, 1, 176)    32560       activation_160[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_1 (None, 1, 1, 176)    32560       activation_162[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left1_10[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right1_10
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left2_10[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right2_10
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left5_10[
__________________________________________________________________________________________________
activation_155 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_157 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_159 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_161 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_163 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_1 (None, 1, 1, 176)    35376       activation_155[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 1, 1, 176)    32560       activation_157[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_1 (None, 1, 1, 176)    35376       activation_159[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 1, 1, 176)    32560       activation_161[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_1 (None, 1, 1, 176)    32560       activation_163[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left1_10[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right1_10
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left2_10[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right2_10
__________________________________________________________________________________________________
normal_left3_10 (AveragePooling (None, 1, 1, 176)    0           normal_bn_1_10[0][0]             
__________________________________________________________________________________________________
normal_left4_10 (AveragePooling (None, 1, 1, 176)    0           adjust_bn_10[0][0]               
__________________________________________________________________________________________________
normal_right4_10 (AveragePoolin (None, 1, 1, 176)    0           adjust_bn_10[0][0]               
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left5_10[
__________________________________________________________________________________________________
normal_add_1_10 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_10 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_10 (Add)           (None, 1, 1, 176)    0           normal_left3_10[0][0]            
                                                                 adjust_bn_10[0][0]               
__________________________________________________________________________________________________
normal_add_4_10 (Add)           (None, 1, 1, 176)    0           normal_left4_10[0][0]            
                                                                 normal_right4_10[0][0]           
__________________________________________________________________________________________________
normal_add_5_10 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_10[0][0]             
__________________________________________________________________________________________________
normal_concat_10 (Concatenate)  (None, 1, 1, 1056)   0           adjust_bn_10[0][0]               
                                                                 normal_add_1_10[0][0]            
                                                                 normal_add_2_10[0][0]            
                                                                 normal_add_3_10[0][0]            
                                                                 normal_add_4_10[0][0]            
                                                                 normal_add_5_10[0][0]            
__________________________________________________________________________________________________
activation_164 (Activation)     (None, 1, 1, 1056)   0           normal_concat_9[0][0]            
__________________________________________________________________________________________________
activation_165 (Activation)     (None, 1, 1, 1056)   0           normal_concat_10[0][0]           
__________________________________________________________________________________________________
adjust_conv_projection_11 (Conv (None, 1, 1, 176)    185856      activation_164[0][0]             
__________________________________________________________________________________________________
normal_conv_1_11 (Conv2D)       (None, 1, 1, 176)    185856      activation_165[0][0]             
__________________________________________________________________________________________________
adjust_bn_11 (BatchNormalizatio (None, 1, 1, 176)    704         adjust_conv_projection_11[0][0]  
__________________________________________________________________________________________________
normal_bn_1_11 (BatchNormalizat (None, 1, 1, 176)    704         normal_conv_1_11[0][0]           
__________________________________________________________________________________________________
activation_166 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_11[0][0]             
__________________________________________________________________________________________________
activation_168 (Activation)     (None, 1, 1, 176)    0           adjust_bn_11[0][0]               
__________________________________________________________________________________________________
activation_170 (Activation)     (None, 1, 1, 176)    0           adjust_bn_11[0][0]               
__________________________________________________________________________________________________
activation_172 (Activation)     (None, 1, 1, 176)    0           adjust_bn_11[0][0]               
__________________________________________________________________________________________________
activation_174 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_11[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left1_1 (None, 1, 1, 176)    35376       activation_166[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 1, 1, 176)    32560       activation_168[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_1 (None, 1, 1, 176)    35376       activation_170[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 1, 1, 176)    32560       activation_172[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_1 (None, 1, 1, 176)    32560       activation_174[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left1_11[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right1_11
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left2_11[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right2_11
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left5_11[
__________________________________________________________________________________________________
activation_167 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_169 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_171 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_173 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_175 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_1 (None, 1, 1, 176)    35376       activation_167[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 1, 1, 176)    32560       activation_169[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_1 (None, 1, 1, 176)    35376       activation_171[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 1, 1, 176)    32560       activation_173[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_1 (None, 1, 1, 176)    32560       activation_175[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left1_11[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right1_11
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left2_11[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right2_11
__________________________________________________________________________________________________
normal_left3_11 (AveragePooling (None, 1, 1, 176)    0           normal_bn_1_11[0][0]             
__________________________________________________________________________________________________
normal_left4_11 (AveragePooling (None, 1, 1, 176)    0           adjust_bn_11[0][0]               
__________________________________________________________________________________________________
normal_right4_11 (AveragePoolin (None, 1, 1, 176)    0           adjust_bn_11[0][0]               
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left5_11[
__________________________________________________________________________________________________
normal_add_1_11 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_11 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_11 (Add)           (None, 1, 1, 176)    0           normal_left3_11[0][0]            
                                                                 adjust_bn_11[0][0]               
__________________________________________________________________________________________________
normal_add_4_11 (Add)           (None, 1, 1, 176)    0           normal_left4_11[0][0]            
                                                                 normal_right4_11[0][0]           
__________________________________________________________________________________________________
normal_add_5_11 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_11[0][0]             
__________________________________________________________________________________________________
normal_concat_11 (Concatenate)  (None, 1, 1, 1056)   0           adjust_bn_11[0][0]               
                                                                 normal_add_1_11[0][0]            
                                                                 normal_add_2_11[0][0]            
                                                                 normal_add_3_11[0][0]            
                                                                 normal_add_4_11[0][0]            
                                                                 normal_add_5_11[0][0]            
__________________________________________________________________________________________________
activation_176 (Activation)     (None, 1, 1, 1056)   0           normal_concat_10[0][0]           
__________________________________________________________________________________________________
activation_177 (Activation)     (None, 1, 1, 1056)   0           normal_concat_11[0][0]           
__________________________________________________________________________________________________
adjust_conv_projection_12 (Conv (None, 1, 1, 176)    185856      activation_176[0][0]             
__________________________________________________________________________________________________
normal_conv_1_12 (Conv2D)       (None, 1, 1, 176)    185856      activation_177[0][0]             
__________________________________________________________________________________________________
adjust_bn_12 (BatchNormalizatio (None, 1, 1, 176)    704         adjust_conv_projection_12[0][0]  
__________________________________________________________________________________________________
normal_bn_1_12 (BatchNormalizat (None, 1, 1, 176)    704         normal_conv_1_12[0][0]           
__________________________________________________________________________________________________
activation_178 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_12[0][0]             
__________________________________________________________________________________________________
activation_180 (Activation)     (None, 1, 1, 176)    0           adjust_bn_12[0][0]               
__________________________________________________________________________________________________
activation_182 (Activation)     (None, 1, 1, 176)    0           adjust_bn_12[0][0]               
__________________________________________________________________________________________________
activation_184 (Activation)     (None, 1, 1, 176)    0           adjust_bn_12[0][0]               
__________________________________________________________________________________________________
activation_186 (Activation)     (None, 1, 1, 176)    0           normal_bn_1_12[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left1_1 (None, 1, 1, 176)    35376       activation_178[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right1_ (None, 1, 1, 176)    32560       activation_180[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left2_1 (None, 1, 1, 176)    35376       activation_182[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_right2_ (None, 1, 1, 176)    32560       activation_184[0][0]             
__________________________________________________________________________________________________
separable_conv_1_normal_left5_1 (None, 1, 1, 176)    32560       activation_186[0][0]             
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left1_12[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right1_12
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left2_12[
__________________________________________________________________________________________________
separable_conv_1_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_1_normal_right2_12
__________________________________________________________________________________________________
separable_conv_1_bn_normal_left (None, 1, 1, 176)    704         separable_conv_1_normal_left5_12[
__________________________________________________________________________________________________
activation_179 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left1_
__________________________________________________________________________________________________
activation_181 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right1
__________________________________________________________________________________________________
activation_183 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left2_
__________________________________________________________________________________________________
activation_185 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_right2
__________________________________________________________________________________________________
activation_187 (Activation)     (None, 1, 1, 176)    0           separable_conv_1_bn_normal_left5_
__________________________________________________________________________________________________
separable_conv_2_normal_left1_1 (None, 1, 1, 176)    35376       activation_179[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right1_ (None, 1, 1, 176)    32560       activation_181[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left2_1 (None, 1, 1, 176)    35376       activation_183[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_right2_ (None, 1, 1, 176)    32560       activation_185[0][0]             
__________________________________________________________________________________________________
separable_conv_2_normal_left5_1 (None, 1, 1, 176)    32560       activation_187[0][0]             
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left1_12[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right1_12
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left2_12[
__________________________________________________________________________________________________
separable_conv_2_bn_normal_righ (None, 1, 1, 176)    704         separable_conv_2_normal_right2_12
__________________________________________________________________________________________________
normal_left3_12 (AveragePooling (None, 1, 1, 176)    0           normal_bn_1_12[0][0]             
__________________________________________________________________________________________________
normal_left4_12 (AveragePooling (None, 1, 1, 176)    0           adjust_bn_12[0][0]               
__________________________________________________________________________________________________
normal_right4_12 (AveragePoolin (None, 1, 1, 176)    0           adjust_bn_12[0][0]               
__________________________________________________________________________________________________
separable_conv_2_bn_normal_left (None, 1, 1, 176)    704         separable_conv_2_normal_left5_12[
__________________________________________________________________________________________________
normal_add_1_12 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left1_
                                                                 separable_conv_2_bn_normal_right1
__________________________________________________________________________________________________
normal_add_2_12 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left2_
                                                                 separable_conv_2_bn_normal_right2
__________________________________________________________________________________________________
normal_add_3_12 (Add)           (None, 1, 1, 176)    0           normal_left3_12[0][0]            
                                                                 adjust_bn_12[0][0]               
__________________________________________________________________________________________________
normal_add_4_12 (Add)           (None, 1, 1, 176)    0           normal_left4_12[0][0]            
                                                                 normal_right4_12[0][0]           
__________________________________________________________________________________________________
normal_add_5_12 (Add)           (None, 1, 1, 176)    0           separable_conv_2_bn_normal_left5_
                                                                 normal_bn_1_12[0][0]             
__________________________________________________________________________________________________
normal_concat_12 (Concatenate)  (None, 1, 1, 1056)   0           adjust_bn_12[0][0]               
                                                                 normal_add_1_12[0][0]            
                                                                 normal_add_2_12[0][0]            
                                                                 normal_add_3_12[0][0]            
                                                                 normal_add_4_12[0][0]            
                                                                 normal_add_5_12[0][0]            
__________________________________________________________________________________________________
activation_188 (Activation)     (None, 1, 1, 1056)   0           normal_concat_12[0][0]           
==================================================================================================
Total params: 4,269,716
Trainable params: 4,232,978
Non-trainable params: 36,738
__________________________________________________________________________________________________

from keras.datasets import cifar10

(x_train, y_train), (x_test, y_test) = cifar10.load_data()

showX10x10()

y_train

array([[6],
       [9],
       [9],
       ...,
       [9],
       [1],
       [1]], dtype=uint8)

Look at output of this headless network (the features)

y0 = conv_base.predict(x_train[0:2])
y0.shape

(2, 1, 1, 1056)

plt.plot(np.reshape(y0[0],np.prod(y0[0].shape)));
plt.plot(np.reshape(y0[1],np.prod(y0[1].shape)));

Put classifier on network

from keras import models
from keras import layers

model1 = models.Sequential()
model1.add(conv_base)
model1.add(layers.Flatten())
model1.add(layers.Dense(256, activation='relu'))
model1.add(layers.Dense(10, activation='sigmoid'))

model1.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
NASNet (Model)               (None, 1, 1, 1056)        4269716   
_________________________________________________________________
flatten_2 (Flatten)          (None, 1056)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 256)               270592    
_________________________________________________________________
dense_4 (Dense)              (None, 10)                2570      
=================================================================
Total params: 4,542,878
Trainable params: 4,506,140
Non-trainable params: 36,738
_________________________________________________________________

Freeze convolutional base and fit

conv_base.trainable = False

model1.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
=================================================================
Total params: 515,146
Trainable params: 515,146
Non-trainable params: 0
_________________________________________________________________

from keras import optimizers
model1.compile(loss='sparse_categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model1.fit(x_train, y_train,
                    epochs=10,
                    batch_size=20,
                    validation_data=(x_test, y_test))

Fine Tuning

Rather than freeze entire convolutional base, we can freeze it partially (mostly)

conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

from keras import optimizers
model2.compile(loss='sparse_categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model2.fit(x_train, y_train,
                    epochs=10,
                    batch_size=20,
                    validation_data=(x_test, y_test))

Train on 50000 samples, validate on 10000 samples
Epoch 1/10
20860/50000 [===========>..................] - ETA: 6:01 - loss: 2.7936 - acc: 0.1379