Blue Season

Efficientnet in 5 minutes

2019-09-20T00:00:00+00:00

EfficientNet is evolved from the MobileNet V2 building blocks, with the key insight that scaling up the width, depth or resolution can improve a network’s performance, and a balanced scaling of all three is the key to maximizing improvements. It achieves the state of the art performance with much less parameters and FLOPS than other architectures.

All code shown here are powered by PyWarm, a high level PyTorch API that makes the network definitions super clean.

Building blocks

The main building block, called MBConv, is similar to the bottleneck block from MobileNet V2.

def conv_pad_same(x, size, kernel=1, stride=1, **kw):
    """ Same padding so that out_size*stride == in_size. """
    pad = 0
    if kernel != 1 or stride != 1:
        in_size, s, k = [torch.as_tensor(v)
            for v in (x.shape[2:], stride, kernel)]
        pad = torch.max(((in_size+s-1)//s-1)*s+k-in_size, torch.tensor(0))
        left, right = pad//2, pad-pad//2
        if torch.all(left == right):
            pad = tuple(left.tolist())
        else:
            left, right = left.tolist(), right.tolist()
            pad = sum(zip(left[::-1], right[::-1]), ())
            x = F.pad(x, pad)
            pad = 0
    return W.conv(x, size, kernel, stride=stride, padding=pad, **kw)


def conv_bn_act(x, size, kernel=1, stride=1, groups=1, 
        bias=False, eps=1e-3, momentum=1e-2, act=swish):
    x = conv_pad_same(x, size, kernel, stride=stride, groups=groups, bias=bias)
    return W.batch_norm(x, eps=eps, momentum=momentum, activation=act)


def mb_block(x, size_out, expand=1, kernel=1, stride=1,
        se_ratio=0.25, dc_ratio=0.2):
    """ Mobilenet Bottleneck Block. """
    size_in = x.shape[1]
    size_mid = size_in*expand
    y = conv_bn_act(x, size_mid, 1) if expand > 1 else x
    y = conv_bn_act(y, size_mid, kernel, stride=stride, groups=size_mid)
    y = squeeze_excitation(y, int(size_in*se_ratio))
    y = conv_bn_act(y, size_out, 1, act=None)
    if stride == 1 and size_in == size_out:
        y = drop_connect(y, dc_ratio)
        y += x
    return y

EfficientNet uses Swish instead of ReLU6.

def swish(x):
    return x*torch.sigmoid(x)

Adds squeeze and excitation at the block output.
- Use global average pooling to squeeze the entire receptive field into 1 channel embedding.
- Remap the 1 channel embedding into more channels via excitation.

def squeeze_excitation(x, size_se):
    if size_se == 0:
        return x
    size_in = x.shape[1]
    x = F.adaptive_avg_pool2d(x, 1)
    x = W.conv(x, size_se, 1, activation=swish)
    return W.conv(x, size_in, 1, activation=swish)

Uses drop connect instead of dropout.
- Randomly bypass layers by setting them to identity.

def drop_connect(x, rate):
    if rate == 0:
        return x
    rate = 1.00-rate
    drop_mask = rate + torch.rand([x.shape[0], 1, 1, 1],
        device=x.device, requires_grad=False)
    return x/rate*drop_mask.floor()

Scaling

The paper proposes to scale up width, depth, and resolution propotionally in order to maximize performance gains.

Width: number of channels (hidden dimensions).
Depth: total number of layers.
Resolution: the input image size. The authors used bicubic upsampling to increase the resolution of input images.
Intuitively, if we treat the network as a cylinder, then depth will determine its height and the width and resolution will determine its diameter.
Therefore, when scaling these three dimensions, the ratio should be roughly:

depth : width*width : resolution*resolution ~= 1 : 1 : 1

The paper propsed 8 baselines, B0 to B7, by gradually scaling up theses dimensions under some total FLOPS constraints.
The B0 model has a specification as follows:

spec_b0 = (
# size, expand, kernel, stride, repeat, squeeze_excitation, drop_connect
    (16, 1, 3, 1, 1, 0.25, 0.2),
    (24, 6, 3, 2, 2, 0.25, 0.2),
    (40, 6, 5, 2, 2, 0.25, 0.2),
    (80, 6, 3, 2, 3, 0.25, 0.2),
    (112, 6, 5, 1, 3, 0.25, 0.2),
    (192, 6, 5, 2, 4, 0.25, 0.2),
    (320, 6, 3, 1, 1, 0.25, 0.2), )


class WarmEfficientNet(nn.Module):
    def __init__(self):
        super().__init__()
        warm.up(self, [2, 3, 32, 32])
    def forward(self, x):
        x = conv_bn_act(x, 32, kernel=3, stride=2)
        for size, expand, kernel, stride, repeat, se, dc in spec_b0:
            for i in range(repeat):
                stride = stride if i == 0 else 1
                x = mb_block(x, size, expand, kernel, stride, se, dc)
        x = conv_bn_act(x, 1280)
        x = F.adaptive_avg_pool2d(x, 1)
        x = W.dropout(x, 0.2)
        x = x.view(x.shape[0], -1)
        x = W.linear(x, 1000)
        return x

Other tricks

AutoAugment of inputs
- Automatically search for best input augmentations that lead to highest validation accuracy
Training weight decay
Learning rate decay

Notes

Some users have suggested that EfficientNet, though theoretically uses much fewer FLOPS and number of parameters, does not translate directly to faster GPU inference and less GPU memory usage. In fact, some complained that the training speed is much slower than the well-established ResNets.
People have also mentioned that EfficientNet is very hard to train. You will need to copy the hyperparameters very precisely and empoly all the tricks mentioned in the paper in order to getting close to the reported performance.

A coding style for Python

2019-09-18T00:00:00+00:00

Beautiful is better than ugly.
    - PEP 20, The Zen of Python

PEP 8

Python already has an official style guide, PEP 8. The first doctrine:

“A Foolish Consistency is the Hobgoblin of Little Minds”

Is the particular reason for an upgrade from PEP 8.

For items not listed here, adhere to the PEP 8.

Max Line Length

120 characters. 79 is too short.

Closing Parentheses

Put closing parentheses at the same level of the last object. Do not put them on a new line.

x = dict(a=1, b=2,
    c=3, d=4, ) # closing parentheses same line, yes

x = dict(a=1, b=2,
    c=3, d=4
) # closing parentheses separate line, no

Rationale: Python structures code with indentations rather than parentheses.

Indentation

Indent 1 level (4 spaces) for line continuation, or 2 levels to distinguish from the next line. Never align with opening delimiter:

foo = very_long_function_name(arg,
    var_one=1, var_two=2, var_three=3, ) # 1 level indent, yes


foo = very_long_function_name(
    arg,
    var_one=1,
    var_two=2,
    var_three=3, ) # 1 level indent and align, yes


def very_long_function_name(arg, var_one=1, var_two=2,
        var_three=3, ): # 2 level indent to separate from the next line, yes
    implementation_starts_here = 0

# what if the function name plus indent is really long, like 70 characters?
foo = very_long_function_name(arg,
                              var_one=1,
                              var_two=2,
                              var_three=3) # random align, no

Rationale: Regularly structured code is beautiful. Caparicious alignment is ugly.

Blank Lines

Completely avoid blank lines inside function and methods. Instead, organize / refactor the code to be cleaner and shorter. If there is a really strong need, use an empty comment at the same level of indentation.

if __name__ == '__main__':
    step_1_part_1() # no blank lines, yes
    step_1_part_2()
    step_2_part_1()
    step_2_part_2()
    step_3_part_3()


if __name__ == '__main__':
    step_1_part_1()
    step_1_part_2()
    #
    step_2_part_1() # blank comment to separate major blocks, ok
    step_2_part_2()
    step_3_part_3()

if __name__ == '__main__':
    step_1_part_1() # blank lines, no

    step_1_part_2()

    step_2_part_1()

    step_2_part_2()

    step_3_part_3()

Rationale: If you think the block is getting too dense, it probably means you need to refactor it.

Inline comments

Keep them to a minimum. Comments should reflect intentions, not echo implementaions.

Rationale: If the code is very readable, the comment is likely to be redundant.

String Quotes

In code, use single-quote '' whenever possible. For docstring, use triple double quotes: """ """.

if __name__ == '__main__': # single quote, yes
    # more stuff

if __name__ == "__main__": # double quote, no
    # more stuff

Rationale: Easier to type. Simple is better than complex.

Type Annotations

Prefer detailed type information in the docstring than type annotations in the definitions. However, if the toolset can provide useful functionalities from type annotation, it’s absolutly ok to use.

Rationale: docstrings are meant to be read by end users.

Naming Conventions

If you don’t want others to mess around with an object, add a single underscore before its name:

_do_not_mess_with_me

If a function / method alters its inputs, add a single underscore after it:

i_mess_with_inputs_(x) # content of x will change after each call

Transformer in 5 minutes

2019-09-08T00:00:00+00:00

The transformer architecture from Attention is all you need is the most important technology for natural language processing in recent years. This post is basically a TL;DR + ELI5 version explaining this ingenious model design.

All code shown here are powered by PyWarm, a high level PyTorch API that makes the network definitions super clean.

Transformer

Transformer == 6 encoders + 6 decoders
The encoders encode a source sequence (for example, an English sentence) into a memory representation
The decoders compare the memory with a target sequence (for example, a corresponding sentence in Spanish) to gain experience in order to handle future situations

def transformer(x, y, **kw):
    """ x is the source sequence, y is the target sequence """
    x = encoder(x, **kw)
    x = decoder(x, y, **kw)
    return x

Encoder

Each encoder consists of two blocks:

A self-attention block, that compares different positions in the source sequence to evaluate their relative importances (weights)
A feed-forward block, that learns the memory representation
We will discuss these two blocks shortly

def encoder(x, num_encoder=6, **kw):
    for i in range(num_encoder):
        x = residual_add(x, multi_head_attention, **kw)
        x = residual_add(x, feed_forward, **kw)
    return W.layer_norm(x)

Decoder

Each decoder consists of three blocks:

A self-attention block for the memory (i.e. the output of the encoder stack)
- Because the decoder has access to the entire memory sequence all at once, for any time steps, it may think that future steps can contribute to the past steps, which breaks causality
- To ensure causality, a mask is used to prevent the future leaks into the past:
A memory-target attention block that compares memory with target to establish the relative importances
A feed-forward block, that learns to gain experience for future tasks

def decoder(x, y, num_decoder=6, mask_x=None, mask_y=None, **kw):
    """ x is the memory sequence, y is the target sequence """
    for i in range(num_decoder):
        y = residual_add(y, multi_head_attention, mask=mask_y, **kw)
        y = residual_add(x, multi_head_attention, y=y, mask=mask_x, **kw)
        y = residual_add(y, feed_forward, **kw)
    return W.layer_norm(y)

Feedforward

A feed-forward block is just two linear layers one after another

A linear layer (sometimes also called dense, or fully connected layer) is essentially the most basic form of neural networks.

def feed_forward(x, size_ff=2048, dropout=0.1, **kw):
    y = W.linear(x, size_ff, activation='relu')
    y = W.dropout(y, dropout)
    return W.linear(y, x.shape[1])

Attention

The attention mechanism is often considered the most important contribution of the transformer architecture.

In the most general sense, attention maps one sequence with another sequence to establish relative importance between all pairs of elements
If the second sequence is the same as the first one, it is then a self-attention
The attention algorithm works as follows:
- First, a query vector is derived from the first sequence
- Then, a pair of key vector and value vectors are derived from the second sequence
- The similarity between the key and query is measured and used as the attention weights
- The attentions are then applied to the value vector to get a memory vector as output
A multi-head attention is obtained by splitting the key, query and value into shorter vectors to work in parallel
- This may work better than a single-head attention because each head may attend to different parts better

def multi_head_attention(
        x, y=None, num_head=8, dropout=0.1, mask=None, **kw):
    def split_heads(t):
        return t.reshape(batch, num_head, size//num_head, t.shape[-1])
    def merge_heads(t):
        return t.reshape(batch, -1, t.shape[-1])
    if y is None:
        y = x # self attention
    batch, size = x.shape[:2]
    assert size%num_head == 0
    assert y.shape[:2] == x.shape[:2]
    q = W.linear(x, size) # query
    k = W.linear(y, size) # key
    v = W.linear(y, size) # value
    q = split_heads(q)
    k = split_heads(k)
    v = split_heads(v)
    q *= (size//num_head)**(-0.5)
    a = q.transpose(2, 3).contiguous().matmul(k) # attention weights
    if mask is not None:
        a += mask
    a = F.softmax(a, dim=-1)
    a = W.dropout(a, dropout)
    x = v.matmul(a.transpose(2, 3).contiguous())
    x = merge_heads(x)
    return W.linear(x, size)

Residual Connection

All blocks in the transformer use the residual connection paradigm, which enables deeper neural networks
In each block there is also a normalization step, which makes the network easier to train

def residual_add(x, layer, dropout=0.1, **kw):
    y = W.layer_norm(x)
    y = layer(y, **kw)
    y = W.dropout(y, dropout)
    return x+y

Notes

Technically, the transformer architecture from the original paper also includes (word) embedding and positional encoding for the source and target, but usually people consider these as preprocessing steps and not part of the transformer. Therefore, we will skip them this time and cover these topics in later posts. Just remember they preprocess the input sequences into a format better suited for neural networks.
Attention is all you need
The architecture diagram from the original paper is also included here as a reference